Author

Donghan Yu

29 papers · 1998–2026

1998peak 5 in 20242026

Frequent co-authors

Papers

showing 1–25 of 29 RSS

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Marah Abdin, Jyoti Aneja, Hany Awadalla +126 more

cs.CL · April 2024 · cited 2,528× · arXiv ↗ · more like this ·

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce…

JAKET: Joint Pre-training of Knowledge Graph and Language Understanding

Donghan Yu, Chenguang Zhu, Yiming Yang +1 more

cs.CL · 2 Oct 2020 · cited 176× · arXiv ↗ · more like this ·

Knowledge graphs (KGs) contain rich information about world knowledge, entities and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains a challenge to efficiently integrate information from KG into language modeling. And the understanding of a knowledge graph requires related context. We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and relations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experimental results on several knowledge-aware NLP tasks show that our…

DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases

Donghan Yu, Sheng Zhang, Patrick Ng +7 more

cs.CL · October 2022 · cited 173× · arXiv ↗ · more like this ·

Question answering over knowledge bases (KBs) aims to answer natural language questions with factual information such as entities and relations in KBs. Previous methods either generate logical forms that can be executed over KBs to obtain final answers or predict answers directly. Empirical results show that the former often produces more accurate answers, but it suffers from non-execution issues due to potential syntactic and semantic errors in the generated logical forms. In this work, we propose a novel framework DecAF that jointly generates both logical forms and direct answers, and then combines the merits of them to get the final answers. Moreover, different from most of the previous methods, DecAF is based on simple free-text retrieval without relying on any entity linking tools -- this simplification eases its adaptation to different datasets. DecAF achieves new state-of-the-art…

KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering

Donghan Yu, Chenguang Zhu, Yuwei Fang +6 more

cs.CL · October 2021 · cited 116× · arXiv ↗ · more like this ·

Current Open-Domain Question Answering (ODQA) model paradigm often contains a retrieving module and a reading module. Given an input question, the reading module predicts the answer from the relevant passages which are retrieved by the retriever. The recent proposed Fusion-in-Decoder (FiD), which is built on top of the pretrained generative model T5, achieves the state-of-the-art performance in the reading module. Although being effective, it remains constrained by inefficient attention on all retrieved passages which contain a lot of noise. In this work, we propose a novel method KG-FiD, which filters noisy passages by leveraging the structural relationship among the retrieved passages with a knowledge graph. We initiate the passage node embedding from the FiD encoder and then use graph neural network (GNN) to update the representation for reranking. To improve the efficiency, we build…

Dict-BERT: Enhancing Language Model Pre-training with Dictionary

Wenhao Yu, Chenguang Zhu, Yuwei Fang +5 more

cs.CL · October 2021 · cited 77× · arXiv ↗ · more like this ·

Pre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on large-scale corpora. Since PLMs capture word semantics in different contexts, the quality of word representations highly depends on word frequency, which usually follows a heavy-tailed distributions in the pre-training corpus. Therefore, the embeddings of rare words on the tail are usually poorly optimized. In this work, we focus on enhancing language model pre-training by leveraging definitions of the rare words in dictionaries (e.g., Wiktionary). To incorporate a rare word definition as a part of input, we fetch its definition from the dictionary and append it to the end of the input text sequence. In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word and sentence-level…

Improving Hyper-Relational Knowledge Graph Completion

Donghan Yu, Yiming Yang

cs.LG · 16 Apr 2021 · cited 47× · arXiv ↗ · more like this ·

Different from traditional knowledge graphs (KGs) where facts are represented as entity-relation-entity triplets, hyper-relational KGs (HKGs) allow triplets to be associated with additional relation-entity pairs (a.k.a qualifiers) to convey more complex information. How to effectively and efficiently model the triplet-qualifier relationship for prediction tasks such as HKG completion is an open challenge for research. This paper proposes to improve the best-performing method in HKG completion, namely STARE, by introducing two novel revisions: (1) Replacing the computation-heavy graph neural network module with light-weight entity/relation embedding processing techniques for efficiency improvement without sacrificing effectiveness; (2) Adding a qualifier-oriented auxiliary training task for boosting the prediction power of our approach on HKG completion. The proposed approach…

Multi-site User Behavior Modeling and Its Application in Video Recommendation

Chunfeng Yang, Huan Yan, Donghan Yu +2 more

Information Systems · 28 Jul 2017 · cited 35× · DOI ↗ · more like this ·

As online video service continues to grow in popularity, video content providers compete hard for more eyeball engagement. Some users visit multiple video sites to enjoy videos of their interest while some visit exclusively one site. However, due to the isolation of data, mining and exploiting user behaviors in multiple video websites remain unexplored so far. In this work, we try to model user preferences in six popular video websites with user viewing records obtained from a large ISP in China. The empirical study shows that users exhibit both consistent cross-site interests as well as site-specific interests. To represent this dichotomous pattern of user preferences, we propose a generative model of Multi-site Probabilistic Factorization (MPF) to capture both the cross-site as well as site-specific preferences. Besides, we discuss the design principle of our model by analyzing the…

Multi-Site User Behavior Modeling and Its Application in Video Recommendation

Huan Yan, Chunfeng Yang, Donghan Yu +3 more

Information Systems · IEEE Transactions on Knowledge and Data Engineering · 12 Jul 2019 · cited 23× · DOI ↗ · more like this ·

Item Recommendation for Word-of-Mouth Scenario in Social E-Commerce

Chen Gao, Chao Huang, Donghan Yu +4 more

Information Systems · IEEE Transactions on Knowledge and Data Engineering · 20 Aug 2020 · cited 16× · DOI ↗ · more like this ·

Social commerce, which is different from traditional e-commerce where people purchase products via initiative searching or recommendations from the platform, transforms a social community into an inclusive place to do business by enabling people to share products with their friends. A user ( sharer ), can share a link of a product to their social-connected friends ( receiver ). Once a receiver purchases the product, the sharer can earn commission provided by the platform. To promote sales, the platform can also assist sharers by providing product candidates which are more likely to be purchased during the social sharing. We define this task of generating sharing suggestions as item recommendation for word-of-mouth scenario, and to the best of our knowledge, this is a new task that has never been explored. In this article, we propose a TriM (short for Tri ad…

Knowledge Embedding Based Graph Convolutional Network

Donghan Yu, Yiming Yang, Ruohong Zhang +1 more

cs.LG · June 2020 · cited 13× · arXiv ↗ · more like this ·

Recently, a considerable literature has grown up around the theme of Graph Convolutional Network (GCN). How to effectively leverage the rich structural information in complex graphs, such as knowledge graphs with heterogeneous types of entities and relations, is a primary open challenge in the field. Most GCN methods are either restricted to graphs with a homogeneous type of edges (e.g., citation links only), or focusing on representation learning for nodes only instead of jointly propagating and updating the embeddings of both nodes and edges for target-driven objectives. This paper addresses these limitations by proposing a novel framework, namely the Knowledge Embedding based Graph Convolutional Network (KE-GCN), which combines the power of GCNs in graph-based belief propagation and the strengths of advanced knowledge embedding (a.k.a. knowledge graph embedding) methods, and goes…

Retrieval-Enhanced Generative Model for Large-Scale Knowledge Graph Completion

Donghan Yu, Yiming Yang

Artificial Intelligence · 18 Jul 2023 · cited 10× · DOI ↗ · more like this ·

The task of knowledge graph completion (KGC) is of great importance. To achieve scalability when dealing with large-scale knowledge graphs, recent works formulate KGC as a sequence-to-sequence process, where the incomplete triplet (input) and the missing entity (output) are both verbalized as text sequences. However, inference with these methods relies solely on the model parameters for implicit reasoning and neglects the use of KG itself, which limits the performance since the model lacks the capacity to memorize a vast number of triplets. To tackle this issue, we introduce ReSKGC, a Retrieval-enhanced Seq2seq KGC model, which selects semantically relevant triplets from the KG and uses them as evidence to guide output generation with explicit reasoning. Our method has demonstrated state-of-the-art performance on benchmark datasets Wikidata5M and WikiKG90Mv2, which contain about 5M and…

Long-tailed Extreme Multi-label Text Classification by the Retrieval of Generated Pseudo Label Descriptions

Ruohong Zhang, Yau-Shian Wang, Yiming Yang +3 more

Artificial Intelligence · 2023 · cited 7× · DOI ↗ · more like this ·

Extreme Multi-label Text Classification (XMTC) has been a tough challenge in machine learning research and applications due to the sheer sizes of the label spaces and the severe data scarcity problem associated with the long tail of rare labels in highly skewed distributions. This paper addresses the challenge of tail label prediction by leveraging the power of dense neural retrieval model in mapping input documents (as queries) to relevant label descriptions. To further enhance the quality of label descriptions, we propose to generate pseudo label descriptions from a trained bag-of-words (BoW) classifier, which demonstrates better classification performance under severe scarce data conditions.The proposed approach achieves the state-of-the-art (SOTA) performance of overall label prediction on XMTC benchmark datasets and especially outperforms the SOTA models in the tail label…

Graph-Revised Convolutional Network

Donghan Yu, Ruohong Zhang, Zhengbao Jiang +2 more

cs.LG · November 2019 · cited 7× · arXiv ↗ · more like this ·

Graph Convolutional Networks (GCNs) have received increasing attention in the machine learning community for effectively leveraging both the content features of nodes and the linkage patterns across graphs in various applications. As real-world graphs are often incomplete and noisy, treating them as ground-truth information, which is a common practice in most GCNs, unavoidably leads to sub-optimal solutions. Existing efforts for addressing this problem either involve an over-parameterized model which is difficult to scale, or simply re-weight observed edges without dealing with the missing-edge issue. This paper proposes a novel framework called Graph-Revised Convolutional Network (GRCN), which avoids both extremes. Specifically, a GCN-based graph revision module is introduced for predicting missing edges and revising edge weights w.r.t. downstream tasks via joint optimization. A…

A realistic cancer risk assessment of inorganic arsenic

Donghan Yu

Sociology and Political Science · Journal of Environmental Science and Health Part A · August 1998 · cited 6× · DOI ↗ · more like this ·

Abstract The study presents a realistic evaluation of the skin cancer risk associated with inorganic arsenic exposure through drinking water ingestion. The study uses a probabilistic risk assessment, which is recognized as an important tool in the characterization of health hazards associated with environmental toxin exposure. Compared to the approach with point values, this approach considers the uncertainty of its input values and evaluates realistic estimates of the health risks. Hence, the method provides useful information to the risk manager needing to make a wise decision regarding cancer risk associated with inorganic arsenic. Results have shown that the current level of inorganic arsenic in drinking water exceeds the current cancer risk criterion (10‐6) for the majority of general population. Hence, the study suggests that measures to reduce arsenic levels in water supplies…

Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

Microsoft, :, Abdelrahman Abouelenin +73 more

cs.CL · 7 Mar 2025 · cited 5× · arXiv ↗ · more like this ·

We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-quality web and synthetic data, significantly outperforming recent open-source models of similar size and matching the performance of models twice its size on math and coding tasks requiring complex reasoning. This achievement is driven by a carefully curated synthetic data recipe emphasizing high-quality math and coding datasets. Compared to its predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of 200K tokens to better support multilingual applications, as well as group query attention for more efficient long-sequence generation. Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model. Its novel modality extension approach…

Long-tailed Extreme Multi-label Text Classification with Generated Pseudo Label Descriptions

Ruohong Zhang, Yau-Shian Wang, Yiming Yang +3 more

cs.LG · 2 Apr 2022 · cited 3× · arXiv ↗ · more like this ·

Extreme Multi-label Text Classification (XMTC) has been a tough challenge in machine learning research and applications due to the sheer sizes of the label spaces and the severe data scarce problem associated with the long tail of rare labels in highly skewed distributions. This paper addresses the challenge of tail label prediction by proposing a novel approach, which combines the effectiveness of a trained bag-of-words (BoW) classifier in generating informative label descriptions under severe data scarce conditions, and the power of neural embedding based retrieval models in mapping input documents (as queries) to relevant label descriptions. The proposed approach achieves state-of-the-art performance on XMTC benchmark datasets and significantly outperforms the best methods so far in the tail label prediction. We also provide a theoretical analysis for relating the BoW and neural…

Multi-LoRA Composition for Image Generation

Ming Zhong, Yelong Shen, Shuohang Wang +6 more

cs.CV · February 2024 · cited 2× · arXiv ↗ · more like this ·

Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated images. Nonetheless, existing methods face challenges in effectively composing multiple LoRAs, especially as the number of LoRAs to be integrated grows, thus hindering the creation of complex imagery. In this paper, we study multi-LoRA composition through a decoding-centric perspective. We present two training-free methods: LoRA Switch, which alternates between different LoRAs at each denoising step, and LoRA Composite, which simultaneously incorporates all LoRAs to guide more cohesive image synthesis. To evaluate the proposed approaches, we establish ComposLoRA, a new comprehensive testbed as part of this research. It features a diverse range of LoRA categories with 480 composition sets. Utilizing an evaluation…

CompleQA: Benchmarking the Impacts of Knowledge Graph Completion Methods on Question Answering

Donghan Yu, 裕二池谷, Chenyan Xiong +1 more

Artificial Intelligence · 2023 · cited 1× · DOI ↗ · more like this ·

How much success in Knowledge Graph Completion (KGC) would translate into the performance enhancement in downstream tasks is an important question that has not been studied in depth. In this paper, we introduce a novel benchmark, namely CompleQA, to comprehensively assess the influence of representative KGC methods on Knowledge Graph Question Answering (KGQA), one of the most important downstream applications. This benchmark includes a knowledge graph with 3 million triplets across 5 distinct domains, coupled with over 5000 question-answering pairs and a completion dataset that is well-aligned with these questions. Our evaluation of four well-known KGC methods in combination with two state-of-the-art KGQA systems shows that effective KGC can significantly mitigate the impact of knowledge graph incompleteness on question-answering performance. Surprisingly, we also find that the…

Correlation-aware Unsupervised Change-point Detection via Graph Neural Networks

Ruohong Zhang, Yu Hao, Donghan Yu +3 more

cs.LG · April 2020 · cited 1× · arXiv ↗ · more like this ·

Change-point detection (CPD) aims to detect abrupt changes over time series data. Intuitively, effective CPD over multivariate time series should require explicit modeling of the dependencies across input variables. However, existing CPD methods either ignore the dependency structures entirely or rely on the (unrealistic) assumption that the correlation structures are static over time. In this paper, we propose a Correlation-aware Dynamics Model for CPD, which explicitly models the correlation structure and dynamics of variables by incorporating graph neural networks into an encoder-decoder framework. Extensive experiments on synthetic and real-world datasets demonstrate the advantageous performance of the proposed model on CPD tasks over strong baselines, as well as its ability to classify the change-points as correlation changes or independent changes. Keywords: Multivariate Time…

Learning to Reason as Action Abstractions with Scalable Mid-Training RL

Shenao Zhang, Donghan Yu, Yihao Feng +4 more

cs.LG · September 2025 · cited 0× · arXiv ↗ · more like this ·

Large language models excel with reinforcement learning (RL), but fully unlocking this potential requires a mid-training stage. An effective mid-training phase should identify a compact set of useful actions and enable fast selection among them through online RL. We formalize this intuition by presenting the first theoretical result on how mid-training shapes post-training: it characterizes an action subspace that minimizes both the value approximation error from pruning and the RL error during subsequent planning. Our analysis reveals two key determinants of mid-training effectiveness: pruning efficiency, which shapes the prior of the initial RL policy, and its impact on RL convergence, which governs the extent to which that policy can be improved via online interactions. These results suggest that mid-training is most effective when the decision space is compact and the effective…

Apple Intelligence Foundation Language Models: Tech Report 2025

Ethan Li, Anders Boesen Lindbo Larsen, Chen Zhang +395 more

cs.LG · July 2025 · cited 0× · arXiv ↗ · more like this ·

We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model optimized for Apple silicon through architectural innovations such as KV-cache sharing and 2-bit quantization-aware training; and ii a scalable server model built on a novel Parallel-Track Mixture-of-Experts PT-MoE transformer that combines track parallelism, mixture-of-experts sparse computation, and interleaved global-local attention to deliver high quality with competitive cost on Apple's Private Cloud Compute platform. Both models are trained on large-scale multilingual and multimodal datasets sourced via responsible web crawling, licensed corpora, and high-quality synthetic data, then further refined with supervised fine-tuning and reinforcement learning on a new asynchronous platform. The resulting models…

Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation

Siru Ouyang, Shuohang Wang, Minhao Jiang +4 more

cs.CL · 14 Oct 2024 · cited 0× · arXiv ↗ · more like this ·

Speculative decoding stands as a pivotal technique to expedite inference in autoregressive (large) language models. This method employs a smaller draft model to speculate a block of tokens, which the target model then evaluates for acceptance. Despite a wealth of studies aimed at increasing the efficiency of speculative decoding, the influence of generation configurations on the decoding process remains poorly understood, especially concerning decoding temperatures. This paper delves into the effects of decoding temperatures on speculative decoding's efficacy. Beginning with knowledge distillation (KD), we first highlight the challenge of decoding at higher temperatures, and demonstrate KD in a consistent temperature setting could be a remedy. We also investigate the effects of out-of-domain testing sets with out-of-range temperatures. Building upon these findings, we take an initial…

Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning

Yifang Chen, Shuohang Wang, Ziyi Yang +6 more

cs.LG · 9 Jul 2024 · cited 0× · arXiv ↗ · more like this ·

Reinforcement learning with human feedback (RLHF), as a widely adopted approach in current large language model pipelines, is \textit{bottlenecked by the size of human preference data}. While traditional methods rely on offline preference dataset constructions, recent approaches have shifted towards online settings, where a learner uses a small amount of labeled seed data and a large pool of unlabeled prompts to iteratively construct new preference data through self-generated responses and high-quality reward/preference feedback. However, most current online algorithms still focus on preference labeling during policy model updating with given feedback oracles, which incurs significant expert query costs. \textit{We are the first to explore cost-effective proxy reward oracles construction strategies for further labeling preferences or rewards with extremely limited labeled data and…

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

Shenao Zhang, Donghan Yu, Hiteshi Sharma +6 more

cs.LG · May 2024 · cited 0× · arXiv ↗ · more like this ·

Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere to human intentions. Unlike offline alignment with a fixed dataset, online feedback collection from humans or AI on model generations typically leads to more capable reward models and better-aligned LLMs through an iterative process. However, achieving a globally accurate reward model requires systematic exploration to generate diverse responses that span the vast space of natural language. Random sampling from standard reward-maximizing LLMs alone is insufficient to fulfill this requirement. To address this issue, we propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. By solving the inner-level problem with the…

A probabilistic assessment of skin cancer risk from inorganic arsenic

Donghan Yu

Sociology and Political Science · International Journal of Environmental Studies · December 1999 · cited 0× · DOI ↗ · more like this ·

The study presents a probabilistic assessment of the skin cancer risk associated ‐with inorganic arsenic exposure through drinking water ingestion. This study can provide information to determine whether a drinking water standard associated with inorganic arsenic in the United States is health‐protective. The study uses a probabilistic risk assessment, which is recognized as an important tool in the characterization of health hazards associated with environmental toxin exposure. Compared to the current approach, which produces conservative estimates, this approach considers the uncertainty of its input values and evaluates realistic estimates of the health risks. Hence, the method provides useful information to the risk manager needing to make a wise decision regarding cancer risk associated with inorganic arsenic. Based on the probabilistic skin cancer assessment of inorganic arsenic…