arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Ruochen Zhao

32 papers · 2017–2026

2017: 12018: 02019: 02020: 02021: 12022: 02023: 112024: 112025: 12026: 7
2017peak 11 in 20232026

Frequent co-authors

Papers

showing 1–25 of 32 RSS

Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework

Ruochen Zhao, Xingxuan Li, Shafiq Joty +2 more

cs.CL · · cited 228× · arXiv ↗ · more like this ·

As large language models (LLMs) have become the norm in NLP, demonstrating good performance in generation and reasoning tasks, one of its most fatal disadvantages is the lack of factual correctness. Generating unfactual texts not only leads to lower performances but also degrades the trust and validity of their applications. Chain-of-Thought (CoT) prompting improves trust and model performance on complex reasoning tasks by generating interpretable reasoning chains, but still suffers from factuality concerns in knowledge-intensive tasks. In this paper, we propose the Verify-and-Edit framework for CoT prompting, which seeks to increase prediction factuality by post-editing reasoning chains according to external knowledge. Building on top of GPT-3, our framework lead to accuracy improvements in multiple open-domain question-answering tasks.

Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources

Xingxuan Li, Ruochen Zhao, Yew Ken Chia +4 more

cs.CL · · cited 194× · arXiv ↗ · more like this ·

We present chain-of-knowledge (CoK), a novel framework that augments large language models (LLMs) by dynamically incorporating grounding information from heterogeneous sources. It results in more factual rationales and reduced hallucination in generation. Specifically, CoK consists of three stages: reasoning preparation, dynamic knowledge adapting, and answer consolidation. Given a knowledge-intensive question, CoK first prepares several preliminary rationales and answers while identifying the relevant knowledge domains. If there is no majority consensus among the answers from samples, CoK corrects the rationales step by step by adapting knowledge from the identified domains. These corrected rationales can plausibly serve as a better foundation for the final answer consolidation. Unlike prior studies that primarily use unstructured data, CoK also leverages structured knowledge sources…

Retrieving Multimodal Information for Augmented Generation: A Survey

Ruochen Zhao, Hailin Chen, Weishi Wang +8 more

cs.CL · · cited 159× · arXiv ↗ · more like this ·

As Large Language Models (LLMs) become popular, there emerged an important trend of using multimodality to augment the LLMs' generation ability, which enables LLMs to better interact with the world. However, there lacks a unified perception of at which stage and how to incorporate different modalities. In this survey, we review methods that assist and augment generative models by retrieving multimodal knowledge, whose formats range from images, codes, tables, graphs, to audio. Such methods offer a promising solution to important concerns such as factuality, reasoning, interpretability, and robustness. By providing an in-depth review, this survey is expected to provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs.

ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?

Hailin Chen, Fangkai Jiao, Xingxuan Li +5 more

cs.CL · · cited 36× · arXiv ↗ · more like this ·

Upon its release in late 2022, ChatGPT has brought a seismic shift in the entire landscape of AI, both in research and commerce. Through instruction-tuning a large language model (LLM) with supervised fine-tuning and reinforcement learning from human feedback, it showed that a model could answer human questions and follow instructions on a broad panel of tasks. Following this success, interests in LLMs have intensified, with new LLMs flourishing at frequent interval across academia and industry, including many start-ups focused on LLMs. While closed-source LLMs (e.g., OpenAI's GPT, Anthropic's Claude) generally outperform their open-source counterparts, the progress on the latter has been rapid with claims of achieving parity or even better on certain tasks. This has crucial implications not only on research but also on business. In this work, on the first anniversary of ChatGPT, we…

Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?

Chengwei Qin, Qian Li, Ruochen Zhao +1 more

cs.CL · · cited 9× · arXiv ↗ · more like this ·

Prompt tuning (PT) which only tunes the embeddings of an additional sequence of tokens per task, keeping the pre-trained language model (PLM) frozen, has shown remarkable performance in few-shot learning. Despite this, PT has been shown to rely heavily on good initialization of the prompt embeddings. In this work, we study meta prompt tuning (MPT) to systematically explore how meta-learning can help improve (if it can) cross-task generalization in PT through learning to initialize the prompt embeddings from other relevant tasks. We empirically analyze a representative set of meta learning algorithms in a wide range of adaptation settings with different source/target task configurations on a large set of few-shot tasks. With extensive experiments and analysis, we demonstrate the effectiveness of MPT. We find the improvement to be significant particularly on classification tasks. For…

Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges

Bosheng Ding, Chengwei Qin, Ruochen Zhao +7 more

cs.CL · · cited 6× · arXiv ↗ · more like this ·

In the rapidly evolving field of large language models (LLMs), data augmentation (DA) has emerged as a pivotal technique for enhancing model performance by diversifying training examples without the need for additional data collection. This survey explores the transformative impact of LLMs on DA, particularly addressing the unique challenges and opportunities they present in the context of natural language processing (NLP) and beyond. From both data and learning perspectives, we examine various strategies that utilize LLMs for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for diverse forms of further training. Additionally, this paper highlights the primary open challenges faced in this domain, ranging from controllable data augmentation to multi-modal data augmentation. This survey highlights a paradigm shift introduced by LLMs…

Randomized Smoothing with Masked Inference for Adversarially Robust Text Classifications

Han Cheol Moon, Shafiq Joty, Ruochen Zhao +2 more

cs.CL · · cited 6× · arXiv ↗ · more like this ·

Large-scale pre-trained language models have shown outstanding performance in a variety of NLP tasks. However, they are also known to be significantly brittle against specifically crafted adversarial examples, leading to increasing interest in probing the adversarial robustness of NLP systems. We introduce RSMI, a novel two-stage framework that combines randomized smoothing (RS) with masked inference (MI) to improve the adversarial robustness of NLP systems. RS transforms a classifier into a smoothed classifier to obtain robust representations, whereas MI forces a model to exploit the surrounding context of a masked token in an input sequence. RSMI improves adversarial robustness by 2 to 3 times over existing state-of-the-art methods on benchmark datasets. We also perform in-depth qualitative analysis to validate the effectiveness of the different stages of RSMI and probe the impact of…

Expanding the clinical application of OPM-MEG using an effective automatic suppression method for the dental brace metal artifact

Ruonan Wang, Kaiwen Fu, Ruochen Zhao +5 more

Signal Processing · NeuroImage · · cited 5× · DOI ↗ · more like this ·

Optically pumped magnetometer magnetoencephalography (OPM-MEG) holds significant promise for clinical functional brain imaging due to its superior spatiotemporal resolution. However, effectively suppressing metallic artifacts, particularly from devices such as orthodontic braces and vagal nerve stimulators remains a major challenge, hindering the wider clinical application of wearable OPM-MEG devices. A comprehensive analysis of metal artifact characteristics from time, frequency, and time-frequency perspectives was conducted for the first time using an OPM-MEG device in clinical medicine. This study focused on patients with metal orthodontics, examining the modulation of metal artifacts by breath and head movement, the incomplete regular sub-Gaussian distribution, and the high absolute power ratio in the 0.5-8 Hz band. The existing metal artifact suppression algorithms applied to…

Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions

Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia +3 more

cs.CL · · cited 4× · arXiv ↗ · more like this ·

As LLMs continuously evolve, there is an urgent need for a reliable evaluation method that delivers trustworthy results promptly. Currently, static benchmarks suffer from inflexibility and unreliability, leading users to prefer human voting platforms like Chatbot Arena. However, human evaluations require significant manual effort. To address this, we propose the Auto-Arena, an innovative framework that automates the entire evaluation process using LLM-powered agents. Firstly, an LLM examiner generates questions. Then, two LLM candidates engage in a multi-round peer battle based on individual questions, aiming at revealing their true performance differences. Finally, a committee of LLM judges collaboratively discusses and decides the winner, reducing bias and enhancing fairness. During the peer battles, we observe intriguing scenarios where the LLM candidates display competitive…

Beyond Corrected Memory: Execution Consistency in Multi-Agent Systems

Zhe Yu, Zixuan Wang, Peidong Wang +3 more

cs.AI · · cited 3× · arXiv ↗ · more like this ·

Shared memory coordinates agents' actions, but correct records do not establish that those actions satisfy task requirements. Memory governance and failure diagnosis regulate or inspect recorded information; they do not by themselves establish whether it is sufficient to judge task duties. We define execution consistency through duties governing state use, information handoffs, and final-state agreement, with explicit evidence conditions for judging fulfillment. Our core claim is that identical retained records can correspond to compliant and violating executions under the same task rule. Controlled removal of evidence such as receipt, action dependence, or response validity leaves 82.4% of opposite-label pairs indistinguishable; restoration separates 97.9% of the merged pairs. Natural-log annotations identify the defined violations in actual executions. However, existing logs do not…

A Comprehensive Survey of Contamination Detection Methods in Large Language Models

Mathieu Ravaut, Bosheng Ding, Fangkai Jiao +6 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

With the rise of Large Language Models (LLMs) in recent years, abundant new opportunities are emerging, but also new challenges, among which contamination is quickly becoming critical. Business applications and fundraising in Artificial Intelligence (AI) have reached a scale at which a few percentage points gained on popular question-answering benchmarks could translate into dozens of millions of dollars, placing high pressure on model integrity. At the same time, it is becoming harder and harder to keep track of the data that LLMs have seen; if not impossible with closed-source models like GPT-4 and Claude-3 not divulging any information on the training set. As a result, contamination becomes a major issue: LLMs' performance may not be reliable anymore, as the high performance may be at least partly due to their previous exposure to the data. This limitation jeopardizes real capability…

Flexible template matching for observational study design

Ruochen Zhao, Bo Lü

Statistics and Probability · Statistics in Medicine · · cited 3× · DOI ↗ · more like this ·

Matching is a popular design for inferring causal effect with observational data. Unlike model-based approaches, it is a nonparametric method to group treated and control subjects with similar characteristics together, hence to re-create a randomization-like scenario. The application of matched design for real world data may be limited by: (1) the causal estimand of interest; (2) the sample size of different treatment arms. We propose a flexible design of matching, based on the idea of template matching, to overcome these challenges. It first identifies the template group which is representative of the target population, then match subjects from the original data to this template group and make inference. We provide theoretical justification on how it unbiasedly estimates the average treatment effect using matched pairs and the average treatment effect on the treated when the treatment…

A multiple imputation approach in enhancing causal inference for overall survival in randomized controlled trials with crossover

Ruochen Zhao, Junjing Lin, Junjing Lin +5 more

Statistics and Probability · Journal of Biopharmaceutical Statistics · · cited 2× · DOI ↗ · more like this ·

Crossover or treatment-switching in randomized controlled trials presents notable challenges not only in the development and approval of new drugs but also poses a complex issue in their reimbursement, especially in oncology. When the investigational treatment is superior to control, crossover from control to investigational treatment upon disease progression or for other reasons will likely cause the underestimation of treatment benefit. Rank Preserving Structural Failure Time (RPSFT) and Two-Stage Estimation (TSE) methods are commonly employed to adjust for treatment switching by estimating counterfactual survival times. However, these methods may induce informative censoring by adjusting censoring times for switchers while leaving those for non-switchers unchanged. Existing approaches such as re-censoring or inverse probability of censoring weighting (IPCW) are often used alongside…

Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

Xingxuan Li, Weiwen Xu, Ruochen Zhao +3 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

State-of-the-art large language models (LLMs) exhibit impressive problem-solving capabilities but may struggle with complex reasoning and factual correctness. Existing methods harness the strengths of chain-of-thought and retrieval-augmented generation (RAG) to decompose a complex problem into simpler steps and apply retrieval to improve factual correctness. These methods work well on straightforward reasoning tasks but often falter on challenging tasks such as competitive programming and mathematics, due to frequent reasoning errors and irrelevant knowledge retrieval. To address this, we introduce Critic-guided planning with Retrieval-augmentation, CR-Planner, a novel framework that leverages fine-tuned critic models to guide both reasoning and retrieval processes through planning. CR-Planner solves a problem by iteratively selecting and executing sub-goals. Initially, it identifies…

Lifelong Event Detection with Embedding Space Separation and Compaction

Chengwei Qin, Ruirui Chen, Ruochen Zhao +2 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

To mitigate forgetting, existing lifelong event detection methods typically maintain a memory module and replay the stored memory data during the learning of a new task. However, the simple combination of memory data and new-task samples can still result in substantial forgetting of previously acquired knowledge, which may occur due to the potential overlap between the feature distribution of new data and the previously learned embedding space. Moreover, the model suffers from overfitting on the few memory samples rather than effectively remembering learned patterns. To address the challenges of forgetting and overfitting, we propose a novel method based on embedding space separation and compaction. Our method alleviates forgetting of previously learned tasks by forcing the feature distribution of new data away from the previous embedding space. It also mitigates overfitting by a memory…

Exploration of College Students’ Psychological Education Teaching Mode Based on Cognitive Behavioral Theory and Its Practice in the Internet Era

Xiaomi Xiang, Ruochen Zhao

Education · Applied Mathematics and Nonlinear Sciences · · cited 1× · DOI ↗ · more like this ·

Abstract The Internet era brings to the teaching of mental health courses not only the sharing of the best quality teaching resources, but also will profoundly impact the traditional teaching mode of mental health education classes. Based on Cognitive Behavioral Theory (CBT), this study constructed a hybrid online + offline psychological education teaching mode in the Internet era and designed a controlled experiment between it and the traditional teaching mode to analyze the significant difference through the independent samples t-test method to realize the comparative analysis of the teaching mode. Additionally, the influence factors for evaluating the effectiveness of the teaching method were explored using the CART categorical regression tree model. In Group I, which adopted the blended teaching mode, there were significant differences in the male students’ mental health factors in…

Targeting COVID-19 and Human Resources for Health News Information Extraction: Algorithm Development and Validation (Preprint)

Mathieu Ravaut, Ruochen Zhao, Duy Phung +6 more

Artificial Intelligence · · cited 1× · DOI ↗ · more like this ·

<sec> <title>BACKGROUND</title> Global pandemics like COVID-19 put a high amount of strain on health care systems and health workers worldwide. These crises generate a vast amount of news information published online across the globe. This extensive corpus of articles has the potential to provide valuable insights into the nature of ongoing events and guide interventions and policies. However, the sheer volume of information is beyond the capacity of human experts to process and analyze effectively. </sec> <sec> <title>OBJECTIVE</title> The aim of this study was to explore how natural language processing (NLP) can be leveraged to build a system that allows for quick analysis of a high volume of news articles. Along with this, the objective was to create a workflow comprising human-computer symbiosis to derive valuable insights to support health workforce strategic policy dialogue…

Why Do Agent Teams Fail Across Boundaries? A Failure Taxonomy for Agent-to-Agent Systems with Private States

Sarath Shekkizhar, Prathyusha Jwalapuram, Zixuan Ke +12 more

cs.MA · · cited 0× · arXiv ↗ · more like this ·

Agents are increasingly interacting with other agents that act for different principals (people or organizations on whose behalf they work), such as a buyer's personal agent negotiating with an enterprise's seller agent. In such agent-to-agent (A2A) scenarios, each agent has its own environment and holds information and authority that the other cannot see. Moreover, the interests of each agent may be aligned (cooperative), partly aligned (mixed-motive), or opposed (competitive). Existing multi-agent system failure studies assume one orchestrator, shared workspace, or single principal. We ask how A2A systems fail when these assumptions are removed. We study more than 2,500 recorded runs of A2A systems built from different agent harnesses and models, spanning 488 scenarios across eight families (legal, CRM, security, commerce, etc.). From these traces, we derive $A^2$-Failures, a taxonomy…

EMMA: Event-Driven Multimodal Memory System with Multi-Agent Retrieval

Ruochen Zhao, Jiazheng Kang, Gezi Li +2 more

cs.CV · · cited 0× · arXiv ↗ · more like this ·

Long video understanding requires reasoning over hours of content that far exceed the context window of multimodal large language models (MLLMs). Existing approaches compress video into fixed-length clips, yet events are fragmented and unrelated content is merged, yielding noisy retrieval. To overcome this challenge, we propose EMMA, an Event-driven Multimodal Memory system with Multi-Agent retrieval, which organizes memory around semantic events in three stages: (1) Event segmentation: a dual-stream detector adaptively locates event boundaries by fusing visual semantic drift with spoken transitions, rather than fixed windows. (2) Memory construction: built on the detected events, a three-level hierarchy of raw video, clip-level memory, and event-level memory stores content at progressively higher semantic abstraction. (3) Memory retrieval: a multi-agent framework collaboratively…

Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models

Yadong Xi, Rongsheng Zhang, Tangjie Lv +2 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Reinforcement learning with binary correctness rewards trains correctness, not calibrated confidence. The confidence that reasoning models verbalize is systematically overconfident, and the problem is not merely one of scale: verbalized confidence tracks how willing a model is to commit to an answer, not how likely the answer is to be right. Post-hoc rescaling therefore fits one distribution but rarely transfers. We look inside the model instead. On factual question answering, a linear probe on the hidden state between the chain of thought and the answer is substantially better calibrated: its expected calibration error is 5 to 38 times lower than that of the verbalized score across four benchmarks and two model families. However, when used to pick among N sampled answers, that same probe nearly ties majority voting yet falls far short of the oracle. Internal states answer "how certain…

AgREE: Agentic Reasoning for Knowledge Graph Completion on Emerging Entities

Ruochen Zhao, Simone Conia, Eric Peng +2 more

cs.AI · · cited 0× · arXiv ↗ · more like this ·

Open-domain Knowledge Graph Completion (KGC) faces significant challenges in an ever-changing world, especially when considering the continual emergence of new entities in daily news. Existing approaches for KGC mainly rely on pretrained language models' parametric knowledge, pre-constructed queries, or single-step retrieval, typically requiring substantial supervision and training data. Even so, they often fail to capture comprehensive and up-to-date information about unpopular and/or emerging entities. To this end, we introduce Agentic Reasoning for Emerging Entities (AgREE), a novel agent-based framework that combines iterative retrieval actions and multi-step reasoning to dynamically construct rich knowledge graph triplets. Experiments show that, despite requiring zero training efforts, AgREE significantly outperforms existing methods in constructing knowledge graph triplets…

Analysis of Teaching Capacity Building of College Teachers’ Innovation and Entrepreneurship Education under Comprehensive Multiple Fuzzy Evaluation System

Ruochen Zhao, Ruochen Zhao, Xiaomi Xiang +3 more

Education · Applied Mathematics and Nonlinear Sciences · · cited 0× · DOI ↗ · more like this ·

Abstract How can we effectively promote and improve the quality of innovation and entrepreneurship education in colleges and universities? The key lies in enhancing the capacity of teachers to impart innovative entrepreneurial education. The article takes the requirements of innovation and entrepreneurship education and the connotation of teaching ability as the orientation and establishes the evaluation index system of dual innovation education teaching ability of college teachers by combining the CIPP model. The IAHP method and the improved CRITIC method, respectively, calculate the subjective and objective weights of each index in the index system, and then they introduce the game theory idea to amalgamate and assign weights to the index system Based on the gray fuzzy mathematical theory, a fuzzy comprehensive evaluation model of the teaching ability of innovation and…

PromptSum: Parameter-Efficient Controllable Abstractive Summarization

Mathieu Ravaut, Hailin Chen, Ruochen Zhao +3 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Prompt tuning (PT), a parameter-efficient technique that only tunes the additional prompt embeddings while keeping the backbone pre-trained language model (PLM) frozen, has shown promising results in language understanding tasks, especially in low-resource scenarios. However, effective prompt design methods suitable for generation tasks such as summarization are still lacking. At the same time, summarization guided through instructions (discrete prompts) can achieve a desirable double objective of high quality and controllability in summary generation. Towards a goal of strong summarization performance under the triple conditions of parameter-efficiency, data-efficiency, and controllability, we introduce PromptSum, a method combining PT with a multi-task objective and discrete entity prompts for abstractive summarization. Our model achieves competitive ROUGE results on popular…

Explaining Language Models' Predictions with High-Impact Concepts

Ruochen Zhao, Shafiq Joty, Yongjie Wang +1 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

The emergence of large-scale pretrained language models has posed unprecedented challenges in deriving explanations of why the model has made some predictions. Stemmed from the compositional nature of languages, spurious correlations have further undermined the trustworthiness of NLP systems, leading to unreliable model explanations that are merely correlated with the output predictions. To encourage fairness and transparency, there exists an urgent demand for reliable explanations that allow users to consistently understand the model's behavior. In this work, we propose a complete framework for extending concept-based interpretability methods to NLP. Specifically, we propose a post-hoc interpretability method for extracting predictive high-level features (concepts) from the pretrained model's hidden layer activations. We optimize for features whose existence causes the output…