arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Yahan Yu

22 papers · 2022–2026

2022: 22023: 42024: 32025: 82026: 5
2022peak 8 in 20252026

Frequent co-authors

Papers

showing 1–22 of 22 RSS

MM-LLMs: Recent Advances in MultiModal Large Language Models

Duzhen Zhang, Yahan Yu, Jiahua Dong +4 more

cs.CL · · cited 479× · arXiv ↗ · more like this ·

In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies. The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks. In this paper, we provide a comprehensive survey aimed at facilitating further research of MM-LLMs. Initially, we outline general design formulations for model architecture and training pipeline. Subsequently, we introduce a taxonomy encompassing 126 MM-LLMs, each characterized by its specific formulations. Furthermore, we review the performance of selected MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs. Finally, we explore promising directions for MM-LLMs while concurrently maintaining…

Continual Named Entity Recognition without Catastrophic Forgetting

Duzhen Zhang, Wei Cong, Jiahua Dong +4 more

cs.CL · · cited 30× · arXiv ↗ · more like this ·

Continual Named Entity Recognition (CNER) is a burgeoning area, which involves updating an existing model by incorporating new entity types sequentially. Nevertheless, continual learning approaches are often severely afflicted by catastrophic forgetting. This issue is intensified in CNER due to the consolidation of old entity types from previous steps into the non-entity type at each step, leading to what is known as the semantic shift problem of the non-entity type. In this paper, we introduce a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones, thereby more effectively mitigating the problem of catastrophic forgetting. Additionally, we develop a confidence-based pseudo-labeling for the non-entity type, \emph{i.e.,} predicting entity types using the old model to handle the semantic shift of…

Decomposing Logits Distillation for Incremental Named Entity Recognition

Duzhen Zhang, Yahan Yu, Feilong Chen +1 more

Artificial Intelligence · · cited 16× · DOI ↗ · more like this ·

Incremental Named Entity Recognition (INER) aims to continually train a model with new data, recognizing emerging entity types without forgetting previously learned ones. Prior INER methods have shown that Logits Distillation (LD), which involves preserving predicted logits via knowledge distillation, effectively alleviates this challenging issue. In this paper, we discover that a predicted logit can be decomposed into two terms that measure the likelihood of an input token belonging to a specific entity type or not. However, the traditional LD only preserves the sum of these two terms without considering the change in each component. To explicitly constrain each term, we propose a novel Decomposing Logits Distillation (DLD) method, enhancing the model's ability to retain old knowledge and mitigate catastrophic forgetting. Moreover, DLD is model-agnostic and easy to implement. Extensive…

Stroke Extraction of Chinese Character Based on Deep Structure Deformable Image Registration

Meng Li, Yahan Yu, Yi Yang +2 more

cs.CV · · cited 9× · arXiv ↗ · more like this ·

Stroke extraction of Chinese characters plays an important role in the field of character recognition and generation. The most existing character stroke extraction methods focus on image morphological features. These methods usually lead to errors of cross strokes extraction and stroke matching due to rarely using stroke semantics and prior information. In this paper, we propose a deep learning-based character stroke extraction method that takes semantic features and prior information of strokes into consideration. This method consists of three parts: image registration-based stroke registration that establishes the rough registration of the reference strokes and the target as prior information; image semantic segmentation-based stroke segmentation that preliminarily separates target strokes into seven categories; and high-precision extraction of single strokes. In the stroke…

A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering

Yu Li, Bojie Hu, Fengshuo Zhang +4 more

Computer Vision and Pattern Recognition · · cited 7× · DOI ↗ · more like this ·

Recent studies have pointed out that many well-developed Visual Question Answering (VQA) systems suffer from bias problem. Despite the remarkable performance gained on In-Distribution (ID) datasets, the VQA model might merely capture the superficial correlation from question to answer rather than showing real reasoning abilities. Therefore, when switching to Out-of-Distribution (OOD) dataset, whose test distribution is unknown or even reversed with the training set, significant drops might be demonstrated.Although efforts have been devoted to easing the negative bias effect brought by language prior and analysing its inherent cause, they are still limited by the following two aspects. First, most current debiasing methods achieve promising OOD generalization ability with a major sacrifice of the ID performance. Second, existing researches are restricted by exploiting comprehensive…

GHAN: Graph-Based Hierarchical Aggregation Network for Text-Video Retrieval

Yahan Yu, Bojie Hu, Yu Li

Computer Vision and Pattern Recognition · · cited 7× · DOI ↗ · more like this ·

Text-video retrieval focuses on two aspects: cross-modality interaction and video-language encoding. Currently, the mainstream approach is to train a joint embedding space for multimodal interactions. However, there are structural and semantic differences between text and video, making this approach challenging for fine-grained understanding. In order to solve this, we propose an end-to-end graph-based hierarchical aggregation network for text-video retrieval according to the hierarchy possessed by text and video. We design a token-level weighted network to refine intra-modality representations and construct a graph-based message passing attention network for global-local alignment across modality. We conduct experiments on the public datasets MSR-VTT-9K, MSR-VTT-7K and MSVD, and achieve Recall@1 of 73.0%, 65.6%, and 64.0% , which is 25.7%, 16.5%, and 14.2% better than the current…

Flexible Weight Tuning and Weight Fusion Strategies for Continual Named Entity Recognition

Yahan Yu, Duzhen Zhang, Xiuyi Chen +1 more

Artificial Intelligence · · cited 5× · DOI ↗ · more like this ·

Continual Named Entity Recognition (CNER) is dedicated to sequentially learning new entity types while mitigating catastrophic forgetting of old entity types.Traditional CNER approaches commonly employ knowledge distillation to retain old knowledge within the current model.However, because only the representations of old and new models are constrained to be consistent, the reliance solely on distillation in existing methods still suffers from catastrophic forgetting.To further alleviate the forgetting issue of old entity types, this paper introduces flexible Weight Tuning (WT) and Weight Fusion (WF) strategies for CNER.The WT strategy, applied at each training step, employs a learning rate schedule on the parameters of the current model.After learning the current task, the WF strategy dynamically integrates knowledge from both the current and previous models for inference.Notably, these…

When Large Language Models Meet Speech: A Survey on Integration Approaches

Zhengdong Yang, Shuichiro Shimizu, Yahan Yu +1 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

Recent advancements in large language models (LLMs) have spurred interest in expanding their application beyond text-based tasks. A large number of studies have explored integrating other modalities with LLMs, notably speech modality, which is naturally related to text. This paper surveys the integration of speech with LLMs, categorizing the methodologies into three primary approaches: text-based, latent-representation-based, and audio-token-based integration. We also demonstrate how these methods are applied across various speech-related applications and highlight the challenges in this field to offer inspiration for

Anti-Money Laundering Risk Identification of Financial Institutions based on Aspect-Level Graph Neural Networks

Yahan Yu, Yixuan Xu, Jian Wang +2 more

Sociology and Political Science · · cited 3× · DOI ↗ · more like this ·

The contemporary financial industry is a highly information-based industry. The digital system can establish a complete information system around various attributes and behaviors of bank accounts. In the core business system, most of this information is constantly changing and recorded in real time. Therefore, we can achieve the goal of monitoring the money laundering risk of the account by analyzing the relevant element data and specific characteristics of the account. The risk assessment and customer classification indicator system for accounts is composed of four basic elements: customer characteristics, location, business development and industry conditions. Account money laundering risk indicators are composed of various basic elements and their risk sub-items. We propose an aspect-based (aspect-level) graph convolutional neural network, starting from different perspectives, to…

SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models

Zhen Wan, Chao-Han Huck Yang, Yahan Yu +8 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind…

Progressive LoRA for Multimodal Continual Instruction Tuning

Yahan Yu, Duzhen Zhang, Yong Ren +3 more

Signal Processing · · cited 1× · DOI ↗ · more like this ·

Multimodal Continual Instruction Tuning (MCIT) empowers Multimodal Large Language Models (MLLMs) to adapt to everevolving requirements without continuous costly retraining.However, MCIT faces challenges in mitigating Catastrophic Forgetting (CF) and enhancing Knowledge Transfer (KT).Existing works combine Mixture-of-Expert (MoE) and LoRA to address these.However, using a fixed number of shared LoRA blocks across tasks can lead to the overwriting of acquired knowledge, making MLLMs harder to handle CF and KT.Therefore, we propose the Progressive LoRA framework (ProgLoRA), which contains a progressive LoRA pool and trains a new LoRA block for each incremental task to reduce knowledge interference.Specifically, ProgLoRA has two key mechanisms: task-aware allocation for effectively leveraging acquired knowledge at current task and task recall for realigning the model with learned…

APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models

Tian Lan, Han Zhang, Shanshan Wang +3 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Large language models are increasingly used in social interactions, where appropriate responses require recognizing implicit cues, inferring mental states, identifying conflicts, and considering potential consequences. However, existing benchmarks often focus on specific social abilities, constrained task formats, or particular interaction settings, offering limited coverage of social awareness across diverse contexts. To address this gap, we introduce APRiL, an open-ended benchmark for Chinese social awareness with instance-specific rubrics. APRiL contains 2,196 Social Awareness Evaluation Instances across six categories and two complementary tasks: Social Scenario Analysis evaluates models' understanding of social cues, mental states, and conflict structures, while Socially Aware Response assesses whether they can translate this understanding into appropriate, natural responses.…

Type-Balanced Contextual Learning for Incremental Named Entity Recognition

Duzhen Zhang, Yahan Yu, Xiuyi Chen +2 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Incremental Named Entity Recognition (INER) stands as a pivotal task in information extraction, emphasizing the successive identification of new entity types within unstructured text. Faced with the continuous influx of entity types, INER grapples with two significant challenges: the widespread issue of catastrophic forgetting and the unique shift issue of the non-entity type semantics. While pseudo-labeling-based INER methods have proven effective in addressing these challenges, a previously overlooked issue arises: the biased context problem. Our analysis shows that, in new sentences, the contextual associations of tokens representing old entity types exhibit a significantly stronger bias towards new entity types compared to their contexts in old sentences. This tendency intensifies the degradation of old knowledge while promoting the overfitting of new knowledge. To solve this biased…

Frequency-Domain Subspace Reuse and Integration for Multimodal Continual Instruction Tuning

Yahan Yu, Chenhui Chu

Signal Processing · Journal of Natural Language Processing · · cited 0× · DOI ↗ · more like this ·

Multimodal continual instruction tuning (MCIT) typically relies on attaching task-specific LoRA modules to solve catastrophic forgetting, yet sequentially growing LoRA parameters often suffer from capacity allocation and inter-task interference as tasks accumulate. Recent studies extended LoRA to the frequency domain to improve capacity allocation, but ignored task-specific subspaces that can be reused and integrated across tasks. As a result, subspaces remain redundant and underutilized in long and complex task sequences. To address these issues, we propose a Continual FreQuency LoRA (CFQ-LoRA). Specifically, we introduce a lightweight policy to reuse the similar LoRA subspace, improving scalability while maintaining strong retention under long task sequences. We also adopt input-dependent weights to aggregate and integrate existing task-specific LoRA subspaces for task-agnostic…

Learning Deliberately, Acting Intuitively: Unlocking Test-Time Reasoning in Multimodal LLMs

Yahan Yu, Yuyang Dong, Masafumi Oyamada

cs.CV · · cited 0× · arXiv ↗ · more like this ·

Reasoning is essential for large language models (LLMs), especially in complex tasks such as mathematical problem solving. However, multimodal reasoning still faces challenges in modality alignment and training scalability, as many existing methods rely on additional annotations or complex rule-based rewards. To address these issues, we propose the Deliberate-to-Intuitive reasoning framework (D2I), which improves the understanding and reasoning abilities of multimodal LLMs (MLLMs) without extra annotations or complex rewards. During training, D2I uses deliberate reasoning strategies supervised only by rule-based format rewards to enhance modality alignment. During inference, it shifts to intuitive reasoning by removing these explicit strategies, allowing the model to implicitly apply the acquired abilities in its responses. D2I outperforms baselines on both in-domain and out-of-domain…

Enhancing Multimodal Continual Instruction Tuning with BranchLoRA

Duzhen Zhang, Yong Ren, Zhong-Zhi Li +5 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Multimodal Continual Instruction Tuning (MCIT) aims to finetune Multimodal Large Language Models (MLLMs) to continually align with human intent across sequential tasks. Existing approaches often rely on the Mixture-of-Experts (MoE) LoRA framework to preserve previous instruction alignments. However, these methods are prone to Catastrophic Forgetting (CF), as they aggregate all LoRA blocks via simple summation, which compromises performance over time. In this paper, we identify a critical parameter inefficiency in the MoELoRA framework within the MCIT context. Based on this insight, we propose BranchLoRA, an asymmetric framework to enhance both efficiency and performance. To mitigate CF, we introduce a flexible tuning-freezing mechanism within BranchLoRA, enabling branches to specialize in intra-task knowledge while fostering inter-task collaboration. Moreover, we incrementally…

Semantic-Retention Attack for Continual Named Entity Recognition

Yahan Yu, Zhengdong Yang, Fei Cheng +1 more

Artificial Intelligence · IEEE Transactions on Audio Speech and Language Processing · · cited 0× · DOI ↗ · more like this ·

Continual Named Entity Recognition (CNER) aims to learn new entity types while preventing the catastrophic forgetting of previously learned types, expanding the flexibility of NER. However, the robustness of CNER models, which are typically evaluated through adversarial attacks, hasn't been fully investigated due to two crucial challenges. Firstly, most works only focus on attacking NER models, in which the model is designed for a fixed dataset without considering the dynamic nature of real-world scenarios. These methods are not tailored to the challenges of the continual learning setting. Secondly, current textual attacks discretely change target words, which are widely used in sentence-level or document-level tasks. But in the word-level CNER task, even changes in one character may lead to a shift in the true label, making it difficult to reliably evaluate the results of the attack.…

Progressive Multimodal Alignment for Continual Instruction Tuning

Duzhen Zhang, Yahan Yu, Qiaoyi Su +2 more

cs.CV · · arXiv ↗ · more like this ·

Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as…

Revisiting Anthropomorphic Reflection Markers in Large Language Model Reasoning

Yahan Yu, Noa Nakanishi, Fei Cheng

cs.CL · · arXiv ↗ · more like this ·

Large Language Models (LLMs) often produce explicit reflective traces during complex reasoning, accompanied by anthropomorphic markers such as wait, hmm, and alternatively. Although these markers are commonly used as visible indicators of reflection, their mechanisms remain unclear, which leaves the risk of overthinking associated with redundant and repetitive reflection markers. In this work, we revisit anthropomorphic reflection markers, examining their necessity for reasoning and role in the reflection. We suppress these markers through prompt-level and token-level interventions, and analyze their effects on task performance across four benchmarks and two model scales. Our results show that anthropomorphic markers are not uniformly necessary for reasoning performance: suppressing them can preserve or improve performance in several settings, especially under larger sampling budgets.…

Guiding Perception-Reasoning Closer to Human in Blind Image Quality Assessment

Yuan Li, Yahan Yu, Youyuan Lin +3 more

cs.CV · · arXiv ↗ · more like this ·

Humans assess image quality through a perception-reasoning cascade, integrating sensory cues with implicit reasoning to form self-consistent judgments. In this work, we investigate how a model can acquire both human-like and self-consistent reasoning capability for blind image quality assessment (BIQA). We first collect human evaluation data that capture several aspects of human perception-reasoning pipeline. Then, we adopt reinforcement learning, using human annotations as reward signals to guide the model toward human-like perception and reasoning. To enable the model to internalize self-consistent reasoning capability, we design a reward that drives the model to infer the image quality purely from self-generated descriptions. Empirically, our approach achieves score prediction performance comparable to state-of-the-art BIQA systems under general metrics, including Pearson and…

MedKGent: A Large Language Model Agent Framework for Constructing Temporally Evolving Medical Knowledge Graph

Duzhen Zhang, Zixiao Wang, Zhong-Zhi Li +10 more

cs.CL · · arXiv ↗ · more like this ·

The rapid expansion of medical literature challenges the scalable structuring of domain knowledge. Knowledge Graphs (KGs) offer a solution, yet current construction methods lack generalizability and ignore the temporal dynamics of evolving knowledge. To address this, we introduce MedKGent, a Large Language Model (LLM) agent framework for building temporally evolving medical KGs. Using over 10 million PubMed abstracts from 1975 to 2023, MedKGent incrementally constructs a KG daily via two specialized agents. The Extractor Agent identifies knowledge triples and assigns confidence scores, while the Constructor Agent integrates these triples into a temporal graph, reinforcing recurring knowledge and resolving conflicts. The resulting KG contains 156,275 entities and 2,971,384 triples, making it, to our knowledge, the largest LLM-derived medical KG to date. Automated and expert assessments…

Federated Incremental Named Entity Recognition

Duzhen Zhang, Yahan Yu, Chenxing Li +2 more

cs.CL · · arXiv ↗ · more like this ·

Federated Named Entity Recognition (FNER) boosts model training within each local client by aggregating the model updates of decentralized local clients, without sharing their private data. However, existing FNER methods assume fixed entity types and local clients in advance, leading to their ineffectiveness in practical applications. In a more realistic scenario, local clients receive new entity types continuously, while new local clients collecting novel data may irregularly join the global FNER training. This challenging setup, referred to here as Federated Incremental NER, renders the global model suffering from heterogeneous forgetting of old entity types from both intra-client and inter-client perspectives. To overcome these challenges, we propose a Local-Global Forgetting Defense (LGFD) model. Specifically, to address intra-client forgetting, we develop a structural knowledge…