arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Yu Su

756 papers · 1984–2026

1984: 11985: 01986: 01987: 01988: 11989: 01990: 11991: 11992: 01993: 01994: 41995: 41996: 01997: 21998: 01999: 02000: 32001: 52002: 172003: 92004: 72005: 122006: 202007: 252008: 142009: 172010: 182011: 192012: 222013: 192014: 152015: 212016: 202017: 212018: 352019: 382020: 322021: 472022: 452023: 712024: 642025: 742026: 52
1984peak 74 in 20252026

Frequent co-authors

Papers

showing 1–25 of 756 RSS

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Xiang Yue, Yuansheng Ni, Kai Zhang +19 more

cs.CL · · cited 2,676× · arXiv ↗ · more like this ·

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 14 open-source LMMs as well as the proprietary GPT-4V(ision) and Gemini highlights the substantial…

Mind2Web: Towards a Generalist Agent for the Web

Xiang Deng, Yu Gu, Boyuan Zheng +5 more

cs.CL · · cited 1,501× · arXiv ↗ · more like this ·

We introduce Mind2Web, the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. Existing datasets for web agents either use simulated websites or only cover a limited set of websites and tasks, thus not suitable for generalist web agents. With over 2,000 open-ended tasks collected from 137 websites spanning 31 domains and crowdsourced action sequences for the tasks, Mind2Web provides three necessary ingredients for building generalist web agents: 1) diverse domains, websites, and tasks, 2) use of real-world websites instead of simulated and simplified ones, and 3) a broad spectrum of user interaction patterns. Based on Mind2Web, we conduct an initial exploration of using large language models (LLMs) for building generalist web agents. While the raw HTML of real-world websites are often…

AgentBench: Evaluating LLMs as Agents

Xiao Liu, Hao Yu, Hanchen Zhang +19 more

cs.AI · · cited 1,407× · arXiv ↗ · more like this ·

The potential of Large Language Model (LLM) as agents has been widely acknowledged recently. Thus, there is an urgent need to quantitatively \textit{evaluate LLMs as agents} on challenging tasks in interactive environments. We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities. Our extensive test over \num API-based and open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and many OSS competitors that are no larger than 70B. We identify the typical reasons of failures in environments and LLMs, showing that poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles for developing usable LLM agents.…

LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models

Chan Hee Song, Jiaman Wu, Clayton Washington +3 more

cs.AI · · cited 798× · arXiv ↗ · more like this ·

This study focuses on using large language models (LLMs) as a planner for embodied agents that can follow natural language instructions to complete complex tasks in a visually-perceived environment. The high data cost and poor sample efficiency of existing methods hinders the development of versatile agents that are capable of many tasks and can learn new tasks quickly. In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models to do few-shot planning for embodied agents. We further propose a simple but effective way to enhance LLMs with physical grounding to generate and update plans that are grounded in the current environment. Experiments on the ALFRED dataset show that our method can achieve very competitive few-shot performance: Despite using less than 0.5% of paired training data, LLM-Planner achieves competitive performance with recent…

MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

Kai Zhang, Lingbo Mo, Wenhu Chen +2 more

cs.CV · · cited 755× · arXiv ↗ · more like this ·

Text-guided image editing is widely needed in daily life, ranging from personal use to professional applications such as Photoshop. However, existing methods are either zero-shot or trained on an automatically synthesized dataset, which contains a high volume of noise. Thus, they still require lots of manual tuning to produce desirable outcomes in practice. To address this issue, we introduce MagicBrush (https://osu-nlp-group.github.io/MagicBrush/), the first large-scale, manually annotated dataset for instruction-guided real image editing that covers diverse scenarios: single-turn, multi-turn, mask-provided, and mask-free editing. MagicBrush comprises over 10K manually annotated triplets (source image, instruction, target image), which supports trainining large-scale text-guided image editing models. We fine-tune InstructPix2Pix on MagicBrush and show that the new model can produce…

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Boyuan Zheng, Boyu Gou, Jihyung Kil +2 more

cs.IR · · cited 645× · arXiv ↗ · more like this ·

The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering. In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website. We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web. We evaluate on the recent MIND2WEB benchmark. In addition to standard offline evaluation on cached websites, we enable a new online evaluation setting by developing a tool that allows running web agents on live websites. We show that GPT-4V presents a great potential for web agents -- it can successfully complete 51.1 of the tasks on live…

MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

Xiang Yue, Xingwei Qu, Ge Zhang +5 more

cs.CL · · cited 614× · arXiv ↗ · more like this ·

We introduce MAmmoTH, a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset. MathInstruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by us. It presents a unique hybrid of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As a result, the MAmmoTH series substantially outperform existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32%. Remarkably, our MAmmoTH-7B model reaches 33%…

EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction

Qi Liu, Zhenya Huang, Yu Yin +4 more

cs.CY · · cited 521× · arXiv ↗ · more like this ·

For offering proactive services to students in intelligent education, one of the fundamental tasks is predicting their performance (e.g., scores) on future exercises, where it is necessary to track each student's knowledge acquisition during her exercising activities. However, existing approaches can only exploit the exercising records of students, and the problem of extracting rich information existed in the exercise's materials (e.g., knowledge concepts, exercise content) to achieve both precise predictions of student performance and interpretable analysis of knowledge acquisition remains underexplored. In this paper, we present a holistic study of student performance prediction. To directly achieve the primary goal of prediction, we first propose a general Exercise-Enhanced Recurrent Neural Network (EERNN) framework by exploring both student's records and the exercise contents. In…

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

Jian Xie, Kai Zhang, Jiangjie Chen +5 more

cs.CL · · cited 493× · arXiv ↗ · more like this ·

Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not…

HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models

Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu +2 more

cs.CL · · cited 392× · arXiv ↗ · more like this ·

In order to thrive in hostile and ever-changing natural environments, mammalian brains evolved to store large amounts of knowledge about the world and continually integrate new information while avoiding catastrophic forgetting. Despite the impressive accomplishments, large language models (LLMs), even with retrieval-augmented generation (RAG), still struggle to efficiently and effectively integrate a large amount of new experiences after pre-training. In this work, we introduce HippoRAG, a novel retrieval framework inspired by the hippocampal indexing theory of human long-term memory to enable deeper and more efficient knowledge integration over new experiences. HippoRAG synergistically orchestrates LLMs, knowledge graphs, and the Personalized PageRank algorithm to mimic the different roles of neocortex and hippocampus in human memory. We compare HippoRAG with existing RAG methods on…

Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases

Yu Gu, Sue Kase, Michelle Vanni +4 more

cs.CL · · cited 350× · arXiv ↗ · more like this ·

Existing studies on question answering on knowledge bases (KBQA) mainly operate with the standard i.i.d assumption, i.e., training distribution over questions is the same as the test distribution. However, i.i.d may be neither reasonably achievable nor desirable on large-scale KBs because 1) true user distribution is hard to capture and 2) randomly sample training examples from the enormous space would be highly data-inefficient. Instead, we suggest that KBQA models should have three levels of built-in generalization: i.i.d, compositional, and zero-shot. To facilitate the development of KBQA models with stronger generalization, we construct and release a new large-scale, high-quality dataset with 64,331 questions, GrailQA, and provide evaluation settings for all three levels of generalization. In addition, we propose a novel BERT-based KBQA model. The combination of our dataset and…

Hierarchical Ensemble of Global and Local Classifiers for Face Recognition

Yu Su, Shiguang Shan, Xilin Chen +1 more

Computer Vision and Pattern Recognition · · cited 314× · DOI ↗ · more like this ·

In the literature of psychophysics and neurophysiology, many studies have shown that both global and local features are crucial for face representation and recognition. This paper proposes a novel face recognition method which combines both global and local discriminative features. In this method, global features are extracted from whole face images by Fourier transform and local features are extracted from some spatially partitioned image patches by Gabor wavelet transform. After this, multiple classifiers are obtained by applying Fisher Discriminant Analysis on global Fourier features and local patches of Gabor features. All these classifiers are combined to form a hierarchical ensemble by sum rule. We evaluated the proposed method using Face Recognition Grand Challenge (FRGC) experimental protocols and database known as the largest data sets available. Experimental results on FRGC…

ITU-T Recommendation G.729 Annex B: a silence compression scheme for use with G.729 optimized for V.70 digital simultaneous voice and data applications

A. Benyassine, E. Shlomot, Huan-yu Su +3 more

Signal Processing · IEEE Communications Magazine · · cited 308× · DOI ↗ · more like this ·

This article describes Annex B to ITU-T Recommendation G.729. Annex B defines a low-bit-rate silence compression scheme designed and optimized to work in conjunction with both the full version of G.729 and its low-complexity Annex A. To achieve good quality low-bit-rate silence compression, a robust frame-based voice activity detector module is essential to detect inactive voice frames, also called silence or background noise frames. For these detected inactive voice frames, a discontinuous transmission module measures the changes over time of the inactive voice signal characteristics and decides whether a new silence information descriptor frame should be sent to maintain the reproduction quality of the background noise at the receiving end. If such a frame is needed, the spectrum and energy parameters describing the perceptual characteristics of the background noise are efficiently…

Exercise-Enhanced Sequential Modeling for Student Performance Prediction

Yu Su, Qingwen Liu, Qi Liu +6 more

Computer Science Applications · Proceedings of the AAAI Conference on Artificial Intelligence · · cited 254× · DOI ↗ · more like this ·

In online education systems, for offering proactive services to students (e.g., personalized exercise recommendation), a crucial demand is to predict student performance (e.g., scores) on future exercising activities. Existing prediction methods mainly exploit the historical exercising records of students, where each exercise is usually represented as the manually labeled knowledge concepts, and the richer information contained in the text description of exercises is still underexplored. In this paper, we propose a novel Exercise-Enhanced Recurrent Neural Network (EERNN) framework for student performance prediction by taking full advantage of both student exercising records and the text of each exercise. Specifically, for modeling the student exercising process, we first design a bidirectional LSTM to learn each exercise representation from its text description without any expertise and…

RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

Chan Hee Song, Valts Blukis, Jonathan Tremblay +3 more

cs.CV · · cited 200× · arXiv ↗ · more like this ·

Spatial understanding is a crucial capability that enables robots to perceive their surroundings, reason about their environment, and interact with it meaningfully. In modern robotics, these capabilities are increasingly provided by vision-language models. However, these models face significant challenges in spatial reasoning tasks, as their training data are based on general-purpose image datasets that often lack sophisticated spatial understanding. For example, datasets frequently do not capture reference frame comprehension, yet effective spatial reasoning requires understanding whether to reason from ego-, world-, or object-centric perspectives. To address this issue, we introduce RoboSpatial, a large-scale dataset for spatial understanding in robotics. It consists of real indoor and tabletop scenes, captured as 3D scans and egocentric images, and annotated with rich spatial…

Environment Sound Classification Using a Two-Stream CNN Based on Decision-Level Fusion

Yu Su, Ke Zhang, Jingyu Wang +1 more

Signal Processing · Sensors · · cited 197× · DOI ↗ · more like this ·

With the popularity of using deep learning-based models in various categorization problems and their proven robustness compared to conventional methods, a growing number of researchers have exploited such methods in environment sound classification tasks in recent years. However, the performances of existing models use auditory features like log-mel spectrogram (LM) and mel frequency cepstral coefficient (MFCC), or raw waveform to train deep neural networks for environment sound classification (ESC) are unsatisfactory. In this paper, we first propose two combined features to give a more comprehensive representation of environment sounds Then, a fourfour-layer convolutional neural network (CNN) is presented to improve the performance of ESC with the proposed aggregated features. Finally, the CNN trained with different features are fused using the Dempster-Shafer evidence theory to…

Logical Natural Language Generation from Open-Domain Tables

Wenhu Chen, Jianshu Chen, Yu Su +2 more

cs.CL · · cited 187× · arXiv ↗ · more like this ·

Neural natural language generation (NLG) models have recently shown remarkable progress in fluency and coherence. However, existing studies on neural NLG are primarily focused on surface-level realizations with limited emphasis on logical inference, an important aspect of human thinking and language. In this paper, we suggest a new NLG task where a model is tasked with generating natural language statements that can be \emph{logically entailed} by the facts in an open-domain semi-structured table. To facilitate the study of the proposed logical NLG problem, we use the existing TabFact dataset \cite{chen2019tabfact} featured with a wide range of logical/symbolic inferences as our testbed, and propose new automatic metrics to evaluate the fidelity of generation models w.r.t.\ logical inference. The new task poses challenges to the existing monotonic generation frameworks due to the…

Task-Oriented Dialogue as Dataflow Synthesis

Semantic Machines, Jacob Andreas, John Bufe +43 more

cs.CL · · cited 179× · arXiv ↗ · more like this ·

We describe an approach to task-oriented dialogue in which dialogue state is represented as a dataflow graph. A dialogue agent maps each user utterance to a program that extends this graph. Programs include metacomputation operators for reference and revision that reuse dataflow fragments from previous turns. Our graph-based state enables the expression and manipulation of complex user intents, and explicit metacomputation makes these intents easier for learned models to predict. We introduce a new dataset, SMCalFlow, featuring complex dialogues about events, weather, places, and people. Experiments show that dataflow graphs and metacomputation substantially improve representability and predictability in these natural dialogues. Additional experiments on the MultiWOZ dataset show that our dataflow representation enables an otherwise off-the-shelf sequence-to-sequence model to match the…

Fuzzy Cognitive Diagnosis for Modelling Examinee Performance

Qi Liu, Runze Wu, Enhong Chen +4 more

Artificial Intelligence · ACM Transactions on Intelligent Systems and Technology · · cited 171× · DOI ↗ · more like this ·

Recent decades have witnessed the rapid growth of educational data mining (EDM), which aims at automatically extracting valuable information from large repositories of data generated by or related to people’s learning activities in educational settings. One of the key EDM tasks is cognitive modelling with examination data, and cognitive modelling tries to profile examinees by discovering their latent knowledge state and cognitive level (e.g. the proficiency of specific skills). However, to the best of our knowledge, the problem of extracting information from both objective and subjective examination problems to achieve more precise and interpretable cognitive analysis remains underexplored. To this end, we propose a fuzzy cognitive diagnosis framework (FuzzyCDF) for examinees’ cognitive modelling with both objective and subjective problems. Specifically, to handle the partially correct…

Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems

Bang Liu, Xinfeng Li, Jiayi Zhang +45 more

cs.AI · · cited 166× · arXiv ↗ · more like this ·

The advent of large language models (LLMs) has catalyzed a transformative shift in artificial intelligence, paving the way for advanced intelligent agents capable of sophisticated reasoning, robust perception, and versatile action across diverse domains. As these agents increasingly drive AI research and practical applications, their design, evaluation, and continuous improvement present intricate, multifaceted challenges. This book provides a comprehensive overview, framing intelligent agents within modular, brain-inspired architectures that integrate principles from cognitive science, neuroscience, and computational research. We structure our exploration into four interconnected parts. First, we systematically investigate the modular foundation of intelligent agents, systematically mapping their cognitive, perceptual, and operational modules onto analogous human brain functionalities…

Learning Process-consistent Knowledge Tracing

Shuanghong Shen, Qi Liu, Enhong Chen +5 more

Artificial Intelligence · · cited 153× · DOI ↗ · more like this ·

Knowledge tracing (KT), which aims to trace students' changing knowledge state during their learning process, has improved students' learning efficiency in online learning systems. Recently, KT has attracted much research attention due to its critical significance in education. However, most of the existing KT methods pursue high accuracy of student performance prediction but neglect the consistency of students' changing knowledge state with their learning process. In this paper, we explore a new paradigm for the KT task and propose a novel model named Learning Process-consistent Knowledge Tracing (LPKT), which monitors students' knowledge state through directly modeling their learning process. Specifically, we first formalize the basic learning cell as the tuple exercise---answer time---answer. Then, we deeply measure the learning gain as well as its diversity from the difference of…

RCD: Relation Map Driven Cognitive Diagnosis for Intelligent Education Systems

Weibo Gao, Qi Liu, Zhenya Huang +6 more

Artificial Intelligence · · cited 147× · DOI ↗ · more like this ·

Cognitive diagnosis (CD) is a fundamental issue in intelligent educational settings, which aims to discover the mastery levels of students on different knowledge concepts. In general, most previous works consider it as an inter-layer interaction modeling problem, e.g., student-exercise interactions in IRT or student-concept interactions in DINA, while the inner-layer structural relations, such as educational interdependencies among concepts, are still underexplored. Furthermore, there is a lack of comprehensive modeling for the student-exercise-concept hierarchical relations in CD systems. To this end, in this paper, we present a novel Relation map driven Cognitive Diagnosis (RCD) framework, uniformly modeling the interactive and structural relations via a multi-layer student-exercise-concept relation map. Specifically, we first represent students, exercises and concepts as individual…

Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning

Baolin Peng, Xiujun Li, Jianfeng Gao +3 more

cs.CL · · cited 147× · arXiv ↗ · more like this ·

Training a task-completion dialogue agent via reinforcement learning (RL) is costly because it requires many interactions with real users. One common alternative is to use a user simulator. However, a user simulator usually lacks the language complexity of human interlocutors and the biases in its design may tend to degrade the agent. To address these issues, we present Deep Dyna-Q, which to our knowledge is the first deep RL framework that integrates planning for task-completion dialogue policy learning. We incorporate into the dialogue agent a model of the environment, referred to as the world model, to mimic real user response and generate simulated experience. During dialogue policy learning, the world model is constantly updated with real user experience to approach real user behavior, and in turn, the dialogue agent is optimized using both real experience and simulated experience.…

Convolutional Knowledge Tracing: Modeling Individualization in Student Learning Process

Shuanghong Shen, Qi Liu, Enhong Chen +6 more

Artificial Intelligence · · cited 144× · DOI ↗ · more like this ·

With the development of online education systems, a growing number of research works are focusing on Knowledge Tracing (KT), which aims to assess students' changing knowledge state and help them learn knowledge concepts more efficiently. However, only given student learning interactions, most of existing KT methods neglect the individualization of students, i.e., the prior knowledge and learning rates differ from student to student. To this end, in this paper, we propose a novel Convolutional Knowledge Tracing (CKT) method to model individualization in KT. Specifically, for individualized prior knowledge, we measure it from students' historical learning interactions. For individualized learning rates, we design hierarchical convolutional layers to extract them based on continuous learning interactions of students. Extensive experiments demonstrate that CKT could obtain better knowledge…

NeuralCD: A General Framework for Cognitive Diagnosis

Fei Wang, Qi Liu, Enhong Chen +4 more

Information Systems · IEEE Transactions on Knowledge and Data Engineering · · cited 136× · DOI ↗ · more like this ·

Cognitive diagnosis is widely applicable in the scenarios where users’ cognitive states need to be assessed, such as games and clinical measurement. Especially in intelligent education, which has become increasingly popular recent decades, cognitive diagnosis serves as a fundamental module for discovering the proficiency level of students on specific knowledge concepts. Existing approaches usually mine linear interactions of student exercising process by manually designed function (e.g., logistic function). However, the cognitive interactions between students and exercises is a complex process, and excessive simplifications would lead to under fitting and thus get inaccurate diagnostic results. Besides, the manually designed interaction functions are relatively inflexible and limits their extensibility. This consequently causes lack of consideration about useful non-numerical…