arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Wenhao Yu

263 papers · 2009–2026

2009: 12010: 02011: 02012: 02013: 02014: 22015: 12016: 32017: 72018: 72019: 92020: 162021: 152022: 312023: 432024: 422025: 502026: 36
2009peak 50 in 20252026

Frequent co-authors

Papers

showing 1–25 of 263 RSS

StarCoder: may the source be with you!

Raymond Li, Loubna Ben Allal, Yangtian Zi +64 more

cs.CL · · cited 1,336× · arXiv ↗ · more like this ·

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python…

StarCoder 2 and The Stack v2: The Next Generation

Anton Lozhkov, Raymond Li, Loubna Ben Allal +63 more

cs.SE · · cited 758× · arXiv ↗ · more like this ·

The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model…

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim +30 more

cs.SE · · cited 701× · arXiv ↗ · more like this ·

Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs.To assess how well LLMs can solve challenging and practical tasks via…

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

Hongliang He, Wenlin Yao, Kaixin Ma +5 more

cs.CL · · cited 472× · arXiv ↗ · more like this ·

The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in creating advanced web agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we establish a new benchmark by compiling real-world tasks from 15 popular websites and introduce an automatic evaluation protocol leveraging multimodal understanding abilities of GPT-4V to evaluate open-ended web agents. We show that WebVoyager achieves a 59.1% task success rate on…

Language to Rewards for Robotic Skill Synthesis

Wenhao Yu, Nimrod Gileadi, Chuyuan Fu +17 more

cs.RO · · cited 455× · arXiv ↗ · more like this ·

Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters…

Generate rather than Retrieve: Large Language Models are Strong Context Generators

Wenhao Yu, Dan Iter, Shuohang Wang +6 more

cs.CL · · cited 448× · arXiv ↗ · more like this ·

Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, resulting in the generated…

Preparing for the Unknown: Learning a Universal Policy with Online System Identification

Wenhao Yu, Jie Tan, C. Karen Liu +1 more

cs.LG · · cited 375× · arXiv ↗ · more like this ·

We present a new method of learning control policies that successfully operate under unknown dynamic models. We create such policies by leveraging a large number of training examples that are generated using a physical simulator. Our system is made of two components: a Universal Policy (UP) and a function for Online System Identification (OSI). We describe our control policy as universal because it is trained over a wide array of dynamic models. These variations in the dynamic model may include differences in mass and inertia of the robots' components, variable friction coefficients, or unknown mass of an object to be manipulated. By training the Universal Policy with this variation, the control policy is prepared for a wider array of possible conditions when executed in an unknown environment. The second part of our system uses the recent state and action history of the system to…

Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models

Wenhao Yu, Hongming Zhang, Xiaoman Pan +3 more

cs.CL · · cited 248× · arXiv ↗ · more like this ·

Retrieval-augmented language models (RALMs) represent a substantial advancement in the capabilities of large language models, notably in reducing factual hallucination by leveraging external knowledge sources. However, the reliability of the retrieved information is not always guaranteed. The retrieval of irrelevant data can lead to misguided responses, and potentially causing the model to overlook its inherent knowledge, even when it possesses adequate information to address the query. Moreover, standard RALMs often struggle to assess whether they possess adequate knowledge, both intrinsic and retrieved, to provide an accurate answer. In situations where knowledge is lacking, these systems should ideally respond with "unknown" when the answer is unattainable. In response to these challenges, we introduces Chain-of-Noting (CoN), a novel approach aimed at improving the robustness of…

Few-Shot Graph Learning for Molecular Property Prediction

Zhichun Guo, Chuxu Zhang, Wenhao Yu +4 more

cs.LG · · cited 227× · arXiv ↗ · more like this ·

The recent success of graph neural networks has significantly boosted molecular property prediction, advancing activities such as drug discovery. The existing deep neural network methods usually require large training dataset for each property, impairing their performances in cases (especially for new molecular properties) with a limited amount of experimental data, which are common in real situations. To this end, we propose Meta-MGNN, a novel model for few-shot molecular property prediction. Meta-MGNN applies molecular graph neural network to learn molecular representation and builds a meta-learning framework for model optimization. To exploit unlabeled molecular information and address task heterogeneity of different molecular properties, Meta-MGNN further incorporates molecular structure, attribute based self-supervised modules and self-attentive task weights into the former…

A Survey of Deep Learning for Mathematical Reasoning

Pan Lu, Liang Qiu, Wenhao Yu +2 more

cs.AI · · cited 209× · arXiv ↗ · more like this ·

Mathematical reasoning is a fundamental aspect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems has garnered significant interest in the fields of machine learning and natural language processing. For example, mathematics serves as a testbed for aspects of reasoning that are challenging for powerful deep learning models, driving new algorithmic and modeling advances. On the other hand, recent advances in large-scale neural language models have opened up new benchmarks and opportunities to use deep learning for mathematical reasoning. In this survey paper, we review the key tasks, datasets, and methods at the intersection of mathematical reasoning and deep learning over the past decade. We also evaluate…

Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions

Alejandro Escontrela, Xue Bin Peng, Wenhao Yu +4 more

cs.AI · · cited 199× · arXiv ↗ · more like this ·

Training a high-dimensional simulated agent with an under-specified reward function often leads the agent to learn physically infeasible strategies that are ineffective when deployed in the real world. To mitigate these unnatural behaviors, reinforcement learning practitioners often utilize complex reward functions that encourage physically plausible behaviors. However, a tedious labor-intensive tuning process is often required to create hand-designed rewards which might not easily generalize across platforms and tasks. We propose substituting complex reward functions with "style rewards" learned from a dataset of motion capture demonstrations. A learned style reward can be combined with an arbitrary task reward to train policies that perform tasks using naturalistic strategies. These natural strategies can also facilitate transfer to the real world. We build upon Adversarial Motion…

Sim-to-Real Transfer for Biped Locomotion

Wenhao Yu, Visak CV Kumar, Greg Turk +1 more

cs.RO · · cited 139× · arXiv ↗ · more like this ·

We present a new approach for transfer of dynamic robot control policies such as biped locomotion from simulation to real hardware. Key to our approach is to perform system identification of the model parameters μ of the hardware (e.g. friction, center-of-mass) in two distinct stages, before policy learning (pre-sysID) and after policy learning (post-sysID). Pre-sysID begins by collecting trajectories from the physical hardware based on a set of generic motion sequences. Because the trajectories may not be related to the task of interest, presysID does not attempt to accurately identify the true value of μ, but only to approximate the range of μ to guide the policy learning. Next, a Projected Universal Policy (PUP) is created by simultaneously training a network that projects μ to a low-dimensional latent variable η and a family of policies that are conditioned on η. The second round of…

Learning from Counterfactual Links for Link Prediction

Tong Zhao, Gang Liu, Daheng Wang +2 more

cs.LG · · cited 136× · arXiv ↗ · more like this ·

Learning to predict missing links is important for many graph-based applications. Existing methods were designed to learn the association between observed graph structure and existence of link between a pair of nodes. However, the causal relationship between the two variables was largely ignored for learning to predict links on a graph. In this work, we visit this factor by asking a counterfactual question: "would the link still exist if the graph structure became different from observation?" Its answer, counterfactual links, will be able to augment the graph data for representation learning. To create these links, we employ causal models that consider the information (i.e., learned representations) of node pairs as context, global graph structural properties as treatment, and link existence as outcome. We propose a novel data augmentation-based link prediction method that creates…

A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods

Zhihan Zhang, Wenhao Yu, Mengxia Yu +2 more

cs.CL · · cited 123× · arXiv ↗ · more like this ·

Multi-task learning (MTL) has become increasingly popular in natural language processing (NLP) because it improves the performance of related tasks by exploiting their commonalities and differences. Nevertheless, it is still not understood very well how multi-task learning can be implemented based on the relatedness of training tasks. In this survey, we review recent advances of multi-task learning methods in NLP, with the aim of summarizing them into two general multi-task training methods based on their task relatedness: (i) joint training and (ii) multi-step training. We present examples in various NLP downstream applications, summarize the task relationships and discuss future directions of this promising topic.

KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering

Donghan Yu, Chenguang Zhu, Yuwei Fang +6 more

cs.CL · · cited 116× · arXiv ↗ · more like this ·

Current Open-Domain Question Answering (ODQA) model paradigm often contains a retrieving module and a reading module. Given an input question, the reading module predicts the answer from the relevant passages which are retrieved by the retriever. The recent proposed Fusion-in-Decoder (FiD), which is built on top of the pretrained generative model T5, achieves the state-of-the-art performance in the reading module. Although being effective, it remains constrained by inefficient attention on all retrieved passages which contain a lot of noise. In this work, we propose a novel method KG-FiD, which filters noisy passages by leveraging the structural relationship among the retrieved passages with a knowledge graph. We initiate the passage node embedding from the FiD encoder and then use graph neural network (GNN) to update the representation for reranking. To improve the efficiency, we build…

Improving Language Models via Plug-and-Play Retrieval Feedback

Wenhao Yu, Zhihan Zhang, Zhenwen Liang +2 more

cs.CL · · cited 115× · arXiv ↗ · more like this ·

Large language models (LLMs) exhibit remarkable performance across various NLP tasks. However, they often generate incorrect or hallucinated information, which hinders their practical applicability in real-world scenarios. Human feedback has been shown to effectively enhance the factuality and quality of generated content, addressing some of these limitations. However, this approach is resource-intensive, involving manual input and supervision, which can be time-consuming and expensive. Moreover, it cannot be provided during inference, further limiting its practical utility in dynamic and interactive applications. In this paper, we introduce ReFeed, a novel pipeline designed to enhance LLMs by providing automatic retrieval feedback in a plug-and-play framework without the need for expensive fine-tuning. ReFeed first generates initial outputs, then utilizes a retrieval model to acquire…

GeoGPT: An assistant for understanding and processing geospatial tasks

Yifan Zhang, Wei Cheng, Zhengting He +1 more

Geography, Planning and Development · International Journal of Applied Earth Observation and Geoinformation · · cited 113× · DOI ↗ · more like this ·

Decision-makers in GIS often need to combine multiple spatial algorithms and operations to solve geospatial tasks. While professionals can understand and solve these tasks by using GIS tools sequentially, developing workflows for various tasks can be inefficient, as even slight differences in tasks require corresponding adjustments in the workflow. Recently, large language models (e.g., ChatGPT) presented a strong performance in semantic understanding and reasoning. Especially, AutoGPT can further extend the capabilities of large language models (LLMs) by automatically reasoning and calling externally defined tools. Inspired by these studies, we attempt to increase the efficiency of developing workflows for handling geoprocessing tasks by integrating the semantic understanding ability inherent in LLMs with mature tools within the GIS community. Specifically, we develop a new framework…

Policy Transfer with Strategy Optimization

Wenhao Yu, C. Karen Liu, Greg Turk

cs.LG · · cited 87× · arXiv ↗ · more like this ·

Computer simulation provides an automatic and safe way for training robotic control policies to achieve complex tasks such as locomotion. However, a policy trained in simulation usually does not transfer directly to the real hardware due to the differences between the two environments. Transfer learning using domain randomization is a promising approach, but it usually assumes that the target environment is close to the distribution of the training environments, thus relying heavily on accurate system identification. In this paper, we present a different approach that leverages domain randomization for transferring control policies to unknown environments. The key idea that, instead of learning a single policy in the simulation, we simultaneously learn a family of policies that exhibit different behaviors. When tested in the target environment, we directly search for the best policy in…

Dict-BERT: Enhancing Language Model Pre-training with Dictionary

Wenhao Yu, Chenguang Zhu, Yuwei Fang +5 more

cs.CL · · cited 77× · arXiv ↗ · more like this ·

Pre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on large-scale corpora. Since PLMs capture word semantics in different contexts, the quality of word representations highly depends on word frequency, which usually follows a heavy-tailed distributions in the pre-training corpus. Therefore, the embeddings of rare words on the tail are usually poorly optimized. In this work, we focus on enhancing language model pre-training by leveraging definitions of the rare words in dictionaries (e.g., Wiktionary). To incorporate a rare word definition as a part of input, we fetch its definition from the dictionary and append it to the end of the input text sequence. In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word and sentence-level…

CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments

Adarsh Jagan Sathyamoorthy, Kasun Weerakoon, Mohamed Elnoor +6 more

cs.RO · · cited 74× · arXiv ↗ · more like this ·

We present ConVOI, a novel method for autonomous robot navigation in real-world indoor and outdoor environments using Vision Language Models (VLMs). We employ VLMs in two ways: first, we leverage their zero-shot image classification capability to identify the context or scenario (e.g., indoor corridor, outdoor terrain, crosswalk, etc) of the robot's surroundings, and formulate context-based navigation behaviors as simple text prompts (e.g. ``stay on the pavement"). Second, we utilize their state-of-the-art semantic understanding and logical reasoning capabilities to compute a suitable trajectory given the identified context. To this end, we propose a novel multi-modal visual marking approach to annotate the obstacle-free regions in the RGB image used as input to the VLM with numbers, by correlating it with a local occupancy map of the environment. The marked numbers ground image…

SayTap: Language to Quadrupedal Locomotion

Yujin Tang, Wenhao Yu, Jie Tan +3 more

cs.RO · · cited 71× · arXiv ↗ · more like this ·

Large language models (LLMs) have demonstrated the potential to perform high-level planning. Yet, it remains a challenge for LLMs to comprehend low-level commands, such as joint angle targets or motor torques. This paper proposes an approach to use foot contact patterns as an interface that bridges human commands in natural language and a locomotion controller that outputs these low-level commands. This results in an interactive system for quadrupedal robots that allows the users to craft diverse locomotion behaviors flexibly. We contribute an LLM prompt design, a reward function, and a method to expose the controller to the feasible distribution of contact patterns. The results are a controller capable of achieving diverse locomotion patterns that can be transferred to real robot hardware. Compared with other design choices, the proposed approach enjoys more than 50% success rate in…

GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT

Yifan Zhang, Cheng Wei, Shangyou Wu +2 more

cs.CL · · cited 69× · arXiv ↗ · more like this ·

Decision-makers in GIS need to combine a series of spatial algorithms and operations to solve geospatial tasks. For example, in the task of facility siting, the Buffer tool is usually first used to locate areas close or away from some specific entities; then, the Intersect or Erase tool is used to select candidate areas satisfied multiple requirements. Though professionals can easily understand and solve these geospatial tasks by sequentially utilizing relevant tools, it is difficult for non-professionals to handle these problems. Recently, Generative Pre-trained Transformer (e.g., ChatGPT) presents strong performance in semantic understanding and reasoning. Especially, AutoGPT can further extend the capabilities of large language models (LLMs) by automatically reasoning and calling externally defined tools. Inspired by these studies, we attempt to lower the threshold of…

Spatial co-location pattern mining of facility points-of-interest improved by network neighborhood and distance decay effects

Wenhao Yu, Tinghua Ai, Yakun He +1 more

Signal Processing · International Journal of Geographical Information Systems · · cited 66× · DOI ↗ · more like this ·

The aim of mining spatial co-location patterns is to find the corresponding subsets of spatial features that have strong spatial correlation in the real world. This is an important technology for the extraction and comprehension of implicit knowledge in large spatial databases. However, existing methods of co-location mining consider events as taking place in a homogeneous and isotropic context in Euclidean space, whereas the physical movement in an urban space is usually constrained by a road network. Furthermore, previous works do not take the ‘distance decay effect’ of spatial interactions into account, which may reduce the effectiveness of the result. Here we propose an improved spatial co-location pattern mining method, including the network-constrained neighborhood and addition of a distance-decay function, to find the spatial dependence between network phenomena (e.g. urban…

MapGPT: an autonomous framework for mapping by integrating large language model and cartographic tools

Yifan Zhang, Zhengting He, Jingxuan Li +3 more

Geography, Planning and Development · Cartography and Geographic Information Science · · cited 65× · DOI ↗ · more like this ·

The mapping process generally involves intricate operations, such as symbol design, layout design, and text annotation, demanding a high level of professional expertise. The high requirement for map producers hinders the promotion and widespread adoption of mapping. Consequently, researchers are concentrating on techniques to automate and enhance the intelligence of the mapping process. For example, some studies attempt to train deep learning models for mapping, including methods like map style transfer. However, these approaches typically treat the entire map as a global input and generate a new map as output, lacking the flexibility to consider and control detailed elements within a map. Therefore, in this paper, we propose a large language model-based intelligent mapping framework, termed MapGPT, which can be used for mapping by considering the map as an integration of various map…

Discovering Frequent Movement Paths From Taxi Trajectory Data Using Spatially Embedded Networks and Association Rules

Wenhao Yu

Transportation · IEEE Transactions on Intelligent Transportation Systems · · cited 59× · DOI ↗ · more like this ·

In view of the complex traffic flows, spatial interactions within a city exhibit the properties of dynamics, connectivity, and repeatability. This paper aims at mining spatial-temporal movement patterns from massive taxi trajectory data for discovering the inherent travel flow information within the urban system. Similar to the role of ocean circulation in a marine system, identifying the frequent paths and cycles of the travel flows within a city would be critical for understanding the principles behind the travel flow surfaces. Thus, we propose a multi-level method for the discovery of movement paths by incorporating the techniques of network analysis and association rules. Specifically, the proposed method begins by constructing a directed network on the subdivision of the study region, in which the node with geolocation represents the corresponding cell and the edge with weight…