arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Yao Mu

158 papers · 2001–2026

2001: 12002: 02003: 02004: 12005: 02006: 02007: 12008: 02009: 02010: 02011: 12012: 02013: 02014: 02015: 12016: 02017: 02018: 02019: 02020: 42021: 72022: 112023: 132024: 212025: 292026: 68
2001peak 68 in 20262026

Frequent co-authors

Papers

showing 1–25 of 158 RSS

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought

Yao Mu, Qinglong Zhang, Mengkang Hu +7 more

cs.RO · · cited 452× · arXiv ↗ · more like this ·

Embodied AI is a crucial frontier in robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments. In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities. To achieve this, we have made the following efforts: (i) We craft a large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate a sequence of sub-goals with the "Chain of Thoughts" mode for effective embodied planning. (ii) We introduce an efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via…

LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

Hao Sha, Yao Mu, Yuxuan Jiang +7 more

cs.RO · · cited 276× · arXiv ↗ · more like this ·

Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability. To address these problems, this work employs Large Language Models (LLMs) as a decision-making component for complex AD scenarios that require human commonsense understanding. We devise cognitive pathways to enable comprehensive reasoning with LLMs, and develop algorithms for translating LLM decisions into actionable driving commands. Through this approach, LLM decisions are seamlessly integrated with low-level controllers by guided parameter matrix adaptation. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination, thanks to the commonsense reasoning…

SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution

Zhixuan Liang, Yao Mu, Hengbo Ma +3 more

cs.RO · · cited 102× · arXiv ↗ · more like this ·

Diffusion models have demonstrated strong potential for robotic trajectory planning. However, generating coherent trajectories from high-level instructions remains challenging, especially for long-range composition tasks requiring multiple sequential skills. We propose SkillDiffuser, an end-to-end hierarchical planning framework integrating interpretable skill learning with conditional diffusion planning to address this problem. At the higher level, the skill abstraction module learns discrete, human-understandable skill representations from visual observations and language instructions. These learned skill embeddings are then used to condition the diffusion model to generate customized latent trajectories aligned with the skills. This allows generating diverse state trajectories that adhere to the learnable skills. By integrating skill learning with conditional trajectory generation…

Chemometric QSAR modeling of acute oral toxicity of Polycyclic Aromatic Hydrocarbons (PAHs) to rat using simple 2D descriptors and interspecies toxicity modeling with mouse

Guohui Sun, Yifan Zhang, Luyu Pei +9 more

Computational Theory and Mathematics · Ecotoxicology and Environmental Safety · · cited 63× · DOI ↗ · more like this ·

The information of the acute oral toxicity for most polycyclic aromatic hydrocarbons (PAHs) in mammals are lacking due to limited experimental resources, leading to a need to develop reliable in silico methods to evaluate the toxicity endpoint. In this study, we developed the quantitative structure-activity relationship (QSAR) models by genetic algorithm (GA) and multiple linear regression (MLR) for the rat acute oral toxicity (LD50) of PAHs following the strict validation principles of QSAR modeling recommended by OECD. The best QSAR model comprised eight simple 2D descriptors with definite physicochemical meaning, which showed that maximum atom-type electrotopological state, van der Waals surface area, mean atomic van der Waals volume, and total number of bonds are main influencing factors for the toxicity endpoint. A true external set (554 compounds) without rat acute oral toxicity…

Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learning

Zhecheng Yuan, Guozheng Ma, Yao Mu +5 more

cs.CV · · cited 44× · arXiv ↗ · more like this ·

One of the key challenges in visual Reinforcement Learning (RL) is to learn policies that can generalize to unseen environments. Recently, data augmentation techniques aiming at enhancing data diversity have demonstrated proven performance in improving the generalization ability of learned policies. However, due to the sensitivity of RL training, naively applying data augmentation, which transforms each pixel in a task-agnostic manner, may suffer from instability and damage the sample efficiency, thus further exacerbating the generalization performance. At the heart of this phenomenon is the diverged action distribution and high-variance value estimation in the face of augmented images. To alleviate this issue, we propose Task-aware Lipschitz Data Augmentation (TLDA) for visual RL, which explicitly identifies the task-correlated pixels with large Lipschitz constants, and only augments…

Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

Baiyu Peng, Jingliang Duan, Jianyu Chen +6 more

cs.LG · · cited 37× · arXiv ↗ · more like this ·

Safety is essential for reinforcement learning (RL) applied in the real world. Adding chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under uncertainty. Existing chance-constrained RL methods like the penalty methods and the Lagrangian methods either exhibit periodic oscillations or learn an over-conservative or unsafe policy. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. We first review the constrained policy optimization process from a feedback control perspective, which regards the penalty weight as the control input and the safe probability as the control output. Based on this, the penalty method is formulated as a proportional controller, and the Lagrangian method is formulated as an integral controller. We then unify them and present a proportional-integral…

Scale-Equivalent Distillation for Semi-Supervised Object Detection

Qiushan Guo, Yao Mu, Jianyu Chen +3 more

cs.CV · · cited 35× · arXiv ↗ · more like this ·

Recent Semi-Supervised Object Detection (SS-OD) methods are mainly based on self-training, i.e., generating hard pseudo-labels by a teacher model on unlabeled data as supervisory signals. Although they achieved certain success, the limited labeled data in semi-supervised learning scales up the challenges of object detection. We analyze the challenges these methods meet with the empirical experiment results. We find that the massive False Negative samples and inferior localization precision lack consideration. Besides, the large variance of object sizes and class imbalance (i.e., the extreme ratio between background and object) hinder the performance of prior arts. Further, we overcome these challenges by introducing a novel approach, Scale-Equivalent Distillation (SED), which is a simple yet effective end-to-end knowledge distillation framework robust to large object size variance and…

XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment

XPolicyLab Community, Tianxing Chen, Yue Chen +67 more

cs.RO · · cited 12× · arXiv ↗ · more like this ·

Robot policy evaluation and deployment remain fragmented by model-specific software dependencies, data representations, and runtime interfaces, so that connecting N policies to M evaluation environments requires O(NM) separate integrations. We present XPolicyLab, a unified standard and open ecosystem that reduces this cost to O(N+M). XPolicyLab specifies common observation, action, and trajectory schemas together with a minimal adapter interface for observation updates, action prediction, batched execution, and episode reset, while a dependency-isolated client/server architecture separates policy inference from environment execution, so that each side retains its native software stack and may run locally or remotely. The ecosystem integrates 42 robot policies and standardizes their installation, debugging, serving, and evaluation workflows. Across these adapters, model-specific code…

Mixed Reinforcement Learning for Efficient Policy Optimization in Stochastic Environments

Yao Mu, Baiyu Peng, Ziqing Gu +5 more

Artificial Intelligence · · cited 10× · DOI ↗ · more like this ·

Reinforcement learning has the potential to control stochastic nonlinear systems in optimal manners successfully. We propose a mixed reinforcement learning (mixed RL) algorithm by simultaneously using dual representations of environmental dynamics to search the optimal policy. The dual representation includes an empirical dynamic model and a set of state-action data. The former can embed the designer's knowledge and reduce the difficulty of learning, and the latter can be used to compensate the model inaccuracy since it reflects the real system dynamics accurately. Such a design has the capability of improving both learning accuracy and training speed. In the mixed RL framework, the additive uncertainty of stochastic model is compensated by using explored state-action data via iterative Bayesian estimator (IBE). The optimal policy is then computed in an iterative way by alternating…

EC<sup>2</sup>: Emergent Communication for Embodied Control

Yao Mu, Shunyu Yao, Mingyu Ding +2 more

Computer Vision and Pattern Recognition · · cited 9× · DOI ↗ · more like this ·

Embodied control requires agents to leverage multimodal pre-training to quickly learn how to act in new environments, where video demonstrations contain visual and motion details needed for low-level perception and control, and language instructions support generalization with abstract, symbolic structures. While recent approaches apply contrastive learning to force alignment between the two modalities, we hypothesize better modeling their complementary differences can lead to more holistic representations for downstream adaption. To this end, we propose Emergent Communication for Embodied Control (EC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), a novel scheme to pre-train video-language representations for few-shot embodied control. The key idea is to learn an unsupervised “language” of videos via emergent communication…

Model-Based Actor-Critic with Chance Constraint for Stochastic System

Baiyu Peng, Yao Mu, Yang Guan +3 more

cs.LG · · cited 9× · arXiv ↗ · more like this ·

Safety is essential for reinforcement learning (RL) applied in real-world situations. Chance constraints are suitable to represent the safety requirements in stochastic systems. Previous chance-constrained RL methods usually have a low convergence rate, or only learn a conservative policy. In this paper, we propose a model-based chance constrained actor-critic (CCAC) algorithm which can efficiently learn a safe and non-conservative policy. Different from existing methods that optimize a conservative lower bound, CCAC directly solves the original chance constrained problems, where the objective function and safe probability is simultaneously optimized with adaptive weights. In order to improve the convergence rate, CCAC utilizes the gradient of dynamic model to accelerate policy optimization. The effectiveness of CCAC is demonstrated by a stochastic car-following task. Experiments…

VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model

Pengying Wu, Yao Mu, Bingxian Wu +4 more

cs.RO · · cited 8× · arXiv ↗ · more like this ·

In the realm of household robotics, the Zero-Shot Object Navigation (ZSON) task empowers agents to adeptly traverse unfamiliar environments and locate objects from novel categories without prior explicit training. This paper introduces VoroNav, a novel semantic exploration framework that proposes the Reduced Voronoi Graph to extract exploratory paths and planning nodes from a semantic map constructed in real time. By harnessing topological and semantic information, VoroNav designs text-based descriptions of paths and images that are readily interpretable by a large language model (LLM). In particular, our approach presents a synergy of path and farsight descriptions to represent the environmental context, enabling LLM to apply commonsense reasoning to ascertain waypoints for navigation. Extensive evaluation on HM3D and HSSD validates VoroNav surpasses existing benchmarks in both success…

AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

Zhixuan Liang, Yao Mu, Mingyu Ding +3 more

cs.LG · · cited 8× · arXiv ↗ · more like this ·

Diffusion models have demonstrated their powerful generative capability in many tasks, with great potential to serve as a paradigm for offline reinforcement learning. However, the quality of the diffusion model is limited by the insufficient diversity of training data, which hinders the performance of planning and the generalizability to new tasks. This paper introduces AdaptDiffuser, an evolutionary planning method with diffusion that can self-evolve to improve the diffusion model hence a better planner, not only for seen tasks but can also adapt to unseen tasks. AdaptDiffuser enables the generation of rich synthetic expert data for goal-conditioned tasks using guidance from reward gradients. It then selects high-quality data via a discriminator to finetune the diffusion model, which improves the generalization ability to unseen tasks. Empirical experiments on two benchmark…

Prototypical Context-Aware Dynamics for Generalization in Visual Control With Model-Based Reinforcement Learning

Jun‐Jie Wang, Qichao Zhang, Yao Mu +6 more

Artificial Intelligence · IEEE Transactions on Industrial Informatics · · cited 7× · DOI ↗ · more like this ·

The latent world model, which efficiently represents high-dimensional observations within a latent space, has shown promise in reinforcement learning-based policies for visual control tasks. Due to a lack of clear environmental context comprehension, its applicability in a variety of contexts with unknown dynamics is constrained. We propose a prototypical context- aware dynamics (ProtoCAD) model to address this issue. This model captures local dynamics using temporally consistent latent contexts and aids generalization in visual control tasks. By grouping prototypes over historical experiences, ProtoCAD collects useful contextual information that improves model-based reinforcement learning dynamics generalization in two ways. First, to guarantee the consistency of prototype assignments for various temporal segments of the same latent trajectory, a temporally consistent prototypes…

Tree-Planner: Efficient Close-loop Task Planning with Large Language Models

Mengkang Hu, Yao Mu, Xinmiao Yu +7 more

cs.CL · · cited 6× · arXiv ↗ · more like this ·

This paper studies close-loop task planning, which refers to the process of generating a sequence of skills (a plan) to accomplish a specific goal while adapting the plan based on real-time observations. Recently, prompting Large Language Models (LLMs) to generate actions iteratively has become a prevalent paradigm due to its superior performance and user-friendliness. However, this paradigm is plagued by two inefficiencies: high token consumption and redundant error correction, both of which hinder its scalability for large-scale testing and applications. To address these issues, we propose Tree-Planner, which reframes task planning with LLMs into three distinct phases: plan sampling, action tree construction, and grounded deciding. Tree-Planner starts by using an LLM to sample a set of potential plans before execution, followed by the aggregation of them to form an action tree.…

RoboFoundry: System-as-Policy Evolution for Self-Learning Embodied Agents

Jingsong Liang, Shuhao Liao, Shizhe Zhang +15 more

cs.RO · · cited 5× · arXiv ↗ · more like this ·

A foundation model should not act in isolation as an embodied agent. Yet, existing methods often optimize individual components of the agent stack, such as memory, context, skills, or action interfaces, rather than treating the supporting system itself as a unified policy. Moreover, interaction alone does not yield self-improvement unless execution experience is converted into persistent, validated system changes. We therefore propose RoboFoundry, the first embodied agentic framework that formulates this process as Self-Evolving System-as-Policy. RoboFoundry diagnoses capability gaps in decision-making and memory management, converts execution traces into validated task-specific system updates, and promotes recurring improvements to the general system. Evolution operates over two complementary surfaces: a context system that manages active internal context and persistent file-system…

HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model

Mengkang Hu, Tianxing Chen, Qiguang Chen +3 more

cs.CL · · cited 5× · arXiv ↗ · more like this ·

Large Language Model (LLM)-based agents exhibit significant potential across various domains, operating as interactive systems that process environmental observations to generate executable actions for target tasks. The effectiveness of these agents is significantly influenced by their memory mechanism, which records historical experiences as sequences of action-observation pairs. We categorize memory into two types: cross-trial memory, accumulated across multiple attempts, and in-trial memory (working memory), accumulated within a single attempt. While considerable research has optimized performance through cross-trial memory, the enhancement of agent performance through improved working memory utilization remains underexplored. Instead, existing approaches often involve directly inputting entire historical action-observation pairs into LLMs, leading to redundancy in long-horizon…

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

Baiyu Peng, Yao Mu, Jingliang Duan +3 more

cs.LG · · cited 5× · arXiv ↗ · more like this ·

Safety is essential for reinforcement learning (RL) applied in real-world tasks like autonomous driving. Chance constraints which guarantee the satisfaction of state constraints at a high probability are suitable to represent the requirements in real-world environment with uncertainty. Existing chance constrained RL methods like the penalty method and the Lagrangian method either exhibit periodic oscillations or cannot satisfy the constraints. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. Taking a control perspective, we first interpret the penalty method and the Lagrangian method as proportional feedback and integral feedback control, respectively. Then, a proportional-integral Lagrangian method is proposed to steady learning process while improving safety. To prevent integral overshooting and reduce…

Design and Construction of Geographic Knowledge Graph

Zhanji Wei, Gang Wan, Yao Mu +2 more

Artificial Intelligence · · cited 5× · DOI ↗ · more like this ·

Since proposed by Google in 2012, knowledge graph has become the backbone of many intelligent systems, such as Apple Siri, IBM Watson and many other renowned Intelligent Natural Language Question Answering System(INLQAS). A geographic knowledge graph can be integrated into Google Maps, Baidu Maps and other similar systems to increase the intelligence of map services and achieve better performance of human-computer interaction. In this paper, The framework of a geographic knowledge graph is constructed. Based on the framework, the geographic ontology is studied first and then a geographic knowledge graph is established through processes of geographic entity and relation extraction, knowledge fusion and storage. Finally, a case study of geographic knowledge query is given to validate the effectiveness of the geographic knowledge graph.

GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models

Zijian Zhang, Yuqing Jiang, Weitao Zhou +6 more

cs.RO · · cited 4× · arXiv ↗ · more like this ·

World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose \textbf{GaussianWAM}, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation…

GraphMimic: Graph-to-Graphs Generative Modeling from Videos for Policy Learning

Guangyan Chen, Te Cui, Meiling Wang +9 more

Artificial Intelligence · · cited 4× · DOI ↗ · more like this ·

Learning from demonstration is a powerful method for robotic skill acquisition. However, the significant expense of collecting such action-labeled robot data presents a major bottleneck. Video data, a rich data source encompassing diverse behavioral and physical knowledge, emerges as a promising alternative. In this paper, we present GraphMimic, a novel paradigm that leverages video data via graph-to-graphs generative modeling, which pre-trains models to generate future graphs conditioned on the graph within a video frame. Specifically, GraphMimic abstracts video frames into object and visual action vertices, and constructs graphs for state representations. The graph generative modeling network then effectively models internal structures and spatial relationships within the constructed graphs, aiming to generate future graphs. The generated graphs serve as conditions for the control…

Symmetric projection optimizer: concise and efficient solving engineering problems using the fundamental wave of the Fourier series

Haoxiang Su, Zhenghong Dong, Yi Liu +3 more

Artificial Intelligence · Scientific Reports · · cited 4× · DOI ↗ · more like this ·

The fitness function value is a kind of important information in the search process, which can be more targeted according to the guidance of the fitness function value. Most existing meta-heuristic algorithms only use the fitness function value as an indicator to compare the current variables as good or bad but do not use the fitness function value in the search process. To address this problem, the mathematical idea of the fitting is introduced into the meta-heuristic algorithm, and a symmetric projection optimizer (SPO) is proposed to solve numerical optimization and engineering problems more efficiently. The SPO algorithm mainly utilizes a new search mechanism, the symmetric projection search (SP) method. The SP method quickly completes the fitting of the projection plane, which is located through the symmetry of the two points and finds the minima in the projection plane according…

FMimic: Foundation Models are Fine-grained Action Learners from Human Videos

Guangyan Chen, Meiling Wang, Te Cui +8 more

cs.RO · · cited 3× · arXiv ↗ · more like this ·

Visual imitation learning (VIL) provides an efficient and intuitive strategy for robotic systems to acquire novel skills. Recent advancements in foundation models, particularly Vision Language Models (VLMs), have demonstrated remarkable capabilities in visual and linguistic reasoning for VIL tasks. Despite this progress, existing approaches primarily utilize these models for learning high-level plans from human demonstrations, relying on pre-defined motion primitives for executing physical interactions, which remains a major bottleneck for robotic systems. In this work, we present FMimic, a novel paradigm that harnesses foundation models to directly learn generalizable skills at even fine-grained action levels, using only a limited number of human videos. Extensive experiments demonstrate that our FMimic delivers strong performance with a single human video, and significantly…

RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis

Yao Mu, Junting Chen, Qinglong Zhang +16 more

cs.RO · · cited 3× · arXiv ↗ · more like this ·

Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI. Despite successes in applying multimodal large language models for high-level understanding, it remains challenging to translate these conceptual understandings into detailed robotic actions while achieving generalization across various scenarios. In this paper, we propose a tree-structured multimodal code generation framework for generalized robotic behavior synthesis, termed RoboCodeX. RoboCodeX decomposes high-level human instructions into multiple object-centric manipulation units consisting of physical preferences such as affordance and safety constraints, and applies code generation to introduce generalization ability across various robotics platforms. To further enhance the capability to map conceptual and perceptual…

RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation

Junting Chen, Yao Mu, Qiaojun Yu +11 more

cs.RO · · cited 3× · arXiv ↗ · more like this ·

Rapid progress in high-level task planning and code generation for open-world robot manipulation has been witnessed in Embodied AI. However, previous studies put much effort into general common sense reasoning and task planning capabilities of large-scale language or multi-modal models, relatively little effort on ensuring the deployability of generated code on real robots, and other fundamental components of autonomous robot systems including robot perception, motion planning, and control. To bridge this ``ideal-to-real'' gap, this paper presents \textbf{RobotScript}, a platform for 1) a deployable robot manipulation pipeline powered by code generation; and 2) a code generation benchmark for robot manipulation tasks in free-form natural language. The RobotScript platform addresses this gap by emphasizing the unified interface with both simulation and real robots, based on abstraction…