arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Haobo Fu

32 papers · 2012–2026

2012: 12013: 02014: 32015: 02016: 02017: 02018: 22019: 02020: 02021: 22022: 32023: 62024: 62025: 62026: 3
2012peak 6 in 20232026

Frequent co-authors

Papers

showing 1–25 of 32 RSS

Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space

Jiechao Xiong, Qing Wang, Zhuoran Yang +7 more

cs.LG · · cited 253× · arXiv ↗ · more like this ·

Most existing deep reinforcement learning (DRL) frameworks consider either discrete action space or continuous action space solely. Motivated by applications in computer games, we consider the scenario with discrete-continuous hybrid action space. To handle hybrid action space, previous works either approximate the hybrid space by discretization, or relax it into a continuous set. In this paper, we propose a parametrized deep Q-network (P- DQN) framework for the hybrid action space without approximation or relaxation. Our algorithm combines the spirits of both DQN (dealing with discrete action space) and DDPG (dealing with continuous action space) by seamlessly integrating them. Empirical results on a simulation example, scoring a goal in simulated RoboCup soccer and the solo mode in game King of Glory (KOG) validate the efficiency and effectiveness of our method.

Robust Optimization Over Time: Problem Difficulties and Benchmark Problems

Haobo Fu, Bernhard Sendhoff, Ke Tang +1 more

Artificial Intelligence · IEEE Transactions on Evolutionary Computation · · cited 40× · DOI ↗ · more like this ·

The focus of most research in evolutionary dynamic optimization has been tracking moving optimum (TMO). Yet, TMO does not capture all the characteristics of real-world dynamic optimization problems (DOPs), especially in situations where a solution's future fitness has to be considered. To account for a solution's future fitness explicitly, we propose to find robust solutions to DOPs, which are formulated as the robust optimization over time (ROOT) problem. In this paper we analyze two robustness definitions in ROOT and then develop two types of benchmark problems for the two robustness definitions in ROOT, respectively. The two types of benchmark problems are motivated by the inappropriateness of existing DOP benchmarks for the study of ROOT. Additionally, we evaluate four representative methods from the literature on our proposed ROOT benchmarks, in order to gain a better understanding…

What are dynamic optimization problems?

Haobo Fu, Peter R. Lewis, Bernhard Sendhoff +2 more

Artificial Intelligence · Aston Publications Explorer (Aston University) · · cited 27× · DOI ↗ · more like this ·

Dynamic Optimization Problems (DOPs) have been widely studied using Evolutionary Algorithms (EAs). Yet, a clear and rigorous definition of DOPs is lacking in the Evolutionary Dynamic Optimization (EDO) community. In this paper, we propose a unified definition of DOPs based on the idea of multiple-decision-making discussed in the Reinforcement Learning (RL) community. We draw a connection between EDO and RL by arguing that both of them are studying DOPs according to our definition of DOPs. We point out that existing EDO or RL research has been mainly focused on some types of DOPs. A conceptualized benchmark problem, which is aimed at the systematic study of various DOPs, is then developed. Some interesting experimental studies on the benchmark reveal that EDO and RL methods are specialized in certain types of DOPs and more importantly new algorithms for DOPs can be developed by combining…

Find robust solutions over time by two-layer multi-objective optimization method

Yinan Guo, Mei‐Rong Alice Chen, Haobo Fu +1 more

Computational Theory and Mathematics · · cited 20× · DOI ↗ · more like this ·

Robust optimization over time is a practical dynamic optimization method, which provides two detailed computable metrics to get the possible robust solutions for dynamic scalar optimization problems. However, the robust solutions fit for more time-varying moments or approximate the optimum more because only one metric is considered as the optimization objective. To find the true robust solution set satisfying maximum both survival time and average fitness simultaneously during all dynamic environments, a novel two-layer multi-objective optimization method is proposed. In the first layer, considering both metrics, the acceptable optimal solutions for each changing environment is found. Subsequently, they are composed of the practical robust solution set in the second layer. Taking the average fitness and the length of the robust solution set as two objectives, the optimal combinations…

Characterizing environmental changes in Robust Optimization Over Time

Haobo Fu, Bernhard Sendhoff, Ke Tang +1 more

Artificial Intelligence · · cited 20× · DOI ↗ · more like this ·

Evolutionary dynamic optimization has been drawing more and more research attention, and yet most work in this area is focused on Tracking Moving Optimum (TMO), which is to optimize the current fitness function at any time point. Recently, we proposed a more practical way to solve dynamic optimization problems, which is referred to as Robust Optimization Over Time (ROOT). In ROOT, we are trying to find solutions whose performances are acceptable over more than one environmental state, i.e., fitness functions. Before any development of benchmarks or algorithms for ROOT, it is necessary to have some understanding of what aspects of an environment can change and more importantly how these changes influence the solving of ROOT problems. In this paper, we develop a number of measures which can be used to characterize and analyse the underlying changing environment in the framework of ROOT.…

L2E: Learning to Exploit Your Opponent

Zhe Wu, Kai Li, Enmin Zhao +5 more

cs.LG · · cited 13× · arXiv ↗ · more like this ·

Opponent modeling is essential to exploit sub-optimal opponents in strategic interactions. Most previous works focus on building explicit models to directly predict the opponents' styles or strategies, which require a large amount of data to train the model and lack adaptability to unknown opponents. In this work, we propose a novel Learning to Exploit (L2E) framework for implicit opponent modeling. L2E acquires the ability to exploit opponents by a few interactions with different opponents during training, thus can adapt to new opponents with unknown styles during testing quickly. We propose a novel opponent strategy generation algorithm that produces effective opponents for training automatically. We evaluate L2E on two poker games and one grid soccer game, which are the commonly used benchmarks for opponent modeling. Comprehensive experimental results indicate that L2E quickly adapts…

Speedup Training Artificial Intelligence for Mahjong via Reward Variance Reduction

Jinqiu Li, Shuang Wu, Haobo Fu +3 more

Artificial Intelligence · · cited 7× · DOI ↗ · more like this ·

Despite significant breakthroughs in developing gaming artificial intelligence (AI), Mahjong remains quite challenging as a popular multi-player imperfect information game. Compared with games such as Go and Texas Hold’em, Mahjong has much more invisible information, unfixed game order, and a complicated scoring system, resulting in high randomness and variance of the rewarding signals during the reinforcement learning process. This paper presents a Mahjong AI by introducing Reward Variance Reduction (RVR) into a new self-play deep reinforcement learning algorithm. RVR handles the invisibility via a relative value network which leverages the global information to guide the model to converge to the optimal strategy under an oracle with perfect information. Moreover, RVR improves the training stability using an expected reward network to adapt to the complex, dynamic, and highly…

Multi-objective Optimization-based Selection for Quality-Diversity by Non-surrounded-dominated Sorting

Ren-Jian Wang, Ke Xue, Haopu Shang +3 more

Computational Theory and Mathematics · · cited 5× · DOI ↗ · more like this ·

Quality-Diversity (QD) algorithms, a subset of evolutionary algorithms, maintain an archive (i.e., a set of solutions) and simulate the natural evolution process through iterative selection and reproduction, with the goal of generating a set of high-quality and diverse solutions. Though having found many successful applications in reinforcement learning, QD algorithms often select the parent solutions uniformly at random, which lacks selection pressure and may limit the performance. Recent studies have treated each type of behavior of a solution as an objective, and selected the parent solutions based on Multi-objective Optimization (MO), which is a natural idea, but has not lead to satisfactory performance as expected. This paper gives the reason for the first time, and then proposes a new MO-based selection method by non-surrounded-dominated sorting (NSS), which considers all possible…

AutoCFR: Learning to Design Counterfactual Regret Minimization Algorithms

Hang Xu, Kai Li, Haobo Fu +2 more

Artificial Intelligence · Proceedings of the AAAI Conference on Artificial Intelligence · · cited 5× · DOI ↗ · more like this ·

Counterfactual regret minimization (CFR) is the most commonly used algorithm to approximately solving two-player zero-sum imperfect-information games (IIGs). In recent years, a series of novel CFR variants such as CFR+, Linear CFR, DCFR have been proposed and have significantly improved the convergence rate of the vanilla CFR. However, most of these new variants are hand-designed by researchers through trial and error based on different motivations, which generally requires a tremendous amount of efforts and insights. This work proposes to meta-learn novel CFR algorithms through evolution to ease the burden of manual algorithm design. We first design a search language that is rich enough to represent many existing hand-designed CFR variants. We then exploit a scalable regularized evolution algorithm with a bag of acceleration techniques to efficiently search over the combinatorial space…

Combining Tree Search and Action Prediction for State-of-the-Art Performance in DouDiZhu

Yunsheng Zhang, Dong Yan, Bei Shi +5 more

Artificial Intelligence · · cited 5× · DOI ↗ · more like this ·

AlphaZero has achieved superhuman performance on various perfect-information games, such as chess, shogi and Go. However, directly applying AlphaZero to imperfect-information games (IIG) is infeasible, due to the fact that traditional MCTS methods cannot handle missing information of other players. Meanwhile, there have been several extensions of MCTS for IIGs, by implicitly or explicitly sampling a state of other players. But, due to the inability to handle private and public information well, the performance of these methods is not satisfactory. In this paper, we extend AlphaZero to multiplayer IIGs by developing a new MCTS method, Action-Prediction MCTS (AP-MCTS). In contrast to traditional MCTS extensions for IIGs, AP-MCTS first builds the search tree based on public information, adopts the policy-value network to generalize between hidden states, and finally predicts other players'…

Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routing

Jinmin He, Kai Li, Yifan Zang +4 more

cs.AI · · cited 4× · arXiv ↗ · more like this ·

Multi-task reinforcement learning endeavors to accomplish a set of different tasks with a single policy. To enhance data efficiency by sharing parameters across multiple tasks, a common practice segments the network into distinct modules and trains a routing network to recombine these modules into task-specific policies. However, existing routing approaches employ a fixed number of modules for all tasks, neglecting that tasks with varying difficulties commonly require varying amounts of knowledge. This work presents a Dynamic Depth Routing (D2R) framework, which learns strategic skipping of certain intermediate modules, thereby flexibly choosing different numbers of modules for each task. Under this framework, we further introduce a ResRouting method to address the issue of disparate routing paths between behavior and target policies during off-policy training. In addition, we design an…

Maximum Entropy Heterogeneous-Agent Reinforcement Learning

Jiarong Liu, Yifan Zhong, Siyi Hu +4 more

cs.MA · · cited 3× · arXiv ↗ · more like this ·

Multi-agent reinforcement learning (MARL) has been shown effective for cooperative games in recent years. However, existing state-of-the-art methods face challenges related to sample complexity, training instability, and the risk of converging to a suboptimal Nash Equilibrium. In this paper, we propose a unified framework for learning stochastic policies to resolve these issues. We embed cooperative MARL problems into probabilistic graphical models, from which we derive the maximum entropy (MaxEnt) objective for MARL. Based on the MaxEnt framework, we propose Heterogeneous-Agent Soft Actor-Critic (HASAC) algorithm. Theoretically, we prove the monotonic improvement and convergence to quantal response equilibrium (QRE) properties of HASAC. Furthermore, we generalize a unified template for MaxEnt algorithmic design named Maximum Entropy Heterogeneous-Agent Mirror Learning (MEHAML), which…

Enhance Reasoning for Large Language Models in the Game Werewolf

Shuang Wu, Liwen Zhu, Tao Yang +4 more

cs.AI · · cited 2× · arXiv ↗ · more like this ·

This paper presents an innovative framework that integrates Large Language Models (LLMs) with an external Thinker module to enhance the reasoning capabilities of LLM-based agents. Unlike augmenting LLMs with prompt engineering, Thinker directly harnesses knowledge from databases and employs various optimization techniques. The framework forms a reasoning hierarchy where LLMs handle intuitive System-1 tasks such as natural language processing, while the Thinker focuses on cognitive System-2 tasks that require complex logical analysis and domain-specific knowledge. Our framework is presented using a 9-player Werewolf game that demands dual-system reasoning. We introduce a communication protocol between LLMs and the Thinker, and train the Thinker using data from 18800 human sessions and reinforcement learning. Experiments demonstrate the framework's effectiveness in deductive reasoning…

Reaching Consensus in Cooperative Multi-Agent Reinforcement Learning with Goal Imagination

Liangzhou Wang, Kaiwen Zhu, Fengming Zhu +6 more

cs.AI · · cited 1× · arXiv ↗ · more like this ·

Reaching consensus is key to multi-agent coordination. To accomplish a cooperative task, agents need to coherently select optimal joint actions to maximize the team reward. However, current cooperative multi-agent reinforcement learning (MARL) methods usually do not explicitly take consensus into consideration, which may cause miscoordination problem. In this paper, we propose a model-based consensus mechanism to explicitly coordinate multiple agents. The proposed Multi-agent Goal Imagination (MAGI) framework guides agents to reach consensus with an Imagined common goal. The common goal is an achievable state with high value, which is obtained by sampling from the distribution of future states. We directly model this distribution with a self-supervised generative model, thus alleviating the "curse of dimensinality" problem induced by multi-agent multi-step policy rollout commonly used…

Policy Space Diversity for Non-Transitive Games

Jian Yao, Weiming Liu, Haobo Fu +4 more

cs.GT · · cited 1× · arXiv ↗ · more like this ·

Policy-Space Response Oracles (PSRO) is an influential algorithm framework for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games. Many previous studies have been trying to promote policy diversity in PSRO. A major weakness in existing diversity metrics is that a more diverse (according to their diversity metrics) population does not necessarily mean (as we proved in the paper) a better approximation to a NE. To alleviate this problem, we propose a new diversity metric, the improvement of which guarantees a better approximation to a NE. Meanwhile, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. By incorporating our diversity regularization into the best response solving in PSRO, we obtain a new PSRO variant, Policy Space Diversity PSRO (PSD-PSRO). We present the convergence property of PSD-PSRO.…

PARAMETRIZED DEEP Q-NETWORKS LEARNING: PLAYING ONLINE BATTLE ARENA WITH DISCRETE-CONTINUOUS HYBRID ACTION SPACE

Jiechao Xiong, Qing Wang, Zhuoran Yang +5 more

Artificial Intelligence · · cited 1× · OpenAlex ↗ · more like this ·

Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either discrete or continuous space. Motivated by the project of design Game AI for King of Glory (KOG), one the world’s most popular mobile game, we consider the scenario with the discrete-continuous hybrid action space. To directly apply existing DLR frameworks, existing approaches either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN) for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the game KOG validates the efficiency and effectiveness of our method.

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu, Zhengcheng Wang, Jiahong Liu +7 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their…

Back to the Definition: Estimating Step-Level Advantages via Trajectory Graphs for Agentic Reinforcement Learning

Xincheng Yao, Haobo Fu, Weiming Liu +1 more

cs.AI · · cited 0× · arXiv ↗ · more like this ·

Group-based reinforcement learning (RL) methods, such as GRPO and its variants, have become a leading paradigm for training reasoning and agentic large language models (LLMs). While their group-normalized advantage estimation is reliable at the response level, it becomes systematically biased at the step level, since coarse-grained trajectory-level advantages are hard to accurately reflect the contribution of individual steps (i.e, failed trajectories may contain valuable steps). Revisiting the foundational RL definition, we notice that GRPO's success on single-turn tasks stems from its advantage estimation strategy, which adheres to the basic definition: the mean reward of multiple actions sampled from the same state constitutes a credible state-value estimate. Extending the faithful estimation to step-level would in principle demand sampling multiple actions from each intermediate…

Deep (Predictive) Discounted Counterfactual Regret Minimization

Hang Xu, Kai Li, Haobo Fu +3 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Counterfactual regret minimization (CFR) is a family of algorithms for effectively solving imperfect-information games. To enhance CFR's applicability in large games, researchers use neural networks to approximate its behavior. However, existing methods are mainly based on vanilla CFR and struggle to effectively integrate more advanced CFR variants. In this work, we propose an efficient model-free neural CFR algorithm, overcoming the limitations of existing methods in approximating advanced CFR variants. At each iteration, it collects variance-reduced sampled advantages based on a value network, fits cumulative advantages by bootstrapping, and applies discounting and clipping operations to simulate the update mechanisms of advanced CFR variants. Experimental results show that, compared with model-free neural algorithms, it exhibits faster convergence in typical imperfect-information…

TacticCraft: Natural Language-Driven Tactical Adaptation for StarCraft II

Weiyu Ma, Jiwen Jiang, Haobo Fu +1 more

cs.AI · · cited 0× · arXiv ↗ · more like this ·

We present an adapter-based approach for tactical conditioning of StarCraft II AI agents. Current agents, while powerful, lack the ability to adapt their strategies based on high-level tactical directives. Our method freezes a pre-trained policy network (DI-Star) and attaches lightweight adapter modules to each action head, conditioned on a tactical tensor that encodes strategic preferences. By training these adapters with KL divergence constraints, we ensure the policy maintains core competencies while exhibiting tactical variations. Experimental results show our approach successfully modulates agent behavior across tactical dimensions including aggression, expansion patterns, and technology preferences, while maintaining competitive performance. Our method enables flexible tactical control with minimal computational overhead, offering practical strategy customization for complex…

Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidance

Jinmin He, Kai Li, Yifan Zang +4 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Multi-task reinforcement learning endeavors to efficiently leverage shared information across various tasks, facilitating the simultaneous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross-Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. In addition, we propose two gating mechanisms to improve the learning efficiency of CTPG…

Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning

Jinmin He, Kai Li, Yifan Zang +4 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Offline multi-task reinforcement learning aims to learn a unified policy capable of solving multiple tasks using only pre-collected task-mixed datasets, without requiring any online interaction with the environment. However, it faces significant challenges in effectively sharing knowledge across tasks. Inspired by the efficient knowledge abstraction observed in human learning, we propose Goal-Oriented Skill Abstraction (GO-Skill), a novel approach designed to extract and utilize reusable skills to enhance knowledge transfer and task performance. Our approach uncovers reusable skills through a goal-oriented skill extraction process and leverages vector quantization to construct a discrete skill library. To mitigate class imbalances between broadly applicable and task-specific skills, we introduce a skill enhancement phase to refine the extracted skills. Furthermore, we integrate these…

Preference Goal Tuning: Post-Training as Latent Control for Frozen Policies

Guangyu Zhao, Kewei Lian, Haoxuan Ru +8 more

cs.AI · · cited 0× · arXiv ↗ · more like this ·

Goal-conditioned policies enable decision-making models to execute diverse behaviors based on specified goals, yet their downstream performance is often highly sensitive to the choice of instructions or prompts. To bypass the limitations of discrete text prompts, we formulate post-training adaptation as a latent control problem, where the goal embedding serves as a continuous control variable to modulate the behavior of a frozen policy. We propose Preference Goal Tuning (PGT), a framework that optimizes this latent control variable to align the induced trajectory distribution with task preferences. Unlike standard fine-tuning that updates policy parameters, PGT keeps the policy frozen and updates only the latent goal using a trajectory-level preference objective. This approach essentially searches for the optimal conditioning input that maximizes the likelihood of preferred behaviors…

Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning

Hanlin Yang, Jian Yao, Weiming Liu +13 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Recovering a spectrum of diverse policies from a set of expert trajectories is an important research topic in imitation learning. After determining a latent style for a trajectory, previous diverse policies recovering methods usually employ a vanilla behavioral cloning learning objective conditioned on the latent style, treating each state-action pair in the trajectory with equal importance. Based on an observation that in many scenarios, behavioral styles are often highly relevant with only a subset of state-action pairs, this paper presents a new principled method in diverse polices recovery. In particular, after inferring or assigning a latent style for a trajectory, we enhance the vanilla behavioral cloning by incorporating a weighting mechanism based on pointwise mutual information. This additional weighting reflects the significance of each state-action pair's contribution to…

Pointer Networks Trained Better via Evolutionary Algorithms

Muyao Zhong, Shengcai Liu, Bingdong Li +3 more

cs.NE · · cited 0× · arXiv ↗ · more like this ·

Pointer Network (PtrNet) is a specific neural network for solving Combinatorial Optimization Problems (COPs). While PtrNets offer real-time feed-forward inference for complex COPs instances, its quality of the results tends to be less satisfactory. One possible reason is that such issue suffers from the lack of global search ability of the gradient descent, which is frequently employed in traditional PtrNet training methods including both supervised learning and reinforcement learning. To improve the performance of PtrNet, this paper delves deeply into the advantages of training PtrNet with Evolutionary Algorithms (EAs), which have been widely acknowledged for not easily getting trapped by local optima. Extensive empirical studies based on the Travelling Salesman Problem (TSP) have been conducted. Results demonstrate that PtrNet trained with EA can consistently perform much better…