Author

Gerhard Neumann

181 papers · 1987–2026

1987peak 24 in 20242026

Frequent co-authors

Papers

showing 1–25 of 181 RSS

An Algorithmic Perspective on Imitation Learning

Takayuki Osa, Joni Pajarinen, Gerhard Neumann +3 more

cs.RO · 16 Nov 2018 · cited 1,073× · arXiv ↗ · more like this ·

As robots and other intelligent agents move from simple environments and problems to more complex, unstructured settings, manually programming their behavior has become increasingly challenging and expensive. Often, it is easier for a teacher to demonstrate a desired behavior rather than attempt to manually engineer it. This process of learning from demonstrations, and the study of algorithms to do so, is called imitation learning. This work provides an introduction to imitation learning. It covers the underlying assumptions, approaches, and how they relate; the rich set of algorithms developed to tackle the problem; and advice on effective tools and implementation. We intend this paper to serve two audiences. First, we want to familiarize machine learning experts with the challenges of imitation learning, particularly those arising in robotics, and the interesting theoretical and…

Deep Reinforcement Learning for Swarm Systems

Maximilian Hüttenrauch, Adrian Šošić, Gerhard Neumann

cs.MA · July 2018 · cited 257× · arXiv ↗ · more like this ·

Recently, deep reinforcement learning (RL) methods have been applied successfully to multi-agent scenarios. Typically, these methods rely on a concatenation of agent states to represent the information content required for decentralized decision making. However, concatenation scales poorly to swarm systems with a large number of homogeneous agents as it does not exploit the fundamental properties inherent to these systems: (i) the agents in the swarm are interchangeable and (ii) the exact number of agents in the swarm is irrelevant. Therefore, we propose a new state representation for deep multi-agent RL based on mean embeddings of distributions. We treat the agents as samples of a distribution and use the empirical mean embedding as input for a decentralized policy. We define different feature spaces of the mean embedding using histograms, radial basis functions and a neural network…

Policy evaluation with temporal differences: a survey and comparison

Christoph Dann, Gerhard Neumann, Jan Peters

Artificial Intelligence · Lincoln Repository (University of Lincoln) · 2014 · cited 167× · OpenAlex ↗ · more like this ·

Policy evaluation is an essential step in most reinforcement learning approaches. It yields a value function, the quality assessment of states for a given policy, which can be used in a policy improvement step. Since the late 1980s, this research area has been dominated by temporal-difference (TD) methods due to their data-efficiency. However, core issues such as stability guarantees in the off-policy scenario, improved sample efficiency and probabilistic treatment of the uncertainty in the estimates have only been tackled recently, which has led to a large number of new approaches.\n\nThis paper aims at making these new developments accessible in a concise overview, with foci on underlying cost functions, the off-policy scenario as well as on regularization in high dimensional feature spaces. By presenting the first extensive, systematic comparative evaluations comparing TD…

Guided Deep Reinforcement Learning for Swarm Systems

Maximilian Hüttenrauch, Adrian Šošić, Gerhard Neumann

cs.MA · 18 Sep 2017 · cited 150× · arXiv ↗ · more like this ·

In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a policy for a group of agents is difficult due to distributed partial observability of the state. Here, we follow a guided approach where a critic has central access to the global state during learning, which simplifies the policy evaluation problem from a reinforcement learning point of view. For example, we can get the positions of all robots of the swarm using a camera image of a scene. This camera image is only available to the critic and not to the control policies of the robots. We follow an actor-critic approach, where the actors base their decisions only on locally…

Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces

Philipp Becker, Harit Pandya, Gregor Gebhardt +3 more

cs.LG · 17 May 2019 · cited 137× · arXiv ↗ · more like this ·

In order to integrate uncertainty estimates into deep time-series modelling, Kalman Filters (KFs) (Kalman et al., 1960) have been integrated with deep learning models, however, such approaches typically rely on approximate inference techniques such as variational inference which makes learning more complex and often less scalable due to approximation errors. We propose a new deep approach to Kalman filtering which can be learned directly in an end-to-end manner using backpropagation without additional approximations. Our approach uses a high-dimensional factorized latent state representation for which the Kalman updates simplify to scalar operations and thus avoids hard to backpropagate, computationally heavy and potentially unstable matrix inversions. Moreover, we use locally linear dynamic models to efficiently propagate the latent state to the next time step. The resulting network…

A survey of preference-based reinforcement learning methods

Christian Wirth, Riad Akrour, Gerhard Neumann +1 more

Artificial Intelligence · Repository KITopen (Karlsruhe Institute of Technology) · 2017 · cited 115× · DOI ↗ · more like this ·

Reinforcement learning (RL) techniques optimize the accumulated long-term reward of a suitably chosen reward function. However, designing such a reward function of ten requires a lot of task-specific prior knowledge. The designer needs to consider different objectives that do not only influence the learned behavior but also the learning progress. To alleviate these issues, preference-based reinforcement learning algorithms (PbRL) have been proposed that can directly learn from an expert's preferences instead of a hand-designed numeric reward. PbRL has gained traction in recent years due to its ability to resolve the reward shaping problem, its ability to learn from non numeric rewards and the possibility to reduce the dependence on expert knowledge. We provide a unified framework for PbRL that describes the task formally and points out the different design principles that affect the…

Data-Efficient Generalization of Robot Skills with Contextual Policy Search

Andras Kupcsik, Marc Peter Deisenroth, Jan Peters +1 more

Artificial Intelligence · Proceedings of the AAAI Conference on Artificial Intelligence · 29 Jun 2013 · cited 91× · DOI ↗ · more like this ·

In robotics, controllers make the robot solve a task within a specific context. The context can describe the objectives of the robot or physical properties of the environment and is always specified before task execution. To generalize the controller to multiple contexts, we follow a hierarchical approach for policy learning: A lower-level policy controls the robot for a given context and an upper-level policy generalizes among contexts. Current approaches for learning such upper-level policies are based on model-free policy search, which require an excessive number of interactions of the robot with its environment. More data-efficient policy search approaches are model based but, thus far, without the capability of learning hierarchical policies. We propose a new model-based policy search approach that can also learn contextual upper-level policies. Our approach is based on learning…

Variational Inference for Policy Search in changing situations

Gerhard Neumann

Artificial Intelligence · Lincoln Repository (University of Lincoln) · 28 Jun 2011 · cited 70× · OpenAlex ↗ · more like this ·

Many policy search algorithms minimize the Kullback-Leibler (KL) divergence to a certain target distribution in order to fit their policy. The commonly used KL-divergence forces the resulting policy to be ’reward-attracted’. The policy tries to reproduce all positively rewarded experience while negative experience is neglected. However, the KL-divergence is not symmetric and we can also minimize the the reversed KL-divergence, which is typically used in variational inference. The policy now becomes ’cost-averse’. It tries to avoid reproducing any negatively-rewarded experience while maximizing exploration. Due to this ’cost-averseness ’ of the policy, Variational Inference for Policy Search (VIP) has several interesting properties. It requires no kernelbandwith nor exploration rate, such settings are determined automatically by the inference. The algorithm meets the performance of…

Model-Free Preference-Based Reinforcement Learning

Christian Wirth, Johannes Fürnkranz, Gerhard Neumann

Artificial Intelligence · Proceedings of the AAAI Conference on Artificial Intelligence · 2 Mar 2016 · cited 68× · DOI ↗ · more like this ·

Specifying a numeric reward function for reinforcement learning typically requires a lot of hand-tuning from a human expert. In contrast, preference-based reinforcement learning (PBRL) utilizes only pairwise comparisons between trajectories as a feedback signal, which are often more intuitive to specify. Currently available approaches to PBRL for control problems with continuous state/action spaces require a known or estimated model, which is often not available and hard to learn. In this paper, we integrate preference-based estimation of the reward function into a model-free reinforcement learning (RL) algorithm, resulting in a model-free PBRL algorithm. Our new algorithm is based on Relative Entropy Policy Search (REPS), enabling us to utilize stochastic policies and to directly control the greediness of the policy update. REPS decreases exploration of the policy slowly by limiting…

Model-Free Trajectory-based Policy Optimization with Monotonic Improvement

Riad Akrour, Abbas Abdolmaleki, Hany Abdulsamad +2 more

cs.LG · June 2016 · cited 52× · arXiv ↗ · more like this ·

Many of the recent trajectory optimization algorithms alternate between linear approximation of the system dynamics around the mean trajectory and conservative policy update. One way of constraining the policy change is by bounding the Kullback-Leibler (KL) divergence between successive policies. These approaches already demonstrated great experimental success in challenging problems such as end-to-end control of physical systems. However, the linear approximation of the system dynamics can introduce a bias in the policy update and prevent convergence to the optimal policy. In this article, we propose a new model-free trajectory-based policy optimization algorithm with guaranteed monotonic improvement. The algorithm backpropagates a local, quadratic and time-dependent \qfunc~learned from trajectory data instead of a model of the system dynamics. Our policy update ensures exact…

Sample-based informationl-theoretic stochastic optimal control

Rudolf Lioutikov, Alexandros Paraschos, Jan Peters +1 more

Artificial Intelligence · Lincoln Repository (University of Lincoln) · May 2014 · cited 47× · DOI ↗ · more like this ·

Many Stochastic Optimal Control (SOC) approaches rely on samples to either obtain an estimate of the value function or a linearisation of the underlying system model. However, these approaches typically neglect the fact that the accuracy of the policy update depends on the closeness of the resulting trajectory distribution to these samples. The greedy operator does not consider such closeness constraint to the samples. Hence, the greedy operator can lead to oscillations or even instabilities in the policy updates. Such undesired behaviour is likely to result in an inferior performance of the estimated policy. We reuse inspiration from the reinforcement learning community and relax the greedy operator used in SOC with an information theoretic bound that limits the `distance' of two subsequent trajectory distributions in a policy update. The introduced bound ensures a smooth and stable…

Learning sequential motor tasks

Christian Daniel, Gerhard Neumann, Oliver Kroemer +1 more

Artificial Intelligence · Lincoln Repository (University of Lincoln) · May 2013 · cited 41× · DOI ↗ · more like this ·

Many real robot applications require the sequential use of multiple distinct motor primitives. This requirement implies the need to learn the individual primitives as well as a strategy to select the primitives sequentially. Such hierarchical learning problems are commonly either treated as one complex monolithic problem which is hard to learn, or as separate tasks learned in isolation. However, there exists a strong link between the robots strategy and its motor primitives. Consequently, a consistent framework is needed that can learn jointly on the level of the individual primitives and the robots strategy. We present a hierarchical learning method which improves individual motor primitives and, simultaneously, learns how to combine these motor primitives sequentially to solve complex motor tasks. We evaluate our method on the game of robot hockey, which is both difficult to learn in…

Sim-to-Real Quadrotor Landing via Sequential Deep Q-Networks and Domain Randomization

Riccardo Polvara, Massimiliano Patacchiola, Marc Hanheide +1 more

Artificial Intelligence · Robotics · 25 Feb 2020 · cited 38× · DOI ↗ · more like this ·

The autonomous landing of an Unmanned Aerial Vehicle (UAV) on a marker is one of the most challenging problems in robotics. Many solutions have been proposed, with the best results achieved via customized geometric features and external sensors. This paper discusses for the first time the use of deep reinforcement learning as an end-to-end learning paradigm to find a policy for UAVs autonomous landing. Our method is based on a divide-and-conquer paradigm that splits a task into sequential sub-tasks, each one assigned to a Deep Q-Network (DQN), hence the name Sequential Deep Q-Network (SDQN). Each DQN in an SDQN is activated by an internal trigger, and it represents a component of a high-level control policy, which can navigate the UAV towards the marker. Different technical solutions have been implemented, for example combining vanilla and double DQNs, and the introduction of a…

Improving Local Trajectory Optimisation using Probabilistic Movement Primitives

R. B. Ashith Shyam, Peter Lightbody, Gautham P. Das +3 more

Computer Vision and Pattern Recognition · Lincoln Repository (University of Lincoln) · November 2019 · cited 38× · DOI ↗ · more like this ·

Local trajectory optimisation techniques are a powerful tool for motion planning. However, they often get stuck in local optima depending on the quality of the initial solution and consequently, often do not find a valid (i.e. collision free) trajectory. Moreover, they often require fine tuning of a cost function to obtain the desired motions. In this paper, we address both problems by combining local trajectory optimisation with learning from demonstrations. The human expert demonstrates how to reach different target end-effector locations in different ways. From these demonstrations, we estimate a trajectory distribution, represented by a Probabilistic Movement Primitive (ProMP). For a new target location, we sample different trajectories from the ProMP and use these trajectories as initial solutions for the local optimisation. As the ProMP generates versatile initial solutions for…

What Matters For Meta-Learning Vision Regression Tasks?

Ning Gao, Hanna Ziesche, Ngo Anh Vien +2 more

cs.CV · 9 Mar 2022 · cited 30× · arXiv ↗ · more like this ·

Meta-learning is widely used in few-shot classification and function regression due to its ability to quickly adapt to unseen tasks. However, it has not yet been well explored on regression tasks with high dimensional inputs such as images. This paper makes two main contributions that help understand this barely explored area. \emph{First}, we design two new types of cross-category level vision regression tasks, namely object discovery and pose estimation of unprecedented complexity in the meta-learning domain for computer vision. To this end, we (i) exhaustively evaluate common meta-learning techniques on these tasks, and (ii) quantitatively analyze the effect of various deep learning techniques commonly used in recent meta-learning algorithms in order to strengthen the generalization capability: data augmentation, domain randomization, task augmentation and meta-regularization.…

Learning of non-parametric control policies with high-dimensional state features

Herke van Hoof, Jan Peters, Gerhard Neumann

Artificial Intelligence · Lincoln Repository (University of Lincoln) · 12 May 2015 · cited 30× · OpenAlex ↗ · more like this ·

Learning complex control policies from highdimensional sensory input is a challenge for\nreinforcement learning algorithms. Kernel methods that approximate values functions\nor transition models can address this problem. Yet, many current approaches rely on\ninstable greedy maximization. In this paper, we develop a policy search algorithm that\nintegrates robust policy updates and kernel embeddings. Our method can learn nonparametric\ncontrol policies for infinite horizon continuous MDPs with high-dimensional\nsensory representations. We show that our method outperforms related approaches, and\nthat our algorithm can learn an underpowered swing-up task task directly from highdimensional\nimage data.

Compatible Natural Gradient Policy Search

Joni Pajarinen, Hong Linh Thai, Riad Akrour +2 more

cs.LG · 7 Feb 2019 · cited 28× · arXiv ↗ · more like this ·

Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural gradient and trust region optimization are equivalent if we use the natural parameterization of a standard exponential policy distribution in combination with compatible value function approximation. Moreover, we show that standard natural gradient updates may reduce the entropy of the policy according to a wrong schedule leading to premature convergence. To control entropy reduction we introduce a new policy search method called compatible policy search (COPOS) which bounds entropy loss. The experimental results show that COPOS yields state-of-the-art results in challenging continuous control tasks and in discrete partially observable tasks.

Non-parametric policy search with limited information loss

Herke van Hoof, Gerhard Neumann, Jan Peters

Artificial Intelligence · Lincoln Repository (University of Lincoln) · 31 Aug 2017 · cited 24× · OpenAlex ↗ · more like this ·

Learning complex control policies from non-linear and redundant sensory input is an important challenge for reinforcement learning algorithms. Non-parametric methods that approximate values functions or transition models can address this problem, by adapting to the complexity of the dataset. Yet, many current non-parametric approaches rely on\nunstable greedy maximization of approximate value functions, which might lead to poor convergence or oscillations in the policy update. A more robust policy update can be obtained by limiting the information loss between successive state-action distributions. In this paper, we develop a policy search algorithm with policy updates that are both robust and non-parametric. Our method can learn non-parametric control policies for infinite horizon continuous Markov decision processes with non-linear and redundant sensory representations. We…

Deriving and improving CMA-ES with information geometric trust regions

Abbas Abdolmaleki, Bob Price, Nuno Lau +2 more

Artificial Intelligence · Proceedings of the Genetic and Evolutionary Computation Conference · 30 Jun 2017 · cited 23× · DOI ↗ · more like this ·

CMA-ES is one of the most popular stochastic search algorithms. It performs favourably in many tasks without the need of extensive parameter tuning. The algorithm has many beneficial properties, including automatic step-size adaptation, efficient covariance updates that incorporates the current samples as well as the evolution path and its invariance properties. Its update rules are composed of well established heuristics where the theoretical foundations of some of these rules are also well understood. In this paper we will fully derive all CMA-ES update rules within the framework of expectation-maximisation-based stochastic search algorithms using information-geometric trust regions. We show that the use of the trust region results in similar updates to CMA-ES for the mean and the covariance matrix while it allows for the derivation of an improved update rule for the step-size. Our…

Stochastic Optimal Control Methods for Investigating the Power of Morphological Computation

Elmar Rückert, Gerhard Neumann

Artificial Intelligence · Artificial Life · 27 Nov 2012 · cited 23× · DOI ↗ · more like this ·

One key idea behind morphological computation is that many difficulties of a control problem can be absorbed by the morphology of a robot. The performance of the controlled system naturally depends on the control architecture and on the morphology of the robot. Because of this strong coupling, most of the impressive applications in morphological computation typically apply minimalistic control architectures. Ideally, adapting the morphology of the plant and optimizing the control law interact so that finally, optimal physical properties of the system and optimal control laws emerge. As a first step toward this vision, we apply optimal control methods for investigating the power of morphological computation. We use a probabilistic optimal control method to acquire control laws, given the current morphology. We show that by changing the morphology of our robot, control problems can be…

Regularizing Reinforcement Learning with State Abstraction

Riad Akrour, Filipe Veiga, Jan Peters +1 more

Artificial Intelligence · MPG.PuRe (Max Planck Society) · October 2018 · cited 21× · DOI ↗ · more like this ·

State abstraction in a discrete reinforcement learning setting clusters states sharing a similar optimal action to yield an easier to solve decision process. In this paper, we generalize the concept of state abstraction to continuous action reinforcement learning by defining an abstract state as a state cluster over which a near-optimal policy of simple shape exists. We propose a hierarchical reinforcement learning algorithm that is able to simultaneously find the state space clustering and the optimal sub-policies in each cluster. The main advantage of the proposed framework is to provide a straightforward way of regularizing reinforcement learning by controlling the behavioral complexity of the learned policy. We apply our algorithm on several benchmark tasks and a robot tactile manipulation task and show that we can match state-of-the-art deep reinforcement learning performance by…

Latent space policy search for robotics

Kevin Sebastian Luck, Gerhard Neumann, Erik Berger +2 more

Artificial Intelligence · Lincoln Repository (University of Lincoln) · September 2014 · cited 20× · DOI ↗ · more like this ·

Learning motor skills for robots is a hard task. In particular, a high number of degrees-of-freedom in the robot can pose serious challenges to existing reinforcement learning methods, since it leads to a high-dimensional search space. However, complex robots are often intrinsically redundant systems and, therefore, can be controlled using a latent manifold of much smaller dimensionality. In this paper, we present a novel policy search method that performs efficient reinforcement learning by uncovering the low-dimensional latent space of actuator redundancies. In contrast to previous attempts at combining reinforcement learning and dimensionality reduction, our approach does not perform dimensionality reduction as a preprocessing step but naturally combines it with policy search. Our evaluations show that the new approach outperforms existing algorithms for learning motor skills with…

Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning

Paul Maria Scheikl, Balázs Gyenes, Tornike Davitashvili +6 more

cs.RO · 10 Oct 2021 · cited 19× · arXiv ↗ · more like this ·

Cognitive cooperative assistance in robot-assisted surgery holds the potential to increase quality of care in minimally invasive interventions. Automation of surgical tasks promises to reduce the mental exertion and fatigue of surgeons. In this work, multi-agent reinforcement learning is demonstrated to be robust to the distribution shift introduced by pairing a learned policy with a human team member. Multi-agent policies are trained directly from images in simulation to control multiple instruments in a sub task of the minimally invasive removal of the gallbladder. These agents are evaluated individually and in cooperation with humans to demonstrate their suitability as autonomous assistants. Compared to human teams, the hybrid teams with artificial agents perform better considering completion time (44.4% to 71.2% shorter) as well as number of collisions (44.7% to 98.0% fewer). Path…

Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertainty

Alireza Ranjbar, Ngo Anh Vien, Hanna Ziesche +2 more

cs.RO · June 2021 · cited 18× · arXiv ↗ · more like this ·

While classic control theory offers state of the art solutions in many problem scenarios, it is often desired to improve beyond the structure of such solutions and surpass their limitations. To this end, residual policy learning (RPL) offers a formulation to improve existing controllers with reinforcement learning (RL) by learning an additive "residual" to the output of a given controller. However, the applicability of such an approach highly depends on the structure of the controller. Often, internal feedback signals of the controller limit an RL algorithm to adequately change the policy and, hence, learn the task. We propose a new formulation that addresses these limitations by also modifying the feedback signals to the controller with an RL policy and show superior performance of our approach on a contact-rich peg-insertion task under position and orientation uncertainty. In…

Efficient Gradient-Free Variational Inference using Policy Search

Oleg Arenz, Gerhard Neumann, Mingjun Zhong

Artificial Intelligence · Lincoln Repository (University of Lincoln) · 2022 · cited 16× · DOI ↗ · more like this ·

Inference from complex distributions is a common problem in machine learning needed for many Bayesian methods. We propose an efficient, gradient-free method for learning general GMM approximations of multimodal distributions based on recent insights from stochastic search methods. Our method establishes information-geometric trust regions to ensure efficient exploration of the sampling space and stability of the GMM updates, allowing for efficient estimation of multi-variate Gaussian variational distributions. For GMMs, we apply a variational lower bound to decompose the learning objective into sub-problems given by learning the individual mixture components and the coefficients. The number of mixture components is adapted online in order to allow for arbitrary exact approximations. We demonstrate on several domains that we can learn significantly better approximations than competing…