Action Robust Reinforcement Learning with Highly Expressive Policy

Abstract

In traditional reinforcement learning, there can be a degradation in the control performance of the policy when the environmental parameters differ between the training and application phase. The policy that minimizes this degradation is referred to as a robust policy. A framework called Noisy action Robust Markov Decision Process (NR-MDP) was proposed for training robust policies, and the Action Robust Deep Determin-istic Policy Gradient (AR-DDPG) algorithm was introduced as a method for solving NR-MDP. The optimal policy in NR-MDP includes policies following various probability distributions, whereas AR-DDPG is restricted to deterministic policies. We propose a new robust reinforcement learning method called Action Robust Q-Learning (AR-QL) that enables the training of optimal policies in NR-MDP by leveraging various sampling techniques to extend the representational capacity of policies, targeting an improvement in policy robustness. To validate this, we confirmed that AR-QL can acquire the optimal policy for a simple NR-MDP problem, for which AR-DDPG fails to obtain the optimal policy. Furthermore, we confirmed that the robust performance of policy trained by AR-QL in the OpenAI's InvertedPendulum environment surpasses that of policy trained by AR-DDPG.