Computer Science · Artificial Intelligence
The Role of Dataset Generation in Offline Reinforcement Learning
See on the mapAbstract
Offline Reinforcement Learning is a sub-field of Reinforcement Learning, where policies are learned from logged interactions alone. Dispensing direct interactions makes the dataset one of the most essential parts of the algorithm, having large influence on the performance of the learned policy. The effects of dataset compositions have mostly been disregarded when comparing recent algorithmic advances in the field. Towards that end, an empirical analysis of the effect of dataset composition on the performance of Offline Reinforcement Learning algorithms for discrete action environments is conducted. Dataset composition is evaluated through two metrics, the Trajectory Quality and the State-Action Coverage. Furthermore, the Entropy of the behavioral policy used to create the dataset is linked to those metrics. \nAnalysis of the conducted experiments suggest that variants of the off-policy Deep Q-Network family rely on the dataset to have high State-Action Coverage. Contrary to that, algorithms constraining the learned policy towards the behavioral policy that generated the dataset perform well across datasets, if they exhibit either high Trajectory Quality, high State-Action Coverage or both. On datasets that have high Trajectory Quality, Behavioral Cloning is performing as good as the best considered Offline Reinforcement Learning algorithms. Furthermore, it was found that policies with intermediate entropy are desirable to create datasets that can be leveraged with Offline Reinforcement Learning, as those datasets pose a tradeoff between Trajectory Quality and State-Action Coverage.