Improving Mobile Interactive Video QoE via Two-Level Online Cooperative Learning

Abstract

Machine learning models, particularly reinforcement learning (RL), have demonstrated great potential in optimizing video streaming applications. However, the state-of-the-art solutions are limited to an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">“offline learning”</i> paradigm, i.e., the RL models are trained in simulators and then are operated in real networks. As a result, they inevitably suffer from the simulation-to-reality gap, showing far less satisfactory performance under real conditions compared with simulated environment. In this article, we close the gap by proposing Legato, an online RL framework for real-time mobile interactive video systems. Legato puts many individual RL agents directly into the video system, which make video bitrate decisions in real-time and evolve their models over time. Legato then employs a two-level cooperative learning mechanism to enhance video QoE. First, Legato proposes a score-based robust learning algorithm to eliminate risks of quality degradation caused by the RL model's exploration attempts. Then, Legato adaptively aggregates agents following a network condition-aware manner to form its corresponding high-level RL model that can help each individual to react to unseen network conditions. We implement Legato on an interactive real-time video system. Based on the exhaustive evaluations, we find that Legato outperforms the state-of-the-art algorithms significantly across a wide range of QoE metrics.