Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. To further advance the technology in this area, in this paper we propose a novel framework to incorporate human prior knowledge in the training of DRL agents. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios, i.e., the unprotected left turn and roundabout, are designed along with human expert demonstrations. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse driving behaviors with human-like features demonstrated by the human expert. We also demonstrate that the imitative expert policy with deep ensemble-based uncertainty estimation can lead to better performance, especially in a more difficult task. As a consequence, the proposed method has shown its great potential to facilitate the applications of DRL-enabled human-like autonomous driving in practice.