Offline Reinforcement Learning Hands-On

Offline Reinforcement Learning (RL) aims to turn large datasets into powerful decision-making engines without any online interactions with the environment. This great promise has motivated a large amount of research that hopes to replicate the success RL has experienced in simulation settings. This work ambitions to reflect upon these efforts from a practitioner viewpoint. We start by discussing the dataset properties that we hypothesise can characterise the type of offline methods that will be the most successful. We then verify these claims through a set of experiments and designed datasets generated from environments with both discrete and continuous action spaces. We experimentally validate that diversity and high-return examples in the data are crucial to the success of offline RL and show that behavioural cloning remains a strong contender compared to its contemporaries. Overall, this work stands as a tutorial to help people build their intuition on today's offline RL methods and their applicability.

Exploring the fitness landscape of a realistic turbofan rotor blade optimization

Aerodynamic shape optimization has established itself as a valuable tool in the engineering design process to achieve highly efficient results. A central aspect for such approaches is the mapping from the design parameters which encode the geometry of the shape to be improved to the quality criteria which describe its performance. The choices to be made in the setup of the optimization process strongly influence this mapping and thus are expected to have a profound influence on the achievable result. In this work we explore the influence of such choices on the effects on the shape optimization of a turbofan rotor blade as it can be realized within an aircraft engine design process. The blade quality is assessed by realistic three dimensional computational fluid dynamics (CFD) simulations. We investigate the outcomes of several optimization runs which differ in various configuration options, such as optimization algorithm, initialization, number of degrees of freedom for the parametrization. For all such variations, we generally find that the achievable improvement of the blade quality is comparable for most settings and thus rather insensitive to the details of the setup. On the other hand, even supposedly minor changes in the settings, such as using a different random seed for the initialization of the optimizer algorithm, lead to very different shapes. Optimized shapes which show comparable performance usually differ quite strongly in their geometries over the complete blade. Our analyses indicate that the fitness landscape for such a realistic turbofan rotor blade optimization is highly multi-modal with many local optima, where very different shapes show similar performance.