Deep regression on manifolds: a 3D rotation case study

Many problems in machine learning involve regressing outputs that do not lie on a Euclidean space, such as a discrete probability distribution, or the pose of an object. An approach to tackle these problems through gradient-based learning consists in including in the deep learning architecture a differentiable function mapping arbitrary inputs of a Euclidean space onto this manifold. In this work, we establish a set of properties that such mapping should satisfy to allow proper training, and illustrate it in the case of 3D rotations. Through theoretical considerations and methodological experiments on a variety of tasks, we compare various differentiable mappings on the 3D rotation space, and conjecture about the importance of the local linearity of the mapping. We notably show that a mapping based on Procrustes orthonormalization of a 3x3 matrix generally performs best among the ones considered, but that rotation-vector representation might also be suitable when restricted to small angles.