Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses

Existing psychophysical studies have revealed that the cross-modal visual-tactile perception is common for humans performing daily activities. However, it is still challenging to build the algorithmic mapping from one modality space to another, namely the cross-modal visual-tactile data translation/generation, which could be potentially important for robotic operation. In this paper, we propose a deep-learning-based approach for cross-modal visual-tactile data generation by leveraging the framework of the generative adversarial networks (GANs). Our approach takes the visual image of a material surface as the visual data, and the accelerometer signal induced by the pen-sliding movement on the surface as the tactile data. We adopt the conditional-GAN (cGAN) structure together with the residue-fusion (RF) module, and train the model with the additional feature-matching (FM) and perceptual losses to achieve the cross-modal data generation. The experimental results show that the inclusion of the RF module, and the FM and the perceptual losses significantly improves cross-modal data generation performance in terms of the classification accuracy upon the generated data and the visual similarity between the ground-truth and the generated data.

Redirection Controller Using Reinforcement Learning

There is a growing demand for planning redirected walking techniques and applying them to physical environments with obstacles. Such techniques are mainly managed using three kinds of methods: direct scripting, generalized controller, and physical- or virtual-environment analysis to determine user redirection. The first approach is effective when a user's path and both physical and virtual environments are fixed; however, it is difficult to handle irregular movements and reuse other environments. The second approach has the potential of reusing any environment but is less optimized. The last approach is highly anticipated and versatile, although it has not been sufficiently developed. In this study, we propose a novel redirection controller using reinforcement learning with advanced plannability/versatility. Our simulation experiments show that the proposed strategy can reduce the number of resets by 20.3% for physical-space conditions with multiple obstacles.