Relaxed Wasserstein with Applications to GANs

We propose a novel class of statistical divergences called \textit{Relaxed Wasserstein} (RW) divergence. RW divergence generalizes Wasserstein divergence and is parametrized by a class of strictly convex and differentiable functions. We establish for RW divergence several probabilistic properties, which are critical for the success of Wasserstein divergence. In particular, we show that RW divergence is dominated by Total Variation (TV) and Wasserstein-$L^2$ divergence, and that RW divergence has continuity, differentiability and duality representation. Finally, we provide a nonasymptotic moment estimate and a concentration inequality for RW divergence. Our experiments on the image generation task demonstrate that RW divergence is a suitable choice for GANs. Indeed, the performance of RWGANs with Kullback-Leibler (KL) divergence is very competitive with other state-of-the-art GANs approaches. Furthermore, RWGANs possess better convergence properties than the existing WGANs with competitive inception scores. To the best of our knowledge, our new conceptual framework is the first to not only provide the flexibility in designing effective GANs scheme, but also the possibility in studying different losses under a unified mathematical framework.