Dataset bias is a well-known problem in the field of computer vision. The presence of implicit bias in any image collection hinders a model trained and validated on a particular dataset to yield similar accuracies when tested on other datasets. In this paper, we propose a novel debiasing technique to reduce the effects of a biased training dataset. Our goal is to augment the training data using a generative network by learning a non-linear mapping from the source domain (training set) to the target domain (testing set) while retaining training set labels. The cycle consistency loss and adversarial loss for generative adversarial networks are used to learn the mapping. A structured similarity index (SSIM) loss is used to enforce label retention while augmenting the training set. Our methods and hypotheses are supported by quantitative comparisons with prior debiasing techniques. These comparisons showcase the superiority of our method and its potential to mitigate the effects of dataset bias during the inference stage.