Class-dependent Compression of Deep Neural Networks

Today's deep neural networks require substantial computation resources for their training, storage and inference, which limits their effective use on resource-constrained devices. On the one hand, many recent research activities explore different options of compressing and optimizing deep models. On the other hand, in many real-world applications we face the class imbalance problem, e.g. higher number of false positives produced by a compressed network may be tolerable, yet the number of false negatives must stay low. The problem originates from either an intrinsic nature of the imbalanced samples within the training data set, or from the fact that some classes are more important for the application domain of the model, e.g. in medical imaging. In this paper, we propose a class-dependent network compression method based on a newly introduced network pruning technique used to search for lottery tickets in an original deep network. We introduce a novel combined loss function to find efficient compressed sub-networks with the same or even lower number of false negatives compared to the original network. Our experimental evaluation using three benchmark data sets shows that the resulting compressed sub-networks achieve up to 50% lower number of false negatives and an overall higher AUC-ROC measure, yet use up to 99% fewer parameters compared to the original network.