Einconv: Exploring Unexplored Tensor Decompositions for Convolutional Neural Networks

Tensor decomposition methods are one of the primary approaches for model compression and fast inference of convolutional neural networks (CNNs). However, despite their potential diversity, only a few typical decompositions such as CP decomposition have been applied in practice; more importantly, no extensive comparisons have been performed between available methods. This raises the simple question of how many decompositions are possible, and which of these is the best. In this paper, we first characterize a decomposition class specific to CNNs by adopting graphical notation, which is considerably flexible. When combining with the nonlinear activations, the class includes renowned CNN modules such as depthwise separable convolution and bottleneck layer. In the experiments, we compare the tradeoff between prediction accuracy and time/space complexities by enumerating all the possible decompositions. Also, we demonstrate, using a neural architecture search, that we can find nonlinear decompositions that outperform existing decompositions.