Recent work on super-resolution show that a very deep convolutional neural networks (CNN) have obtained remarkable performance. However, as CNN models have become deeper and wider, the required computational cost is substantially higher. In this paper, we propose Linear Depthwise Convolution to address this problem in single image super resolution. Specifically, Linear Depthwise Convolution can reduce computational burden on CNN model, preserving information used to reconstruct super-resolved image. The performance improvement of our proposed method is due to removing non-linearity between depthwise convolution and pointwise convolution. We evaluate the proposed approach using Set 5 and Set 14 datasets and show it performs significant better performance.