Plant disease detection is a necessary step in increasing agricultural production. Due to the difficulty of disease detection, farmers spray every form of pesticide on their crops to save them, causing harm to crop growth and food standards. Deep learning can help a lot in detecting such diseases. However, it is highly inconvenient to collect a large amount of data on all forms of disease of a specific plant species. In this paper, we propose a new metrics-based few-shot learning SSM net architecture, which consists of stacked siamese and matching network components to solve the problem of disease detection in low data regimes. We showcase that using the SSM net (stacked siamese matching) method, we were able to achieve better decision boundaries and accuracy of 94.3%, an increase of ~5% from using the traditional transfer learning approach (VGG16 and Xception net) and 3% from using original matching networks. Furthermore, we were able to attain an F1 score of 0.90 using SSM Net, an improvement from 0.30 using transfer learning and 0.80 using original matching networks.