Deep learning can be used to extract meaningful results from images. In this paper, we used convolutional neural networks combined with recurrent neural networks on images of plasmonic structures and extract absorption data form them. To provide the required data for the model we did 100,000 simulations with similar setups and random structures. By designing a deep network we could find a model that could predict the absorption of any structure with similar setup. We used convolutional neural networks to get the spatial information from the images and we used recurrent neural networks to help the model find the relationship between the spatial information obtained from convolutional neural network model. With this design we could reach a very low loss in predicting the absorption compared to the results obtained from numerical simulation in a very short time.