The spread of the Red Pal Weevil (RPW) has become an existential threat for palm trees around the world. In the Middle East, RPW is causing wide-spread damage to date palm Phoenix dactylifera L., having both agricultural impacts on the palm production and environmental impacts. Early detection of RPW is very challenging, especially at large scale. This research proposes a novel remote sensing approach to recognize and monitor red palm weevil in date palm trees, using a combination of vegetation indices, object detection and semantic segmentation techniques. The study area consists of date palm trees with three classes, including healthy palms, smallish palms and severely infected palms. This proposed method achieved a promising 0.947 F1 score on test data set. This work paves the way for deploying artificial intelligence approaches to monitor RPW in large-scale as well as provide guidance for practitioners.