In this review, automatic defect inspection algorithms that analyze Electron Microscope (EM) images of Semiconductor Manufacturing (SM) products are identified, categorized, and discussed. This is a topic of critical importance for the SM industry as the continuous shrinking of device patterns has led to increasing defectivity and a greater prevalence of higher-resolution imaging tools such as EM. These aspects among others threaten to increase costs as a result of increased inspection time-to-solution and decreased yield, respectively. Relevant research papers were systematically identified in four popular publication databases in January 2024. A total of 103 papers were selected after screening for novel contributions relating to automatic EM image analysis algorithms for semiconductor defect inspection. These papers were then categorized based on the inspection tasks they addressed, their evaluation metrics, and the type of algorithms used. A notable finding from this categorization is that reference-based defect detection algorithms were the most popular algorithm type until 2020 when deep learning-based inspection algorithms became more popular, especially for defect classification. Furthermore, four broader research questions were discussed to come to the following conclusions: (i) the key components of inspection algorithms are set up, pre-processing, feature extraction, and final prediction; (ii) the maturity of the manufacturing process affects the data availability and required sensitivity of inspection algorithms; (iii) key challenges for these algorithms relate to the desiderata of minimizing time-to-solution which pushes for high imaging throughput, reducing manual input during algorithm setup, and higher processing throughput; and (iv) three promising directions for future work are suggested based on gaps in the reviewed literature that address key remaining limitations.