The multi-scale defect detection for solar cell electroluminescence (EL) images is a challenging task, due to the feature vanishing as network deepens. To address this problem, a novel Bidirectional Attention Feature Pyramid Network (BAFPN) is designed by combining the novel multi-head cosine non-local attention module with top-down and bottom-up feature pyramid networks through bidirectional cross-scale connections, which can make all layers of the pyramid share similar semantic features. In multi-head cosine non-local attention module, cosine function is applied to compute the similarity matrix of the input features. Furthermore, a novel object detector is proposed, called BAF-Detector, which embeds BAFPN into Region Proposal Network (RPN) in Faster RCNN+FPN to improve the detection effect of multi-scale defects in solar cell EL images. Finally, some experimental results on a large-scale EL dataset including 3629 images, 2129 of which are defective, show that the proposed method performs much better than other methods in terms of multi-scale defects classification and detection results in raw solar cell EL images.