Predicting material properties base on micro structure of materials has long been a challenging problem. Recently many deep learning methods have been developed for material property prediction. In this study, we propose a crystal representation learning framework, Orbital CrystalNet, OCrystalNet, which consists of two parts: atomic descriptor generation and graph representation learning. In OCrystalNet, we first incorporate orbital field matrix (OFM) and atomic features to construct OFM-feature atomic descriptor, and then the atomic descriptor is used as atom embedding in the atom-bond message passing module which takes advantage of the topological structure of crystal graphs to learn crystal representation. To demonstrate the capabilities of OCrystalNet we performed a number of prediction tasks on Material Project dataset and JARVIS dataset and compared our model with other baselines and state of art methods. To further present the effectiveness of OCrystalNet, we conducted ablation study and case study of our model. The results show that our model have various advantages over other state of art models.