Point-of-Interest (POI) recommendation is an important task in location-based social networks. It facilitates the sharing between users and locations. Recently, researchers tend to recommend POIs by long- and short-term interests. However, existing models are mostly based on sequential modeling of successive POIs to capture transitional regularities. A few works try to acquire user's mobility periodicity or POI's geographical influence, but they omit some other spatial-temporal factors. To this end, we propose to jointly model various spatial-temporal factors by context-aware non-successive modeling. In the long-term module, we split user's all historical check-ins into seven sequences by day of week to obtain daily interest, then we combine them by attention. This will capture temporal effect. In the short-term module, we construct four short-term sequences to acquire sequential, spatial, temporal, and spatial-temporal effects, respectively. Attention of interest-level is used to combine all factors and interests. Experiments on two real-world datasets demonstrate the state-of-the-art performance of our method.