Neural network based planners have shown great promises in improving performance and task success rate in autonomous driving. However, it is very challenging to ensure safety of the system with learning enabled components, especially in dense and highly interactive traffic environments. In this work, we propose a neural network based lane changing planner framework that can ensure safety while sustaining system efficiency. To prevent too conservative planning, we assess the aggressiveness and identify the driving behavior of surrounding vehicles, then adapt the planned trajectory for the ego vehicle accordingly. The ego vehicle can proceed to change lanes if a safe evasion trajectory exists even in the worst case, otherwise, it can hesitate around current lateral position or return back to the original lane. We also quantitatively demonstrate the effectiveness of our planner design and its advantage over other baselines through extensive simulations with diverse and comprehensive experimental settings.