Emergency vehicle (EV) service is a key function of cities and is exceedingly challenging due to urban traffic congestion. A key contributor to EV service delay is the lack of communication and cooperation between vehicles blocking EVs. In this paper, we study the improvement of EV service using vehicle-to-vehicle connectivity. We consider the establishment of dynamic queue jumper lanes (DQJLs) based on real-time coordination of connected vehicles. We develop a novel stochastic dynamic programming formulation for the DQJL problem, which explicitly account for the uncertainty of drivers' reaction to approaching EVs. We propose a deep neural network-based approximate dynamic programming (ADP) algorithm that efficiently computes the optimal coordination instructions. We also validate our approach on a micro-simulation testbed using Simulation On Urban Mobility (SUMO).