Vehicle control algorithms exploiting connectivity and automation, such as Connected and Automated Vehicles (CAVs) or Advanced Driver Assistance Systems (ADAS), have the opportunity to improve energy savings. However, lower levels of automation involve a human-machine interaction stage, where the presence of a human driver affects the performance of the control algorithm in closed loop. This occurs for instance in the case of Eco-Driving control algorithms implemented as a velocity advisory system, where the driver is displayed an optimal speed trajectory to follow to reduce energy consumption. Achieving the control objectives relies on the human driver perfectly following the recommended speed. If the driver is unable to follow the recommended speed, a decline in energy savings and poor vehicle performance may occur. This warrants the creation of methods to model and forecast the response of a human driver when operating in the loop with a speed advisory system. This work focuses on developing a sequence to sequence long-short term memory (LSTM)-based driver behavior model that models the interaction of the human driver to a suggested desired vehicle speed trajectory in real-world conditions. A driving simulator is used for data collection and training the driver model, which is then compared against the driving data and a deterministic model. Results show close proximity of the LSTM-based model with the driving data, demonstrating that the model can be adopted as a tool to design human-centered speed advisory systems.