In the simulation-based testing and evaluation of autonomous vehicles (AVs), how background vehicles (BVs) drive directly influences the AV’s driving behavior and further affects the test results. Most existing simulation platforms use either predetermined trajectories or deterministic driving models to model BV behaviors. However, predetermined BV trajectories cannot react to AV maneuvers, and deterministic models are different from real human drivers because of the lack of stochastic components and errors. Both methods lead to unrealistic traffic scenarios. This paper presents a learning-based stochastic driving model that meets the unique needs of AV testing (i.e., interactive and human-like stochasticity). The model is built based on the long short-term memory architecture. By incorporating the concept of quantile regression into the loss function of the model, the stochastic behaviors are reproduced without prior assumption of human drivers. The model is trained with the large-scale naturalistic driving data (NDD) from the Safety Pilot Model Deployment project and compared with a stochastic intelligent driving model (IDM). Analysis of individual trajectories shows that the proposed model can reproduce more similar trajectories of human drivers than IDM. To validate the ability of the proposed model in generating a naturalistic driving environment, traffic simulation experiments are implemented. The results show that traffic flow parameters such as speed, range, and time headway distribution match closely with the NDD, which is of significant importance for AV testing and evaluation.