The energy of ocean waves is the key distinguishing factor of a marine environment compared to other aquatic environments. Waves have a significant impact on the dynamics of marine vehicles. Hence, it is imperative to model waves and the dynamics of vehicles in waves when developing efficient control strategies for autonomous marine vehicles. However, most marine simulators available open-source exclude the realistic modelling of ocean waves and the efficient computation of wave forces on surface vehicles. This paper presents ASV-Swarm, a simulator which provides high fidelity and computationally efficient model of ocean waves and vehicle dynamics in waves. The simulator is suitable for applications requiring high run-time performance, such as with swarms of autonomous marine vehicles, or in developing optimal vehicle control strategies using reinforcement learning techniques. ASV-Swarm also has a low computational overhead making it ideal for onboard simulation for applications such as online learning for adaptation to changes in the environment.