Particle accelerators support a wide array of scientific, industrial, and medical applications. To meet the needs of these applications, accelerator physicists rely heavily on detailed simulations of the complicated particle beam dynamics through the accelerator. One of the most computationally expensive and difficult-to-model effects is the impact of Coherent Synchrotron Radiation (CSR). As a beam travels through a curved trajectory (e.g. due to a bending magnet), it emits radiation that in turn interacts with the rest of the beam. At each step through the trajectory, the electromagnetic field introduced by CSR (called the CSR wakefield) needs to computed and used when calculating the updates to the positions and momenta of every particle in the beam. CSR is one of the major drivers of growth in the beam emittance, which is a key metric of beam quality that is critical in many applications. The CSR wakefield is very computationally intensive to compute with traditional electromagnetic solvers, and this is a major limitation in accurately simulating accelerators. Here, we demonstrate a new approach for the CSR wakefield computation using a neural network solver structured in a way that is readily generalizable to new setups. We validate its performance by adding it to a standard beam tracking test problem and show a ten-fold speedup along with high accuracy.