Traditional tracking of magnetic markers leads to high computational costs due to the requirement for iterative optimization procedures. Furthermore, such approaches rely on the magnetic dipole model for the optimization function, leading to inaccurate results anytime a non-spherical magnet gets close to a sensor in the array. We propose to overcome these limitations by using neural networks to infer the marker's position and orientation. Our method can obtain the magnet's five degrees of freedom (5 DoF) in a single inference step without relying on an initial estimation. As a counterpart, the supervised training phase is data intensive. We solve this by generating synthetic, yet realistic, data via Finite Element Methods simulations. A fast and accurate inference largely compensates for the offline training preparation. We evaluate our system using different cylindrical magnets, tracked with a square array of 16 sensors. We use a portable, neural networks-oriented single-board computer for the sensors' reading and the position inference, making our setup very compact. We compared our tracking outputs with vision-based ground truth data. Our prototype implementation tracks $5$ DoF with an averaged positional error of 4mm and orientation error of 8deg within a 0.2x0.2x0.15m working volume.