The tomographic imaging method magnetic particle imaging (MPI) requires a multi-patch approach for capturing large field of views. This approach consists of a continuous or stepwise spatial shift of a small sub-volume of only few cubic centimeters size, which is scanned using one or multiple excitation fields in the kHz range. Under the assumption of ideal magnetic fields, the MPI system matrix is shift invariant and in turn a single matrix suffices for image reconstruction significantly reducing the calibration time and reconstruction effort. For large field imperfections, however, the method can lead to severe image artifacts. In the present work we generalize the efficient multi-patch reconstruction to work under non-ideal field conditions, where shift invariance holds only approximately for small shifts of the sub-volume. Patches are clustered based on a magnetic-field-based metric such that in each cluster the shift invariance holds in good approximation. The total number of clusters is the main parameter of our method and allows to trade off calibration time and image artifacts. The magnetic-field-based metric allows to perform the clustering without prior knowledge of the system matrices. The developed reconstruction algorithm is evaluated on a multi-patch measurement sequence with 15 patches, where efficient multi-patch reconstruction with a single calibration measurement leads to strong image artifacts. Analysis reveals that calibration measurements can be decreased from 15 to 11 with no visible image artifacts. A further reduction to 9 is possible with only slight degradation in image quality.