We focus on the problem of rearranging a set of objects within a confined space with a nonholonomically constrained mobile robot pusher. This problem is relevant to many real-world domains, including warehouse automation and construction. These domains give rise to instances involving a combination of geometric, kinematic, and physics constraints, which make planning particularly challenging. Prior work often makes simplifying assumptions like the use of holonomic mobile robots or dexterous manipulators capable of unconstrained overhand reaching. Our key insight is we can empower even a constrained mobile pusher to tackle complex rearrangement tasks by enabling it to modify the environment to its favor in a constraint-aware fashion. To this end, we describe a Push-Traversability graph, whose vertices represent poses that the pusher can push objects from and edges represent optimal, kinematically feasible, and stable push-rearrangements of objects. Based on this graph, we develop ReloPush, a planning framework that leverages Dubins curves and standard graph search techniques to generate an efficient sequence of object rearrangements to be executed by the pusher. We evaluate ReloPush across a series of challenging scenarios, involving the rearrangement of densely cluttered workspaces with up to eight objects by a 1tenth mobile robot pusher. ReloPush exhibits orders of magnitude faster runtimes and significantly more robust execution in the real world, evidenced in lower execution times and fewer losses of object contact, compared to two baselines lacking our proposed graph structure.