This article presents GLIM, a 3D range-inertial localization and mapping framework with GPU-accelerated scan matching factors. The odometry estimation module of GLIM employs a combination of fixed-lag smoothing and keyframe-based point cloud matching that makes it possible to deal with a few seconds of completely degenerated range data while efficiently reducing trajectory estimation drift. It also incorporates multi-camera visual feature constraints in a tightly coupled way to further improve the stability and accuracy. The global trajectory optimization module directly minimizes the registration errors between submaps over the entire map. This approach enables us to accurately constrain the relative pose between submaps with a small overlap. Although both the odometry estimation and global trajectory optimization algorithms require much more computation than existing methods, we show that they can be run in real-time due to the careful design of the registration error evaluation algorithm and the entire system to fully leverage GPU parallel processing.