Orthogonal delay-Doppler (DD) division multiplexing (ODDM) has been recently proposed as a promising multicarrier modulation scheme to tackle Doppler spread in high-mobility environments. Accurate channel estimation is of paramount importance to guarantee reliable communication for the ODDM, especially when the delays and Dopplers of the propagation paths are off-grid. In this paper, we propose a novel grid refinement and adjustment-based sparse Bayesian inference (GRASBI) scheme for DD domain channel estimation. The GRASBI involves first formulating the channel estimation problem as a sparse signal recovery through the introduction of a virtual DD grid. Then, an iterative process is proposed that involves (i) sparse Bayesian learning to estimate the channel parameters and (ii) a novel grid refinement and adjustment process to adjust the virtual grid points. The grid adjustment in GRASBI relies on the maximum likelihood principle to attain the adjustment and utilizes refined grids that have much higher resolution than the virtual grid. Moreover, a low-complexity grid refinement and adjustment-based channel estimation scheme is proposed, that can provides a good tradeoff between the estimation accuracy and the complexity. Finally, numerical results are provided to demonstrate the accuracy and efficiency of the proposed channel estimation schemes.