In this paper, we consider a reconfigurable intelligent surface (RIS)-assisted 3D direction-of-arrival (DoA) estimation system, in which a uniform planar array (UPA) RIS is deployed to provide virtual line-of-sight (LOS) links and reflect the uplink pilot signal to sensors. To overcome the mutually coupled problem between the beamforming design at the RIS and DoA estimation, we explore the separable sparse representation structure and propose an alternating optimization algorithm. The grid-based DoA estimation is modeled as a joint-sparse recovery problem considering the grid bias, and the Joint-2D-OMP method is used to estimate both on-grid and off-grid parts. The corresponding Cram\'er-Rao lower bound (CRLB) is derived to evaluate the estimation. Then, the beampattern at the RIS is optimized to maximize the signal-to-noise (SNR) at sensors according to the estimated angles. Numerical results show that the proposed alternating optimization algorithm can achieve lower estimation error compared to benchmarks of random beamforming design.