In this paper, an automatic calibration algorithm is proposed to reduce the depth error caused by internal stray light in amplitude-modulated continuous wave (AMCW) coaxial scanning light detection and ranging (LiDAR). Assuming that the internal stray light inside the proposed AMCW scanning LiDAR is static, the amplitude and phase delay of internal stray light are estimated using the Gaussian mixture model (GMM) and particle swarm optimization (PSO). Specifically, the raw amplitude (amplitude of raw cross-correlation) map of calibration checkboard at specific distance is segmented by GMM with two clusters (the bright and dark regions). The loss function is then defined as L1-norm of difference between mean depth of the amplitude-segmented clusters. This loss function is minimized by PSO to estimate the two optimal target parameters: the amplitude and phase delay of internal stray light. To avoid overfitting at a specific distance, the calibration check board is measured at multiple distances and the average of L1 loss functions from all measured distances is chosen as the actual loss. According to the validation of the proposed algorithm, the original loss is reduced from tens of centimeters to 3.2 mm when the distance of the calibration checkboard is between 1m and 4 m. This accurate depth error correction performance is also maintained in the depth and raw amplitude images of geometrically complex objects. The proposed internal stray light calibration algorithm in this paper can be used for any type of AMCW coaxial scanning LiDAR regardless of its optical characteristics.