This paper presents a novel quaternion-based nonsingular control system for underactuated vertical-take-off and landing (VTOL) Unmanned Aerial Vehicles (UAVs). Position and attitude tracking is challenging regarding singularity and accuracy. Quaternion-based Adaptive Backstepping Control (QABC) is developed to tackle the underactuated issues of UAV control systems in a cascaded way. Leveraging the virtual control (auxiliary control) developed in the QABC, desired attitude components and required thrust are produced. Afterwards, we propose Quaternion-based Sliding Mode Control (QASMC) to enhance the stability and mitigate chattering issues. The sliding surface is modified to avoid singularity compared to conventional SMC. To improve the robustness of controllers, the control parameters are updated using adaptation laws. Furthermore, the asymptotic stability of translational and rotational dynamics is guaranteed by utilizing Lyapunov stability and Barbalet Lemma. Finally, the comprehensive comparison results are provided to verify the effectiveness of the proposed controllers in the presence of unknown time-varying parameter uncertainties and significant initial errors. Keywords: Non-singular Sliding Mode Control, Adaptive Backstepping Control, Unit-quaternion, Drones, Unmanned Aerial Vehicles, Asymptotic Stability, Position and Orientation Control