In the concept of physical human-robot interaction (pHRI), the most important criterion is the safety of a human operator interacting with a high degrees of freedom (DoF) robot. Therefore, a robust control scheme is of high demand to establish safe pHRI and stabilize nonlinear, high DoF systems. In this paper, an adaptive decentralized control strategy is designed to accomplish mentioned objectives. To do so, human upper limb model and exoskeleton model are decentralized and augmented at the subsystem level to be able to design a decentralized control action. Moreover, human exogenous force (HEF) that can resist exoskeleton motion is estimated using radial basic function neural networks (RBFNNs). Estimating both human upper limb and robot rigid body parameters along with HEF estimation makes the controller adaptable to different operators, ensuring their physical safety. The \emph{barrier Lyapunov function} (BLF), on the other hand, is employed to guarantee that the robot will work in a safe workspace while ensuring stability by adjusting the control law. Additionally, unknown actuator uncertainty and constraints are considered in this study to ensure a smooth and safe pHRI. Then, the asymptotic stability of the whole system is established by means of the \emph{virtual stability} concept and \emph{virtual power flows} (VPFs). Numerical and experimental results are provided and compared to PD controller to demonstrate the excellent performance of the proposed controller. As a result, the proposed controller accomplished all the control objectives with nearly zero error and low computed torque, ensuring physical safety in pHRI.