The advent of Ultra-Reliable Low Latency Communication (URLLC) alongside the emergence of Open RAN (ORAN) architectures presents unprecedented challenges and opportunities in Radio Resource Management (RRM) for next-generation communication systems. This paper presents a comprehensive trade-off analysis of Deep Reinforcement Learning (DRL) approaches designed to enhance URLLC performance within ORAN's flexible and dynamic framework. By investigating various DRL strategies for optimising RRM parameters, we explore the intricate balance between reliability, latency, and the newfound adaptability afforded by ORAN principles. Through extensive simulation results, our study compares the efficacy of different DRL models in achieving URLLC objectives in an ORAN context, highlighting the potential of DRL to navigate the complexities introduced by ORAN. The proposed study provides valuable insights into the practical implementation of DRL-based RRM solutions in ORAN-enabled wireless networks. It sheds light on the benefits and challenges of integrating DRL and ORAN for URLLC enhancements. Our findings contribute to the ongoing discourse on advancements in URLLC and ORAN, offering a roadmap for future research to pursue efficient, reliable, and flexible communication systems.