Safe operations of UAVs are of paramount importance for various mission-critical and safety-critical UAV applications. In context of airborne target tracking and following, UAVs need to track a flying target avoiding collision and also closely follow its trajectory. The safety situation becomes critical and more complex when the flying target is non-cooperative and has erratic movements. This paper proposes a method for collision avoidance in an autonomous fast moving dynamic quadrotor UAV tracking and following another target UAV. This is achieved by designing a safety controller that minimally modifies the control input from a trajectory tracking controller and guarantees safety. This method enables pairing our proposed safety controller with already existing flight controllers. Our safety controller uses a control barrier function based quadratic program (CBF-QP) to produce an optimal control input enabling safe operation while also follow the trajectory of the target closely. We implement our solution on AirSim simulator over PX4 flight controller and with numerical results, we validate our approach through several simulation experiments with multiple scenarios and trajectories.