This study proposes a hierarchically integrated framework for safe task and motion planning (TAMP) of bipedal locomotion in a partially observable environment with dynamic obstacles and uneven terrain. The high-level task planner employs linear temporal logic (LTL) for a reactive game synthesis between the robot and its environment and provides a formal guarantee on navigation safety and task completion. To address environmental partial observability, a belief abstraction is employed at the high-level navigation planner to estimate the dynamic obstacles' location when they are out of the robot's local field of view. Accordingly, a synthesized action planner sends a set of locomotion actions including walking step, step height, and heading angle change, to the middle-level motion planner, while incorporating safe locomotion specifications extracted from safety theorems based on a reduced-order model (ROM) of the locomotion process. The motion planner employs the ROM to design safety criteria and a sampling algorithm to generate non-periodic motion plans that accurately track high-level actions. To address external perturbations, this study also investigates safe sequential composition of the keyframe locomotion state and achieves robust transitions against external perturbations through reachability analysis. A set of ROM-based hyperparameters are finally interpolated to design whole-body locomotion gaits generated by trajectory optimization and validate the viable deployment of the ROM-based TAMP to the full-body trajectory generation for a 20-degrees-of-freedom Cassie bipedal robot designed by Agility Robotics. The proposed framework is validated by a set of scenarios in uneven, partially observable environments with dynamical obstacles.