This paper presents visual and 3D structure inspection for steel structures and bridges using a developed climbing robot. The robot can move freely on a steel surface, carry sensors, collect data and then send to the ground station in real time for monitoring as well as further processing. Steel surface image stitching and 3D map building are conducted to provide a current condition of the structure. Also, a computer vision-based method is implemented to detect surface defects on stitched images. The effectiveness of the climbing robot's inspection is tested in multiple circumstances to ensure strong steel adhesion and successful data collection. The detection method was also successfully evaluated on various test images, where steel cracks could be automatically identified, without the requirement of some heuristic reasoning.