Monitoring the technical condition of infrastructure is a crucial element to its maintenance. Currently, the applied methods are outdated, labour intensive and highly inaccurate. At the same time, the latest methods using Artificial Intelligence techniques, despite achieving satisfactory results in the detection of infrastructure damage, are severely limited in their application due to two main factors - labour-intensive gathering of new datasets and high demand for computing power. In the presented work, we propose to utilize Transfer Learning techniques and computer vision to overcome these limiting factor and fully harness the advantages of Artificial Intelligence methods. We describe a framework which enables hassle-free development of unique infrastructure defects detectors on digital images, achieving the accuracy of above 90%. The framework supports semi-automatic creation of new datasets and has modest computing power requirements. It is implemented in the form of a ready-to-use software package distributed under an open software licence and available for the public. Thus, it can be used to immediately implement the methods proposed in this paper in the process of infrastructure management by government units, regardless of their financial capabilities. With the help of introduced framework it is possible to improve the efficiency of infrastructure management and the quality of its life cycle documentation globally, leading to a more accurate mapping of the processes taking place in the infrastructure's life cycle for better infrastructure planning in the future.