A 2D and 3D building map provides invaluable information for understanding human activities and their impacts on Earth and its environment. Despite enormous efforts to improve the quality of building maps, current large-scale building maps have lots of errors and are limited to providing only 2D building information. This study presents a state-of-the-art 2D and 3D building extraction algorithm with airborne LiDAR data that is suitable for large-scale building mapping. Our algorithm operates in a fully unsupervised manner and does not require either any training label or training procedure. Our algorithm requires only simple operations of morphological filtering and planarity-based filtering but can produce an accurate 2D and 3D building map. A quantitative and qualitative evaluation in a large-scale dataset (-550 sqkm) of Denver and New York City showed that our algorithm outperforms the deep learning-based Microsoft's building mapping algorithm even without any parameter tuning. More extensive evaluations in different conditions of landscapes confirmed that our algorithm is scalable and can be improved further with appropriate parameter selection. Our algorithm is more advantageous than other image-based building extraction algorithms in that it is more computationally efficient, more accurate, and more explainable. Our proposed algorithm that can produce an accurate large-scale 2D and 3D building map provides a great potential towards a global-scale 2D and 3D building mapping with airborne LiDAR data.