Indoor localization services are a crucial aspect for the realization of smart cyber-physical systems within cities of the future. Such services are poised to reinvent the process of navigation and tracking of people and assets in a variety of indoor and subterranean environments. The growing ownership of computationally capable smartphones has laid the foundations of portable fingerprinting-based indoor localization through deep learning. However, as the demand for accurate localization increases, the computational complexity of the associated deep learning models increases as well. We present an approach for reducing the computational requirements of a deep learning-based indoor localization framework while maintaining localization accuracy targets. Our proposed methodology is deployed and validated across multiple smartphones and is shown to deliver up to 42% reduction in prediction latency and 45% reduction in prediction energy as compared to the best-known baseline deep learning-based indoor localization model.