Human Activity Recognition (HAR) is one of the key applications of health monitoring that requires continuous use of wearable devices to track daily activities. State-of-the-art works using wearable devices have been following fog/cloud computing architecture where the data is classified at the mobile phones/remote servers. This kind of approach suffers from energy, latency, and privacy issues. Therefore, we follow edge computing architecture where the wearable device solutions provide adequate performance while being energy and memory-efficient. This paper proposes an Adaptive CNN for energy-efficient HAR (AHAR) suitable for low-power edge devices. AHAR uses a novel adaptive architecture that selects a portion of the baseline architecture to use during the inference phase. We validate our methodology in classifying locomotion activities from two datasets- Opportunity and w-HAR. Compared to the fog/cloud computing approaches for the Opportunity dataset, our baseline and adaptive architecture shows a comparable weighted F1 score of 91.79%, and 91.57%, respectively. For the w-HAR dataset, our baseline and adaptive architecture outperforms the state-of-the-art works with a weighted F1 score of 97.55%, and 97.64%, respectively. Evaluation on real hardware shows that our baseline architecture is significantly energy-efficient (422.38x less) and memory-efficient (14.29x less) compared to the works on the Opportunity dataset. For the w-HAR dataset, our baseline architecture requires 2.04x less energy and 2.18x less memory compared to the state-of-the-art work. Moreover, experimental results show that our adaptive architecture is 12.32% (Opportunity) and 11.14% (w-HAR) energy-efficient than our baseline while providing similar (Opportunity) or better (w-HAR) performance with no significant memory overhead.