Manual identification and classification of pneumonia and COVID-19 infection is a cumbersome process that, if delayed can cause irreversible damage to the patient. We have compiled CT scan images from various sources, namely, from the China Consortium of Chest CT Image Investigation (CC-CCII), the Negin Radiology located at Sari in Iran, an open access COVID-19 repository from Havard dataverse, and Sri Ramachandra University, Chennai, India. The images were preprocessed using various methods such as normalization, sharpening, median filter application, binarizing, and cropping to ensure uniformity while training the models. We present an ensemble classification approach using deep learning and machine learning methods to classify patients with the said diseases. Our ensemble model uses pre-trained networks such as ResNet-18 and ResNet-50 for classification and MobileNetV2 for feature extraction. The features from MobileNetV2 are used by the gradient-boosting classifier for the classification of patients. Using ResNet-18, ResNet-50, and the MobileNetV2 aided gradient boosting classifier, we propose an ensemble model with an accuracy of 98 percent on unseen data.