Neonatal seizures are a commonly encountered neurological condition. They are the first clinical signs of a serious neurological disorder. Thus, rapid recognition and treatment are necessary to prevent serious fatalities. The use of electroencephalography (EEG) in the field of neurology allows precise diagnosis of several medical conditions. However, interpreting EEG signals needs the attention of highly specialized staff since the infant brain is developmentally immature during the neonatal period. Detecting seizures on time could potentially prevent the negative effects on the neurocognitive development of the infants. In recent years, neonatal seizure detection using machine learning algorithms have been gaining traction. Since there is a need for the classification of bio-signals to be computationally inexpensive in the case of seizure detection, this research presents a machine learning (ML) based architecture that operates with comparable predictive performance as previous models but with minimum level configuration. The proposed classifier was trained and tested on a public dataset of NICU seizures recorded at the Helsinki University Hospital. Our architecture achieved a best sensitivity of 87%, which is 6% more than that of the standard ML model chosen in this study. The model size of the ML classifier is optimized to just 4.84 KB with minimum prediction time of 182.61 milliseconds, thus enabling it to be deployed on wearable ultra-edge devices for quick and accurate response and obviating the need for cloud-based and other such exhaustive computational methods.