Scaled and Inter-token Relation Enhanced Transformer for Sample-restricted Residential NILM

Recent advancements in transformer models have yielded impressive results in Non-Intrusive Load Monitoring (NILM). However, effectively training a transformer on small-scale datasets remains a challenge. This paper addresses this issue by enhancing the attention mechanism of the original transformer to improve performance. We propose two novel mechanisms: the inter-token relation enhancement mechanism and the dynamic temperature tuning mechanism. The first mechanism reduces the prioritization of intra-token relationships in the token similarity matrix during training, thereby increasing inter-token focus. The second mechanism introduces a learnable temperature tuning for the token similarity matrix, mitigating the over-smoothing problem associated with fixed temperature values. Both mechanisms are supported by rigorous mathematical foundations. We evaluate our approach using the REDD residential NILM dataset, a relatively small-scale dataset and demonstrate that our methodology significantly enhances the performance of the original transformer model across multiple appliance types.

Towards a Deeper Understanding of Transformer for Residential Non-intrusive Load Monitoring

Transformer models have demonstrated impressive performance in Non-Intrusive Load Monitoring (NILM) applications in recent years. Despite their success, existing studies have not thoroughly examined the impact of various hyper-parameters on model performance, which is crucial for advancing high-performing transformer models. In this work, a comprehensive series of experiments have been conducted to analyze the influence of these hyper-parameters in the context of residential NILM. This study delves into the effects of the number of hidden dimensions in the attention layer, the number of attention layers, the number of attention heads, and the dropout ratio on transformer performance. Furthermore, the role of the masking ratio has explored in BERT-style transformer training, providing a detailed investigation into its impact on NILM tasks. Based on these experiments, the optimal hyper-parameters have been selected and used them to train a transformer model, which surpasses the performance of existing models. The experimental findings offer valuable insights and guidelines for optimizing transformer architectures, aiming to enhance their effectiveness and efficiency in NILM applications. It is expected that this work will serve as a foundation for future research and development of more robust and capable transformer models for NILM.

ResEMGNet: A Lightweight Residual Deep Learning Architecture for Neuromuscular Disorder Detection from Raw EMG Signals

Amyotrophic Lateral Sclerosis (ALS) and Myopathy are debilitating neuromuscular disorders that demand accurate and efficient diagnostic approaches. In this study, we harness the power of deep learning techniques to detect ALS and Myopathy. Convolutional Neural Networks (CNNs) have emerged as powerful tools in this context. We present ResEMGNet, designed to identify ALS and Myopathy directly from raw electromyography (EMG) signals. Unlike traditional methods that require intricate handcrafted feature extraction, ResEMGNet takes raw EMG data as input, reducing computational complexity and enhancing practicality. Our approach was rigorously evaluated using various metrics in comparison to existing methods. ResEMGNet exhibited exceptional subject-independent performance, achieving an impressive overall three-class accuracy of 94.43\%.

EMG Signal Classification for Neuromuscular Disorders with Attention-Enhanced CNN

Amyotrophic Lateral Sclerosis (ALS) and Myopathy present considerable challenges in the realm of neuromuscular disorder diagnostics. In this study, we employ advanced deep-learning techniques to address the detection of ALS and Myopathy, two debilitating conditions. Our methodology begins with the extraction of informative features from raw electromyography (EMG) signals, leveraging the Log-spectrum, and Delta Log spectrum, which capture the frequency contents, and spectral and temporal characteristics of the signals. Subsequently, we applied a deep-learning model, SpectroEMG-Net, combined with Convolutional Neural Networks (CNNs) and Attention for the classification of three classes. The robustness of our approach is rigorously evaluated, demonstrating its remarkable performance in distinguishing among the classes: Myopathy, Normal, and ALS, with an outstanding overall accuracy of 92\%. This study marks a contribution to addressing the diagnostic challenges posed by neuromuscular disorders through a data-driven, multi-class classification approach, providing valuable insights into the potential for early and accurate detection.