Mechanistic understanding and validation of large AI models with SemanticLens

Unlike human-engineered systems such as aeroplanes, where each component's role and dependencies are well understood, the inner workings of AI models remain largely opaque, hindering verifiability and undermining trust. This paper introduces SemanticLens, a universal explanation method for neural networks that maps hidden knowledge encoded by components (e.g., individual neurons) into the semantically structured, multimodal space of a foundation model such as CLIP. In this space, unique operations become possible, including (i) textual search to identify neurons encoding specific concepts, (ii) systematic analysis and comparison of model representations, (iii) automated labelling of neurons and explanation of their functional roles, and (iv) audits to validate decision-making against requirements. Fully scalable and operating without human input, SemanticLens is shown to be effective for debugging and validation, summarizing model knowledge, aligning reasoning with expectations (e.g., adherence to the ABCDE-rule in melanoma classification), and detecting components tied to spurious correlations and their associated training data. By enabling component-level understanding and validation, the proposed approach helps bridge the "trust gap" between AI models and traditional engineered systems. We provide code for SemanticLens on https://github.com/jim-berend/semanticlens and a demo on https://semanticlens.hhi-research-insights.eu.

Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning

The classification of medical images is a pivotal aspect of disease diagnosis, often enhanced by deep learning techniques. However, traditional approaches typically focus on unimodal medical image data, neglecting the integration of diverse non-image patient data. This paper proposes a novel Cross-Graph Modal Contrastive Learning (CGMCL) framework for multimodal medical image classification. The model effectively integrates both image and non-image data by constructing cross-modality graphs and leveraging contrastive learning to align multimodal features in a shared latent space. An inter-modality feature scaling module further optimizes the representation learning process by reducing the gap between heterogeneous modalities. The proposed approach is evaluated on two datasets: a Parkinson's disease (PD) dataset and a public melanoma dataset. Results demonstrate that CGMCL outperforms conventional unimodal methods in accuracy, interpretability, and early disease prediction. Additionally, the method shows superior performance in multi-class melanoma classification. The CGMCL framework provides valuable insights into medical image classification while offering improved disease interpretability and predictive capabilities.

Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification

Although binary classification is a well-studied problem, training reliable classifiers under severe class imbalance remains a challenge. Recent techniques mitigate the ill effects of imbalance on training by modifying the loss functions or optimization methods. We observe that different hyperparameter values on these loss functions perform better at different recall values. We propose to exploit this fact by training one model over a distribution of hyperparameter values--instead of a single value--via Loss Conditional Training (LCT). Experiments show that training over a distribution of hyperparameters not only approximates the performance of several models but actually improves the overall performance of models on both CIFAR and real medical imaging applications, such as melanoma and diabetic retinopathy detection. Furthermore, training models with LCT is more efficient because some hyperparameter tuning can be conducted after training to meet individual needs without needing to retrain from scratch.

Image Class Translation Distance: A Novel Interpretable Feature for Image Classification

We propose a novel application of image translation networks for image classification and demonstrate its potential as a more interpretable alternative to conventional black box classification networks. We train a network to translate images between possible classes, and then quantify translation distance, i.e. the degree of alteration needed to conform an image to one class or another. These translation distances can then be examined for clusters and trends, and can be fed directly to a simple classifier (e.g. a support vector machine, SVM), providing comparable accuracy compared to a conventional end-to-end convolutional neural network classifier. In addition, visual inspection of translated images can reveal class-specific characteristics and biases in the training sets, such as visual artifacts that are more frequently observed in one class or another. We demonstrate the approach on a toy 2-class scenario, apples versus oranges, and then apply it to two medical imaging tasks: detecting melanoma from photographs of pigmented lesions and classifying 6 cell types in a bone marrow biopsy smear. This novel application of image-to-image networks shows the potential of the technology to go beyond imagining different stylistic changes and to provide greater insight into image classification and medical imaging datasets.

Skin Cancer Detection utilizing Deep Learning: Classification of Skin Lesion Images using a Vision Transformer

Skin cancer detection still represents a major challenge in healthcare. Common detection methods can be lengthy and require human assistance which falls short in many countries. Previous research demonstrates how convolutional neural networks (CNNs) can help effectively through both automation and an accuracy that is comparable to the human level. However, despite the progress in previous decades, the precision is still limited, leading to substantial misclassifications that have a serious impact on people's health. Hence, we employ a Vision Transformer (ViT) that has been developed in recent years based on the idea of a self-attention mechanism, specifically two configurations of a pre-trained ViT. We generally find superior metrics for classifying skin lesions after comparing them to base models such as decision tree classifier and k-nearest neighbor (KNN) classifier, as well as to CNNs and less complex ViTs. In particular, we attach greater importance to the performance of melanoma, which is the most lethal type of skin cancer. The ViT-L32 model achieves an accuracy of 91.57% and a melanoma recall of 58.54%, while ViT-L16 achieves an accuracy of 92.79% and a melanoma recall of 56.10%. This offers a potential tool for faster and more accurate diagnoses and an overall improvement for the healthcare sector.

Robust Melanoma Thickness Prediction via Deep Transfer Learning enhanced by XAI Techniques

This study focuses on analyzing dermoscopy images to determine the depth of melanomas, which is a critical factor in diagnosing and treating skin cancer. The Breslow depth, measured from the top of the granular layer to the deepest point of tumor invasion, serves as a crucial parameter for staging melanoma and guiding treatment decisions. This research aims to improve the prediction of the depth of melanoma through the use of machine learning models, specifically deep learning, while also providing an analysis of the possible existance of graduation in the images characteristics which correlates with the depth of the melanomas. Various datasets, including ISIC and private collections, were used, comprising a total of 1162 images. The datasets were combined and balanced to ensure robust model training. The study utilized pre-trained Convolutional Neural Networks (CNNs). Results indicated that the models achieved significant improvements over previous methods. Additionally, the study conducted a correlation analysis between model's predictions and actual melanoma thickness, revealing a moderate correlation that improves with higher thickness values. Explainability methods such as feature visualization through Principal Component Analysis (PCA) demonstrated the capability of deep features to distinguish between different depths of melanoma, providing insight into the data distribution and model behavior. In summary, this research presents a dual contribution: enhancing the state-of-the-art classification results through advanced training techniques and offering a detailed analysis of the data and model behavior to better understand the relationship between dermoscopy images and melanoma thickness.

Web-based Melanoma Detection

Melanoma is the most aggressive form of skin cancer, and early detection can significantly increase survival rates and prevent cancer spread. However, developing reliable automated detection techniques is difficult due to the lack of standardized datasets and evaluation methods. This study introduces a unified melanoma classification approach that supports 54 combinations of 11 datasets and 24 state-of-the-art deep learning architectures. It enables a fair comparison of 1,296 experiments and results in a lightweight model deployable to the web-based MeshNet architecture named Mela-D. This approach can run up to 33x faster by reducing parameters 24x to yield an analogous 88.8\% accuracy comparable with ResNet50 on previously unseen images. This allows efficient and accurate melanoma detection in real-world settings that can run on consumer-level hardware.

Diagnosis of Skin Cancer Using VGG16 and VGG19 Based Transfer Learning Models

Today, skin cancer is considered as one of the most dangerous and common cancers in the world which demands special attention. Skin cancer may be developed in different types; including melanoma, actinic keratosis, basal cell carcinoma, squamous cell carcinoma, and Merkel cell carcinoma. Among them, melanoma is more unpredictable. Melanoma cancer can be diagnosed at early stages increasing the possibility of disease treatment. Automatic classification of skin lesions is a challenging task due to diverse forms and grades of the disease, demanding the requirement of novel methods implementation. Deep convolution neural networks (CNN) have shown an excellent potential for data and image classification. In this article, we inspect skin lesion classification problem using CNN techniques. Remarkably, we present that prominent classification accuracy of lesion detection can be obtained by proper designing and applying of transfer learning framework on pre-trained neural networks, without any requirement for data enlargement procedures i.e. merging VGG16 and VGG19 architectures pre-trained by a generic dataset with modified AlexNet network, and then, fine-tuned by a subject-specific dataset containing dermatology images. The convolution neural network was trained using 2541 images and, in particular, dropout was used to prevent the network from overfitting. Finally, the validity of the model was checked by applying the K-fold cross validation method. The proposed model increased classification accuracy by 3% (from 94.2% to 98.18%) in comparison with other methods.

Comparative Analysis of Radiomic Features and Gene Expression Profiles in Histopathology Data Using Graph Neural Networks

This study leverages graph neural networks to integrate MELC data with Radiomic-extracted features for melanoma classification, focusing on cell-wise analysis. It assesses the effectiveness of gene expression profiles and Radiomic features, revealing that Radiomic features, particularly when combined with UMAP for dimensionality reduction, significantly enhance classification performance. Notably, using Radiomics contributes to increased diagnostic accuracy and computational efficiency, as it allows for the extraction of critical data from fewer stains, thereby reducing operational costs. This methodology marks an advancement in computational dermatology for melanoma cell classification, setting the stage for future research and potential developments.

A Comparative Analysis Towards Melanoma Classification Using Transfer Learning by Analyzing Dermoscopic Images

Melanoma is a sort of skin cancer that starts in the cells known as melanocytes. It is more dangerous than other types of skin cancer because it can spread to other organs. Melanoma can be fatal if it spreads to other parts of the body. Early detection is the key to cure, but it requires the skills of skilled doctors to diagnose it. This paper presents a system that combines deep learning techniques with established transfer learning methods to enable skin lesions classification and diagnosis of melanoma skin lesions. Using Convolutional Neural Networks, it presents a method for categorizing melanoma images into benign and malignant images in this research (CNNs). Researchers used 'Deep Learning' techniques to train an expansive number of photos & essentially to get the expected result deep neural networks to need to be trained with a huge number of parameters as dermoscopic images are sensitive & very hard to classify. This paper, has been emphasized building models with less complexity and comparatively better accuracy with limited datasets & partially fewer deep networks so that the system can predict Melanoma at ease from input dermoscopic images as correctly as possible within devices with less computational power. The dataset has been obtained from ISIC Archive. Multiple pre-trained models ResNet101, DenseNet, EfficientNet, InceptionV3 have been implemented using transfer learning techniques to complete the comparative analysis & every model achieved good accuracy. Before training the models, the data has been augmented by multiple parameters to improve the accuracy. Moreover, the results are better than the previous state-of-the-art approaches & adequate to predict melanoma. Among these architectures, DenseNet performed better than the others which gives a validation accuracy of 96.64%, validation loss of 9.43% & test set accuracy of 99.63%.