A Deep Ensemble Learning Approach to Lung CT Segmentation for COVID-19 Severity Assessment

We present a novel deep learning approach to categorical segmentation of lung CTs of COVID-19 patients. Specifically, we partition the scans into healthy lung tissues, non-lung regions, and two different, yet visually similar, pathological lung tissues, namely, ground-glass opacity and consolidation. This is accomplished via a unique, end-to-end hierarchical network architecture and ensemble learning, which contribute to the segmentation and provide a measure for segmentation uncertainty. The proposed framework achieves competitive results and outstanding generalization capabilities for three COVID-19 datasets. Our method is ranked second in a public Kaggle competition for COVID-19 CT images segmentation. Moreover, segmentation uncertainty regions are shown to correspond to the disagreements between the manual annotations of two different radiologists. Finally, preliminary promising correspondence results are shown for our private dataset when comparing the patients' COVID-19 severity scores (based on clinical measures), and the segmented lung pathologies. Code and data are available at our repository: https://github.com/talbenha/covid-seg

The Impact of an Inter-rater Bias on Neural Network Training

The problem of inter-rater variability is often discussed in the context of manual labeling of medical images. It is assumed to be bypassed by automatic model-based approaches for image segmentation which are considered `objective', providing single, deterministic solutions. However, the emergence of data-driven approaches such as Deep Neural Networks (DNNs) and their application to supervised semantic segmentation - brought this issue of raters' disagreement back to the front-stage. In this paper, we highlight the issue of inter-rater bias as opposed to random inter-observer variability and demonstrate its influence on DNN training, leading to different segmentation results for the same input images. In fact, lower Dice scores are calculated if training and test segmentations are of different raters. Moreover, we demonstrate that inter-rater bias in the training examples is amplified when considering the segmentation predictions for the test data. We support our findings by showing that a classifier-DNN trained to distinguish between raters based on their manual annotations performs better when the automatic segmentation predictions rather than the raters' annotations were tested. For this study, we used the ISBI 2015 Multiple Sclerosis (MS) challenge dataset, which includes annotations by two raters with different levels of expertise. The results obtained allow us to underline a worrisome clinical implication of a DNN bias induced by an inter-rater bias during training. Specially, we show that the differences in MS-lesion load estimates increase when the volume calculations are done based on the DNNs' segmentation predictions instead of the manual annotations used for training.

CT-GAN: Malicious Tampering of 3D Medical Imagery using Deep Learning

In 2018, clinics and hospitals were hit with numerous attacks leading to significant data breaches and interruptions in medical services. An attacker with access to medical records can do much more than hold the data for ransom or sell it on the black market. In this paper, we show how an attacker can use deep learning to add or remove evidence of medical conditions from volumetric (3D) medical scans. An attacker may perform this act in order to stop a political candidate, sabotage research, commit insurance fraud, perform an act of terrorism, or even commit murder. We implement the attack using a 3D conditional GAN and show how the framework (CT-GAN) can be automated. Although the body is complex and 3D medical scans are very large, CT-GAN achieves realistic results and can be executed in milliseconds. To evaluate the attack, we focus on injecting and removing lung cancer from CT scans. We show how three expert radiologists and a state-of-the-art deep learning AI could not differentiate between tampered and non-tampered scans. We also evaluate state-of-the-art countermeasures and propose our own. Finally, we discuss the possible attack vectors on modern radiology networks and demonstrate one of the attack vectors on an active CT scanner.