Validation of the Practicability of Logical Assessment Formula for Evaluations with Inaccurate Ground-Truth Labels

Logical assessment formula (LAF) is a new theory proposed for evaluations with inaccurate ground-truth labels (IAGTLs) to assess the predictive models for various artificial intelligence applications. However, the practicability of LAF for evaluations with IAGTLs has not yet been validated in real-world practice. In this paper, to address this issue, we applied LAF to tumour segmentation for breast cancer (TSfBC) in medical histopathology whole slide image analysis (MHWSIA). Experimental results and analysis show the validity of LAF for evaluations with IAGTLs in the case of TSfBC and reflect the potentials of LAF applied to MHWSIA.

Experts' cognition-driven ensemble deep learning for external validation of predicting pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer

In breast cancer imaging, there has been a trend to directly predict pathological complete response (pCR) to neoadjuvant chemotherapy (NAC) from histological images based on deep learning (DL). However, it has been a commonly known problem that the constructed DL-based models numerically have better performances in internal validation than in external validation. The primary reason for this situation lies in that the distribution of the external data for validation is different from the distribution of the training data for the construction of the predictive model. In this paper, we aim to alleviate this situation with a more intrinsic approach. We propose an experts' cognition-driven ensemble deep learning (ECDEDL) approach for external validation of predicting pCR to NAC from histological images in breast cancer. The proposed ECDEDL, which takes the cognition of both pathology and artificial intelligence experts into consideration to improve the generalization of the predictive model to the external validation, more intrinsically approximates the working paradigm of a human being which will refer to his various working experiences to make decisions. The proposed ECDEDL approach was validated with 695 WSIs collected from the same center as the primary dataset to develop the predictive model and perform the internal validation, and 340 WSIs collected from other three centers as the external dataset to perform the external validation. In external validation, the proposed ECDEDL approach improves the AUCs of pCR prediction from 61.52(59.80-63.26) to 67.75(66.74-68.80) and the Accuracies of pCR prediction from 56.09(49.39-62.79) to 71.01(69.44-72.58). The proposed ECDEDL was quite effective for external validation, numerically more approximating the internal validation.

Experts' cognition-driven safe noisy labels learning for precise segmentation of residual tumor in breast cancer

Precise segmentation of residual tumor in breast cancer (PSRTBC) after neoadjuvant chemotherapy is a fundamental key technique in the treatment process of breast cancer. However, achieving PSRTBC is still a challenge, since the breast cancer tissue and tumor cells commonly have complex and varied morphological changes after neoadjuvant chemotherapy, which inevitably increases the difficulty to produce a predictive model that has good generalization with machine learning. To alleviate this situation, in this paper, we propose an experts' cognition-driven safe noisy labels learning (ECDSNLL) approach. In the concept of safe noisy labels learning, which is a typical type of safe weakly supervised learning, ECDSNLL is constructed by integrating the pathology experts' cognition about identifying residual tumor in breast cancer and the artificial intelligence experts' cognition about data modeling with provided data basis. We show the advantages of the proposed ECDSNLL approach and its promising potentials in addressing PSRTBC. We also release a better predictive model for achieving PSRTBC, which can be leveraged to promote the development of related application software.

Logical Assessment Formula and its Principles for Evaluations without Accurate Ground-Truth Labels

Logical assessment formula (LAF) was proposed for evaluations without accurate ground-truth labels (AGTL). In this paper, we reveal the principles of LAF via comprehensive theoretical analyses. From the revealed principles, we summarize the practicability of LAF: 1) LAF can be reasonably applied for evaluations without AGTL on a more difficult task, just acting like usual strategies for evaluations with AGTL; 2) LAF can be applied for evaluations without AGTL from the logical perspective on an easier task, unable to be acting like usual strategies for evaluations with AGTL. Experimental results and analyses of LAF applied on tumour segmentation for breast cancer support the practicability of LAF summarized from the revealed principles.

One-Step Abductive Multi-Target Learning with Diverse Noisy Samples

One-step abductive multi-target learning (OSAMTL) was proposed to handle complex noisy labels. In this paper, giving definition of diverse noisy samples (DNS), we propose one-step abductive multi-target learning with DNS (OSAMTL-DNS) to expand the original OSAMTL to a wider range of tasks that handle complex noisy labels.

Discussion of Ensemble Learning under the Era of Deep Learning

Due to the dominant position of deep learning (mostly deep neural networks) in various artificial intelligence applications, recently, ensemble learning based on deep neural networks (ensemble deep learning) has shown significant performances in improving the generalization of learning system. However, since modern deep neural networks usually have millions to billions of parameters, the time and space overheads for training multiple base deep learners and testing with the ensemble deep learner are far greater than that of traditional ensemble learning. Though several algorithms of fast ensemble deep learning have been proposed to promote the deployment of ensemble deep learning in some applications, further advances still need to be made for many applications in specific fields, where the developing time and computing resources are usually restricted or the data to be processed is of large dimensionality. An urgent problem needs to be solved is how to take the significant advantages of ensemble deep learning while reduce the required time and space overheads so that many more applications in specific fields can benefit from it. For the alleviation of this problem, it is necessary to know about how ensemble learning has developed under the era of deep learning. Thus, in this article, we present discussion focusing on data analyses of published works, the methodology and unattainability of traditional ensemble learning, and recent developments of ensemble deep learning. We hope this article will be helpful to realize the technical challenges faced by future developments of ensemble learning under the era of deep learning.

Handling Noisy Labels via One-Step Abductive Multi-Target Learning

Learning from noisy labels is an important concern because of the lack of accurate ground-truth labels in plenty of real-world scenarios. In practice, various approaches for this concern first make corrections corresponding to potentially noisy-labeled instances, and then update predictive model with information of the made corrections. However, in specific areas, such as medical histopathology whole slide image analysis (MHWSIA), it is often difficult or even impossible for experts to manually achieve the noisy-free ground-truth labels which leads to labels with heavy noise. This situation raises two more difficult problems: 1) the methodology of approaches making corrections corresponding to potentially noisy-labeled instances has limitations due to the heavy noise existing in labels; and 2) the appropriate evaluation strategy for validation/testing is unclear because of the great difficulty in collecting the noisy-free ground-truth labels. In this paper, we focus on alleviating these two problems. For the problem 1), we present a one-step abductive multi-target learning framework (OSAMTLF) that imposes a one-step logical reasoning upon machine learning via a multi-target learning procedure to abduct the predictions of the learning model to be subject to our prior knowledge. For the problem 2), we propose a logical assessment formula (LAF) that evaluates the logical rationality of the outputs of an approach by estimating the consistencies between the predictions of the learning model and the logical facts narrated from the results of the one-step logical reasoning of OSAMTLF. Applying OSAMTLF and LAF to the Helicobacter pylori (H. pylori) segmentation task in MHWSIA, we show that OSAMTLF is able to abduct the machine learning model achieving logically more rational predictions, which is beyond the capability of various state-of-the-art approaches for learning from noisy labels.

Moderately supervised learning: definition and framework

Supervised learning (SL) has achieved remarkable success in numerous artificial intelligence applications. In the current literature, by referring to the properties of the ground-truth labels prepared for a training data set, SL is roughly categorized as fully supervised learning (FSL) and weakly supervised learning (WSL). However, solutions for various FSL tasks have shown that the given ground-truth labels are not always learnable, and the target transformation from the given ground-truth labels to learnable targets can significantly affect the performance of the final FSL solutions. Without considering the properties of the target transformation from the given ground-truth labels to learnable targets, the roughness of the FSL category conceals some details that can be critical to building the optimal solutions for some specific FSL tasks. Thus, it is desirable to reveal these details. This article attempts to achieve this goal by expanding the categorization of FSL and investigating the subtype that plays the central role in FSL. Taking into consideration the properties of the target transformation from the given ground-truth labels to learnable targets, we first categorize FSL into three narrower subtypes. Then, we focus on the subtype moderately supervised learning (MSL). MSL concerns the situation where the given ground-truth labels are ideal, but due to the simplicity in annotation of the given ground-truth labels, careful designs are required to transform the given ground-truth labels into learnable targets. From the perspectives of definition and framework, we comprehensively illustrate MSL to reveal what details are concealed by the roughness of the FSL category. Finally, discussions on the revealed details suggest that MSL should be given more attention.