Understanding Deep Neural Networks via Linear Separability of Hidden Layers

In this paper, we measure the linear separability of hidden layer outputs to study the characteristics of deep neural networks. In particular, we first propose Minkowski difference based linear separability measures (MD-LSMs) to evaluate the linear separability degree of two points sets. Then, we demonstrate that there is a synchronicity between the linear separability degree of hidden layer outputs and the network training performance, i.e., if the updated weights can enhance the linear separability degree of hidden layer outputs, the updated network will achieve a better training performance, and vice versa. Moreover, we study the effect of activation function and network size (including width and depth) on the linear separability of hidden layers. Finally, we conduct the numerical experiments to validate our findings on some popular deep networks including multilayer perceptron (MLP), convolutional neural network (CNN), deep belief network (DBN), ResNet, VGGNet, AlexNet, vision transformer (ViT) and GoogLeNet.

AL-iGAN: An Active Learning Framework for Tunnel Geological Reconstruction Based on TBM Operational Data

In tunnel boring machine (TBM) underground projects, an accurate description of the rock-soil types distributed in the tunnel can decrease the construction risk ({\it e.g.} surface settlement and landslide) and improve the efficiency of construction. In this paper, we propose an active learning framework, called AL-iGAN, for tunnel geological reconstruction based on TBM operational data. This framework contains two main parts: one is the usage of active learning techniques for recommending new drilling locations to label the TBM operational data and then to form new training samples; and the other is an incremental generative adversarial network for geological reconstruction (iGAN-GR), whose weights can be incrementally updated to improve the reconstruction performance by using the new samples. The numerical experiment validate the effectiveness of the proposed framework as well.

GAN-MDF: A Method for Multi-fidelity Data Fusion in Digital Twins

The Internet of Things (IoT) collects real-time data of physical systems, such as smart factory, intelligent robot and healtcare system, and provide necessary support for digital twins. Depending on the quality and accuracy, these multi-source data are divided into different fidelity levels. High-fidelity (HF) responses describe the system of interest accurately but are computed costly. In contrast, low-fidelity (LF) responses have a low computational cost but could not meet the required accuracy. Multi-fidelity data fusion (MDF) methods aims to use massive LF samples and small amounts of HF samples to develop an accurate and efficient model for describing the system with a reasonable computation burden. In this paper, we propose a novel generative adversarial network for MDF in digital twins (GAN-MDF). The generator of GAN-MDF is composed of two sub-networks: one extracts the LF features from an input; and the other integrates the input and the extracted LF features to form the input of the subsequent discriminator. The discriminator of GAN-MDF identifies whether the generator output is a real sample generated from HF model. To enhance the stability of GAN-MDF's training, we also introduce the supervised-loss trick to refine the generator weights during each iteration of the adversarial training. Compared with the state-of-the-art methods, the proposed GAN-MDF has the following advantages: 1) it performs well in the case of either nested or unnested sample structure; 2) there is no specific assumption on the data distribution; and 3) it has high robustness even when very few HF samples are provided. The experimental results also support the validity of GAN-MDF.