Sequential Function-Space Variational Inference via Gaussian Mixture Approximation

Continual learning is learning from a sequence of tasks with the aim of learning new tasks without forgetting old tasks. Sequential function-space variational inference (SFSVI) is a continual learning method based on variational inference which uses a Gaussian variational distribution to approximate the distribution of the outputs of a finite number of selected inducing points. Since the posterior distribution of a neural network is multi-modal, a Gaussian distribution could only match one mode of the posterior distribution, and a Gaussian mixture distribution could be used to better approximate the posterior distribution. We propose an SFSVI method which uses a Gaussian mixture variational distribution. We also compare different types of variational inference methods with and without a fixed pre-trained feature extractor. We find that in terms of final average accuracy, Gaussian mixture methods perform better than Gaussian methods and likelihood-focused methods perform better than prior-focused methods.

RKFNet: A Novel Neural Network Aided Robust Kalman Filter

Driven by the filtering challenges in linear systems disturbed by non-Gaussian heavy-tailed noise, the robust Kalman filters (RKFs) leveraging diverse heavy-tailed distributions have been introduced. However, the RKFs rely on precise noise models, and large model errors can degrade their filtering performance. Also, the posterior approximation by the employed variational Bayesian (VB) method can further decrease the estimation precision. Here, we introduce an innovative RKF method, the RKFNet, which combines the heavy-tailed-distribution-based RKF framework with the deep learning (DL) technique and eliminates the need for the precise parameters of the heavy-tailed distributions. To reduce the VB approximation error, the mixing-parameter-based function and the scale matrix are estimated by the incorporated neural network structures. Also, the stable training process is achieved by our proposed unsupervised scheduled sampling (USS) method, where a loss function based on the Student's t (ST) distribution is utilised to overcome the disturbance of the noise outliers and the filtering results of the traditional RKFs are employed as reference sequences. Furthermore, the RKFNet is evaluated against various RKFs and recurrent neural networks (RNNs) under three kinds of heavy-tailed measurement noises, and the simulation results showcase its efficacy in terms of estimation accuracy and efficiency.

Robust Kalman Filters Based on the Sub-Gaussian $α$-stable Distribution

Motivated by filtering tasks under a linear system with non-Gaussian heavy-tailed noise, various robust Kalman filters (RKFs) based on different heavy-tailed distributions have been proposed. Although the sub-Gaussian $\alpha$-stable (SG$\alpha$S) distribution captures heavy tails well and is applicable in various scenarios, its potential has not yet been explored in RKFs. The main hindrance is that there is no closed-form expression of its mixing density. This paper proposes a novel RKF framework, RKF-SG$\alpha$S, where the signal noise is assumed to be Gaussian and the heavy-tailed measurement noise is modelled by the SG$\alpha$S distribution. The corresponding joint posterior distribution of the state vector and auxiliary random variables is approximated by the Variational Bayesian (VB) approach. Also, four different minimum mean square error (MMSE) estimators of the scale function are presented. The first two methods are based on the Importance Sampling (IS) and Gauss-Laguerre quadrature (GLQ), respectively. In contrast, the last two estimators combine a proposed Gamma series (GS) based method with the IS and GLQ estimators and hence are called GSIS and GSGL. Besides, the RKF-SG$\alpha$S is compared with the state-of-the-art RKFs under three kinds of heavy-tailed measurement noises, and the simulation results demonstrate its estimation accuracy and efficiency.