Automated damage detection is an integral component of each structural health monitoring (SHM) system. Typically, measurements from various sensors are collected and reduced to damage-sensitive features, and diagnostic values are generated by statistically evaluating the features. Since changes in data do not only result from damage, it is necessary to determine the confounding factors (environmental or operational variables) and to remove their effects from the measurements or features. Many existing methods for correcting confounding effects are based on different types of mean regression. This neglects potential changes in higher-order statistical moments, but in particular, the output covariances are essential for generating reliable diagnostics for damage detection. This article presents an approach to explicitly quantify the changes in the covariance, using conditional covariance matrices based on a non-parametric, kernel-based estimator. The method is applied to the Munich Test Bridge and the KW51 Railway Bridge in Leuven, covering both raw sensor measurements (acceleration, strain, inclination) and extracted damage-sensitive features (natural frequencies). The results show that covariances between different vibration or inclination sensors can significantly change due to temperature changes, and the same is true for natural frequencies. To highlight the advantages, it is explained how conditional covariances can be combined with standard approaches for damage detection, such as the Mahalanobis distance and principal component analysis. As a result, more reliable diagnostic values can be generated with fewer false alarms.