Generalized Eigenvalue Based Detection of Signals in Colored Noise: A Sample Deficient Analysis

This paper investigates the signal detection problem in colored noise with an unknown covariance matrix. To be specific, we consider a scenario in which the number of signal bearing samples ($n$) is strictly smaller than the dimensionality of the signal space ($m$). Our test statistic is the leading generalized eigenvalue of the whitened sample covariance matrix (a.k.a. $F$-matrix) which is constructed by whitening the signal bearing sample covariance matrix with noise-only sample covariance matrix. The sample deficiency (i.e., $m>n$) in turn makes this $F$-matrix rank deficient, thereby singular. Therefore, an exact statistical characterization of the leading generalized eigenvalue (l.g.e.) of a singular $F$-matrix is of paramount importance to assess the performance of the detector (i.e., the receiver operating characteristics (ROC)). To this end, we employ the powerful orthogonal polynomial approach to derive a new finite dimensional c.d.f. expression for the l.g.e. of a singular $F$-matrix. It turns out that when the noise only sample covariance matrix is nearly rank deficient and the signal-to-noise ratio is $O(m)$, the ROC profile converges to a limit.