Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding

Solving linear inverse problems plays a crucial role in numerous applications. Algorithm unfolding based, model-aware data-driven approaches have gained significant attention for effectively addressing these problems. Learned iterative soft-thresholding algorithm (LISTA) and alternating direction method of multipliers compressive sensing network (ADMM-CSNet) are two widely used such approaches, based on ISTA and ADMM algorithms, respectively. In this work, we study optimization guarantees, i.e., achieving near-zero training loss with the increase in the number of learning epochs, for finite-layer unfolded networks such as LISTA and ADMM-CSNet with smooth soft-thresholding in an over-parameterized (OP) regime. We achieve this by leveraging a modified version of the Polyak-Lojasiewicz, denoted PL$^*$, condition. Satisfying the PL$^*$ condition within a specific region of the loss landscape ensures the existence of a global minimum and exponential convergence from initialization using gradient descent based methods. Hence, we provide conditions, in terms of the network width and the number of training samples, on these unfolded networks for the PL$^*$ condition to hold. We achieve this by deriving the Hessian spectral norm of these networks. Additionally, we show that the threshold on the number of training samples increases with the increase in the network width. Furthermore, we compare the threshold on training samples of unfolded networks with that of a standard fully-connected feed-forward network (FFNN) with smooth soft-thresholding non-linearity. We prove that unfolded networks have a higher threshold value than FFNN. Consequently, one can expect a better expected error for unfolded networks than FFNN.

A New Signal Representation Using Complex Conjugate Pair Sums

This letter introduces a real valued summation known as Complex Conjugate Pair Sum (CCPS). The space spanned by CCPS and its one circular downshift is called {\em Complex Conjugate Subspace (CCS)}. For a given positive integer $N\geq3$, there exists $\frac{\varphi(N)}{2}$ CCPSs forming $\frac{\varphi(N)}{2}$ CCSs, where $\varphi(N)$ is the Euler's totient function. We prove that these CCSs are mutually orthogonal and their direct sum form a $\varphi(N)$ dimensional subspace $s_N$ of $\mathbb{C}^N$. We propose that any signal of finite length $N$ is represented as a linear combination of elements from a special basis of $s_d$, for each divisor $d$ of $N$. This defines a new transform named as Complex Conjugate Periodic Transform (CCPT). Later, we compared CCPT with DFT (Discrete Fourier Transform) and RPT (Ramanujan Periodic Transform). It is shown that, using CCPT we can estimate the period, hidden periods and frequency information of a signal. Whereas, RPT does not provide the frequency information. For a complex valued input signal, CCPT offers computational benefit over DFT. A CCPT dictionary based method is proposed to extract non-divisor period information.

Orthogonal and Non-Orthogonal Signal Representations Using New Transformation Matrices Having NPM Structure

In this paper, we introduce two types of real-valued sums known as Complex Conjugate Pair Sums (CCPSs) denoted as CCPS$^{(1)}$ and CCPS$^{(2)}$, and discuss a few of their properties. Using each type of CCPSs and their circular shifts, we construct two non-orthogonal Nested Periodic Matrices (NPMs). As NPMs are non-singular, this introduces two non-orthogonal transforms known as Complex Conjugate Periodic Transforms (CCPTs) denoted as CCPT$^{(1)}$ and CCPT$^{(2)}$. We propose another NPM, which uses both types of CCPSs such that its columns are mutually orthogonal, this transform is known as Orthogonal CCPT (OCCPT). After a brief study of a few OCCPT properties like periodicity, circular shift, etc., we present two different interpretations of it. Further, we propose a Decimation-In-Time (DIT) based fast computation algorithm for OCCPT (termed as FOCCPT), whenever the length of the signal is equal to $2^v,\ v{\in} \mathbb{N}$. The proposed sums and transforms are inspired by Ramanujan sums and Ramanujan Period Transform (RPT). Finally, we show that the period (both divisor and non-divisor) and frequency information of a signal can be estimated using the proposed transforms with a significant reduction in the computational complexity over Discrete Fourier Transform (DFT).

On Complex Conjugate Pair Sums and Complex Conjugate Subspaces

In this letter, we study a few properties of Complex Conjugate Pair Sums (CCPSs) and Complex Conjugate Subspaces (CCSs). Initially, we consider an LTI system whose impulse response is one period data of CCPS. For a given input x(n), we prove that the output of this system is equivalent to computing the first order derivative of x(n). Further, with some constraints on the impulse response, the system output is also equivalent to the second order derivative. With this, we show that a fine edge detection in an image can be achieved using CCPSs as impulse response over Ramanujan Sums (RSs). Later computation of projection for CCS is studied. Here the projection matrix has a circulant structure, which makes the computation of projections easier. Finally, we prove that CCS is shift-invariant and closed under the operation of circular cross-correlation.