RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

We propose RoCoFT, a parameter-efficient fine-tuning method for large-scale language models (LMs) based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium-size LMs like BERT and RoBERTa, and larger LMs like Bloom-7B, Llama2-7B, and Llama2-13B, we show that our method gives comparable or better accuracies than state-of-art PEFT methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from neural tangent kernel theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, are numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the selection strategy for rows and columns as well as the optimal rank for effective implementation of our method.

Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learning

Deep learning models excel at capturing complex representations through sequential layers of linear and non-linear transformations, yet their inherent black-box nature and multi-modal training landscape raise critical concerns about reliability, robustness, and safety, particularly in high-stakes applications. To address these challenges, we introduce YES training bounds, a novel framework for real-time, data-aware certification and monitoring of neural network training. The YES bounds evaluate the efficiency of data utilization and optimization dynamics, providing an effective tool for assessing progress and detecting suboptimal behavior during training. Our experiments show that the YES bounds offer insights beyond conventional local optimization perspectives, such as identifying when training losses plateau in suboptimal regions. Validated on both synthetic and real data, including image denoising tasks, the bounds prove effective in certifying training quality and guiding adjustments to enhance model performance. By integrating these bounds into a color-coded cloud-based monitoring system, we offer a powerful tool for real-time evaluation, setting a new standard for training quality assurance in deep learning.

Deep Learning-Enabled One-Bit DoA Estimation

Unrolled deep neural networks have attracted significant attention for their success in various practical applications. In this paper, we explore an application of deep unrolling in the direction of arrival (DoA) estimation problem when coarse quantization is applied to the measurements. We present a compressed sensing formulation for DoA estimation from one-bit data in which estimating target DoAs requires recovering a sparse signal from a limited number of severely quantized linear measurements. In particular, we exploit covariance recovery from one-bit dither samples. To recover the covariance of transmitted signal, the learned iterative shrinkage and thresholding algorithm (LISTA) is employed fed by one-bit data. We demonstrate that the upper bound of estimation performance is governed by the recovery error of the transmitted signal covariance matrix. Through numerical experiments, we demonstrate the proposed LISTA-based algorithm's capability in estimating target locations. The code employed in this study is available online.

Antenna Failure Resilience: Deep Learning-Enabled Robust DOA Estimation with Single Snapshot Sparse Arrays

Recent advancements in Deep Learning (DL) for Direction of Arrival (DOA) estimation have highlighted its superiority over traditional methods, offering faster inference, enhanced super-resolution, and robust performance in low Signal-to-Noise Ratio (SNR) environments. Despite these advancements, existing research predominantly focuses on multi-snapshot scenarios, a limitation in the context of automotive radar systems which demand high angular resolution and often rely on limited snapshots, sometimes as scarce as a single snapshot. Furthermore, the increasing interest in sparse arrays for automotive radar, owing to their cost-effectiveness and reduced antenna element coupling, presents additional challenges including susceptibility to random sensor failures. This paper introduces a pioneering DL framework featuring a sparse signal augmentation layer, meticulously crafted to bolster single snapshot DOA estimation across diverse sparse array setups and amidst antenna failures. To our best knowledge, this is the first work to tackle this issue. Our approach improves the adaptability of deep learning techniques to overcome the unique difficulties posed by sparse arrays with single snapshot. We conduct thorough evaluations of our network's performance using simulated and real-world data, showcasing the efficacy and real-world viability of our proposed solution. The code and real-world dataset employed in this study are available at https://github.com/ruxinzh/Deep_RSA_DOA.

Collaborative Automotive Radar Sensing via Mixed-Precision Distributed Array Completion

This paper investigates the effects of coarse quantization with mixed precision on measurements obtained from sparse linear arrays, synthesized by a collaborative automotive radar sensing strategy. The mixed quantization precision significantly reduces the data amount that needs to be shared from radar nodes to the fusion center for coherent processing. We utilize the low-rank properties inherent in the constructed Hankel matrix of the mixed-precision array, to recover azimuth angles from quantized measurements. Our proposed approach addresses the challenge of mixed-quantized Hankel matrix completion, allowing for accurate estimation of the azimuth angles of interest. To evaluate the recovery performance of the proposed scheme, we establish a quasi-isometric embedding with a high probability for mixed-precision quantization. The effectiveness of our proposed scheme is demonstrated through numerical results, highlighting successful reconstruction.

Ambiguity Function Shaping in FMCW Automotive Radar

Frequency-modulated continuous wave (FMCW) radar with inter-chirp coding produces high side-lobes in the Doppler and range dimensions of the radar's ambiguity function. The high side-lobes may cause miss-detection due to masking between targets that are at similar range and have large received power difference, as is often the case in automotive scenarios. In this paper, we develop a novel code optimization method that attenuates the side-lobes of the radar's ambiguity function. In particular, we introduce a framework for designing radar transmit sequences by shaping the radar Ambiguity Function (AF) to a desired structure. The proposed approach suppresses the average amplitude of the AF of the transmitted signal in regions of interest by efficiently tackling a longstanding optimization problem. The optimization criterion is quartic in nature with respect to the radar transmit code. A cyclic iterative algorithm is introduced that recasts the quartic problem as a unimodular quadratic problem (UQP) which can be tackled using power-method-like iterations (PMLI). Our numerical results demonstrate the effectiveness of the proposed algorithm in designing sequences with desired AF which is of great interest to the future generations of automotive radar sensors.

Beyond Diagonal RIS: Key to Next-Generation Integrated Sensing and Communications?

Reconfigurable intelligent surface (RIS) have introduced unprecedented flexibility and adaptability toward smart wireless channels. Recent research on integrated sensing and communication (ISAC) systems has demonstrated that RIS platforms enable enhanced signal quality, coverage, and link capacity. In this paper, we explore the application of fully-connected beyond diagonal RIS (BD-RIS) to ISAC systems. BD-RIS introduces additional degrees of freedom by allowing non-zero off-diagonal elements for the scattering matrix, potentially enabling further functionalities and performance enhancements. In particular, we consider the joint design objective of maximizing the weighted sum of the signal-to-noise ratio (SNR) at the radar receiver and communication users by leveraging the extra degrees-of-freedom offered in the BD-RIS setting. These degrees-of-freedom are unleashed by formulating an alternating optimization process over known and auxiliary (latent) variables of such systems. Our numerical results reveal the advantages of deploying BD-RIS in the context of ISAC and the effectiveness of the proposed algorithm by improving the SNR values for both radar and communication users by several orders of magnitude.

Automotive Radar Sensing with Sparse Linear Arrays Using One-Bit Hankel Matrix Completion

The design of sparse linear arrays has proven instrumental in the implementation of cost-effective and efficient automotive radar systems for high-resolution imaging. This paper investigates the impact of coarse quantization on measurements obtained from such arrays. To recover azimuth angles from quantized measurements, we leverage the low-rank properties of the constructed Hankel matrix. In particular, by addressing the one-bit Hankel matrix completion problem through a developed singular value thresholding algorithm, our proposed approach accurately estimates the azimuth angles of interest. We provide comprehensive insights into recovery performance and the required number of one-bit samples. The effectiveness of our proposed scheme is underscored by numerical results, demonstrating successful reconstruction using only one-bit data.

Submodular Optimization for Placement of Intelligent Reflecting Surfaces in Sensing Systems

Intelligent reflecting surfaces (IRS) and their optimal deployment are the new technological frontier in sensing applications. Recently, IRS have demonstrated potential in advancing target estimation and detection. While the optimal phase-shift of IRS for different tasks has been studied extensively in the literature, the optimal placement of multiple IRS platforms for sensing applications is less explored. In this paper, we design the placement of IRS platforms for sensing by maximizing the mutual information. In particular, we use this criterion to determine an approximately optimal placement of IRS platforms to illuminate an area where the target has a hypothetical presence. After demonstrating the submodularity of the mutual information criteria, we tackle the design problem by means of a constant-factor approximation algorithm for submodular optimization. Numerical results are presented to validate the proposed submodular optimization framework for optimal IRS placement with worst case performance bounded to $1-1/e\approx 63 \%$.

Matrix Completion from One-Bit Dither Samples

We explore the impact of coarse quantization on matrix completion in the extreme scenario of dithered one-bit sensing, where the matrix entries are compared with time-varying threshold levels. In particular, instead of observing a subset of high-resolution entries of a low-rank matrix, we have access to a small number of one-bit samples, generated as a result of these comparisons. In order to recover the low-rank matrix using its coarsely quantized known entries, we begin by transforming the problem of one-bit matrix completion (one-bit MC) with time-varying thresholds into a nuclear norm minimization problem. The one-bit sampled information is represented as linear inequality feasibility constraints. We then develop the popular singular value thresholding (SVT) algorithm to accommodate these inequality constraints, resulting in the creation of the One-Bit SVT (OB-SVT). Our findings demonstrate that incorporating multiple time-varying sampling threshold sequences in one-bit MC can significantly improve the performance of the matrix completion algorithm. In pursuit of achieving this objective, we utilize diverse thresholding schemes, namely uniform, Gaussian, and discrete thresholds. To accelerate the convergence of our proposed algorithm, we introduce three variants of the OB-SVT algorithm. Among these variants is the randomized sketched OB-SVT, which departs from using the entire information at each iteration, opting instead to utilize sketched data. This approach effectively reduces the dimension of the operational space and accelerates the convergence. We perform numerical evaluations comparing our proposed algorithm with the maximum likelihood estimation method previously employed for one-bit MC, and demonstrate that our approach can achieve a better recovery performance.