Context-aware Biases for Length Extrapolation

Transformers' ability to generalize to longer sequences than they have been trained on, known as length extrapolation, degrades as sequence length increases. Most of Relative Positional Encoding (RPE) methods address this problem by either adding constant linear biases or learning general biases, lacking the ability to specialize for different sequences. In this work, inspired by ALiBi, we propose Context-aware Biases for Length Extrapolation (Cable), that learns token-specific biases for each head in decoder-based transformers. Cable learns adaptive, context-aware biases, overcoming the limitations of fixed patterns by adding dynamic biases specific to each token in the sequence. Results show that when tested on a sequence length of 1024, a GPT-3 Medium (334M parameters) with our positional encoding, trained on a sequence length of 512, achieves better perplexity (-0.65) than a similar network with sinusoidal positional encoding trained on a sequence length of 1024. This is achieved with 48% lower memory usage, and only 3.5% higher training time. Furthermore, our method notably improves the extrapolation ability of existing RPE methods on the Edu-FineWeb10B and WikiText-103 datasets. Code is available at: https://github.com/axiomlab/Cable

Sparsity Based Multi-Source Robust 3D Localization Using a Moving Receiver

Accurately localizing multiple sources is a critical task with various applications in wireless communications, such as emergency services including natural post-disaster search and rescue operations. However, the scenarios where the receiver is moving, are not addressed by recent studies. This paper tackles the angle of arrival (AOA) 3D-localization problem for multiple sparse signal sources with a moving receiver having limited antennas, potentially outnumbered by the sources. First, an energy detector algorithm is proposed to exploit the sparsity of the signal to eliminate the noisy samples of the signals. Subsequently, elevation and azimuth AOAs of sources are roughly estimated using two dimensional multiple signal classification (2D-MUSIC) method. Next, an algorithm is proposed to refine and estimate the elevation and azimuth AOAs more accurately. To this end, we propose a sparse recovery algorithm to exploit the sparsity feature of the signals. Then, we propose a phase smoothing algorithm to refine the estimations in the output of sparse recovery algorithm. Finally, K-SVD algorithm is employed to find the accurate elevation and azimuth AOAs of sources. For localization, a new multi-source 3D-localization algorithm is proposed to estimate the positions of sources in a sequence of time windows. Extensive simulations are carried out to demonstrate the effectiveness of the proposed framework.