Rotary Outliers and Rotary Offset Features in Large Language Models

Transformer-based Large Language Models (LLMs) rely on positional encodings to provide sequence position information to their attention mechanism. Rotary Positional Encodings (RoPE), which encode relative position by rotating queries and keys, have become widely used in modern LLMs. We study the features and patterns that emerge in queries and keys when using rotary embeddings. Our analysis reveals consistent patterns within the same model across layers and attention heads and across different models and architectures. We present and apply analysis techniques and show how the queries and keys use RoPE to construct various attention patterns, including attention sinks. We find and analyze outliers across models in queries and keys and find that they are likely to be found in rotary features with partial cycles. We derive bounds that tell us what rotary frequencies are likely to be selected as outlier features and at what minimum angle the query-key rotary pairs in these features tend to be above and verify the bounds empirically with models of significant architectural differences.

AM-RRT*: Informed Sampling-based Planning with Assisting Metric

In this paper, we present a new algorithm that extends RRT* and RT-RRT* for online path planning in complex, dynamic environments. Sampling-based approaches often perform poorly in environments with narrow passages, a feature common to many indoor applications of mobile robots as well as computer games. Our method extends RRT-based sampling methods to enable the use of an assisting distance metric to improve performance in environments with obstacles. This assisting metric, which can be any metric that has better properties than the Euclidean metric when line of sight is blocked, is used in combination with the standard Euclidean metric in such a way that the algorithm can reap benefits from the assisting metric while maintaining the desirable properties of previous RRT variants - namely probabilistic completeness in tree coverage and asymptotic optimality in path length. We also introduce a new method of targeted rewiring, aimed at shortening search times and path lengths in tasks where the goal shifts repeatedly. We demonstrate that our method offers considerable improvements over existing multi-query planners such as RT-RRT* when using diffusion distance as an assisting metric; finding near-optimal paths with a decrease in search time of several orders of magnitude. Experimental results show planning times reduced by 99.5% and path lengths by 9.8% over existing real-time RRT planners in a variety of environments.