Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation

Dynamic contextualised word embeddings represent the temporal semantic variations of words. We propose a method for learning dynamic contextualised word embeddings by time-adapting a pretrained Masked Language Model (MLM) using time-sensitive templates. Given two snapshots $C_1$ and $C_2$ of a corpora taken respectively at two distinct timestamps $T_1$ and $T_2$, we first propose an unsupervised method to select (a) pivot terms related to both $C_1$ and $C_2$, and (b) anchor terms that are associated with a specific pivot term in each individual snapshot. We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms. Moreover, we propose an automatic method to learn time-sensitive templates from $C_1$ and $C_2$, without requiring any human supervision. Next, we use the generated prompts to adapt a pretrained MLM to $T_2$ by fine-tuning it on the prompts. Experimental results show that our proposed method significantly reduces the perplexity of test sentences selected from $T_2$, thereby outperforming the current state-of-the-art dynamic contextualised word embedding methods.