Orthogonal Attention: A Cloze-Style Approach to Negation Scope Resolution

Negation Scope Resolution is an extensively researched problem, which is used to locate the words affected by a negation cue in a sentence. Recent works have shown that simply finetuning transformer-based architectures yield state-of-the-art results on this task. In this work, we look at Negation Scope Resolution as a Cloze-Style task, with the sentence as the Context and the cue words as the Query. We also introduce a novel Cloze-Style Attention mechanism called Orthogonal Attention, which is inspired by Self Attention. First, we propose a framework for developing Orthogonal Attention variants, and then propose 4 Orthogonal Attention variants: OA-C, OA-CA, OA-EM, and OA-EMB. Using these Orthogonal Attention layers on top of an XLNet backbone, we outperform the finetuned XLNet state-of-the-art for Negation Scope Resolution, achieving the best results to date on all 4 datasets we experiment with: BioScope Abstracts, BioScope Full Papers, SFU Review Corpus and the *sem 2012 Dataset (Sherlock).

Resolving the Scope of Speculation and Negation using Transformer-Based Architectures

Speculation is a naturally occurring phenomena in textual data, forming an integral component of many systems, especially in the biomedical information retrieval domain. Previous work addressing cue detection and scope resolution (the two subtasks of speculation detection) have ranged from rule-based systems to deep learning-based approaches. In this paper, we apply three popular transformer-based architectures, BERT, XLNet and RoBERTa to this task, on two publicly available datasets, BioScope Corpus and SFU Review Corpus, reporting substantial improvements over previously reported results (by at least 0.29 F1 points on cue detection and 4.27 F1 points on scope resolution). We also experiment with joint training of the model on multiple datasets, which outperforms the single dataset training approach by a good margin. We observe that XLNet consistently outperforms BERT and RoBERTa, contrary to results on other benchmark datasets. To confirm this observation, we apply XLNet and RoBERTa to negation detection and scope resolution, reporting state-of-the-art results on negation scope resolution for the BioScope Corpus (increase of 3.16 F1 points on the BioScope Full Papers, 0.06 F1 points on the BioScope Abstracts) and the SFU Review Corpus (increase of 0.3 F1 points).

NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution

Negation is an important characteristic of language, and a major component of information extraction from text. This subtask is of considerable importance to the biomedical domain. Over the years, multiple approaches have been explored to address this problem: simple rule-based systems, Machine Learning classifiers, Conditional Random Field Models, CNNs and more recently BiLSTMs. In this paper, we look at applying Transfer Learning to this problem. First, we extensively review previous literature addressing Negation Detection and Scope Resolution across the 3 datasets that have gained popularity over the years: BioScope Corpus, the Sherlock dataset, and the SFU Review Corpus. We then explore the decision choices involved with using BERT, a popular transfer learning model, for this task, and report a new state-of-the-art for scope resolution across all 3 datasets. Our model, referred to as NegBERT, achieves a token level F1 score on scope resolution of 92.36 on the Sher-lock dataset, 95.68 on the BioScope Abstracts, 91.24 on the BioScope Full Papers, 90.95 on the SFU dataset, out-performing the previous state-of-the-art by a significant margin. We also analyze the model's generalizability to datasets on which it is not trained.