Picture for Frank Binder

Frank Binder

CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Add code
Dec 22, 2021
Jörg Frohberg, Frank Binder
Figure 1 for CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models
Figure 2 for CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models
Figure 3 for CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models
Figure 4 for CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design and benchmark as well as the accompanying API that supports scoring against a crowd-validated human baseline. We test six state-of-the-art models against our benchmark. Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement.

* 8 pages including references, plus 3 pages appendix 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon