dataset collection

We're Afraid Language Models Aren't Modeling Ambiguity

We build a benchmark to evaluate LM understanding of ambiguity, which is an intrinsic feature of language, and find that the task remains extremely challenging, including for GPT-4

Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, Yejin Choi

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Large “instruction-tuned” language models (finetuned to respond to instructions) have demonstrated a remarkable ability to …

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi

WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation

We introduce a paradigm for dataset creation based on human and machine collaboration, and demonstrate its empirical effectiveness for collecting a new large-scale NLI dataset

Alisa Liu, Swabha Swayamdipta, Noah A. Smith, Yejin Choi

CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense

An adversarially-constructed dataset for common sense QA

Michael Chen, Mike D’Arcy, Alisa Liu, Jared Fernandez, Doug Downey