Honors Thesis 2019 - Zhengzhe Yang

FriendsQA: Open-Domain Question Answering Dataset on TV Show Transcripts

Zhengzhe Yang

Highest Honor in Computer Science

Abstract

This thesis presents FriendsQA, a challenging question answering dataset that contains 1,222 dialogues and 10,610 open-domain questions, to tackle machine comprehension on everyday conversations. Each dialogue, involving multiple speakers, is annotated with six types of questions {what, when, why, where, who, how} regarding the dialogue contexts, and the answers are annotated with contiguous spans in the dialogue. A series of crowdsourcing tasks are conducted to ensure good annotation quality, resulting a high inter-annotator agreement of 81.82%. A comprehensive annotation analytics is provided for a deeper understanding in this dataset. Three state-of-the-art QA systems are experimented, R-Net, QANet, and BERT, and evaluated on this dataset. BERT in particular depicts promising results, an accuracy of 74.2% for answer utterance selection and an F1-score of 64.2% for answer span selection, suggesting that the FriendsQA task is hard yet has a great potential of elevating QA research on multiparty dialogue to another level.

Department / School

Computer Science / Emory University

Degree / Year

BS / Spring 2019

Committee

Jinho D. Choi, Computer Science and QTM, Emory University (Chair)
Shun Yan Cheung, Computer Science, Emory University
Jed Brody, Physics, Emory University

Links

Anthology | Paper | Presentation