Abstract
In NLP annotation, it is common to have multiple annotators label the text and then obtain the ground truth labels based on the agreement of major annotators. However, annotators are individuals with different backgrounds, and minors’ opinions should not be simply ignored. As annotation tasks become subjective and topics are controversial in modern NLP tasks, we need NLP systems that can represent people’s diverse voices on subjective matters and predict the level of diversity. This paper examines whether the text of the task and annotators’ demographic background information can be used to estimate the level of disagreement among annotators. Particularly, we extract disagreement labels from the annotators’ voting histories in the five subjective datasets, and then fine-tune language models to predict annotators’ disagreement. Our results show that knowing annotators’ demographic information, like gender, ethnicity, and education level, helps predict disagreements. In order to distinguish the disagreement from the inherent controversy from text content and the disagreement in the annotators’ different perspectives, we simulate everyone’s voices with different combinations of annotators’ artificial demographics and examine its variance of the fine-tuned disagreement predictor. Our paper aims to improve the annotation process for more efficient and inclusive NLP systems through a novel disagreement prediction mechanism. Our code and dataset are publicly available.
Original language | English (US) |
---|---|
Title of host publication | AAAI-23 Special Tracks |
Editors | Brian Williams, Yiling Chen, Jennifer Neville |
Publisher | AAAI press |
Pages | 14523-14530 |
Number of pages | 8 |
ISBN (Electronic) | 9781577358800 |
State | Published - Jun 27 2023 |
Event | 37th AAAI Conference on Artificial Intelligence, AAAI 2023 - Washington, United States Duration: Feb 7 2023 → Feb 14 2023 |
Publication series
Name | Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023 |
---|---|
Volume | 37 |
Conference
Conference | 37th AAAI Conference on Artificial Intelligence, AAAI 2023 |
---|---|
Country/Territory | United States |
City | Washington |
Period | 2/7/23 → 2/14/23 |
Bibliographical note
Publisher Copyright:Copyright © 2023, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.