Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis

Bjorn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, Michael Wojatzki

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Some users of social media are spreading racist, sexist, and otherwise hateful content. For the purpose of training a hate speech detection system, the reliability of the annotations is crucial, but there is no universally agreed-upon definition. We collected potentially hateful messages and asked two groups of internet users to determine whether they were hate speech or not, whether they should be banned or not and to rate their degree of offensiveness. One of the groups was shown a definition prior to completing the survey. We aimed to assess whether hate speech can be annotated reliably, and the extent to which existing definitions are in accordance with subjective ratings. Our results indicate that showing users a definition caused them to partially align their own opinion with the definition but did not improve reliability, which was very low overall. We conclude that the presence of hate speech should perhaps not be considered a binary yes-or-no decision, and raters need more detailed instructions for the annotation.
Original languageEnglish
Title of host publicationNLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication
EditorsStefanie Dipper
PublisherRuhr-Universitat Bochum
Pages6-9
Number of pages4
Publication statusPublished - 9 Nov 2016
Event3rd Workshop on Natural Language Processing for Computer-Mediated Communication / Social Media - Bochum, Germany
Duration: 22 Sep 201622 Sep 2016
https://sites.google.com/site/nlp4cmc2016/

Publication series

NameBochumer Linguistische Arbeitsberichte
Volume17
ISSN (Electronic)2190-0949

Workshop

Workshop3rd Workshop on Natural Language Processing for Computer-Mediated Communication / Social Media
Abbreviated titleNLP4CMC 2016
CountryGermany
CityBochum
Period22/09/1622/09/16
Internet address

Keywords

  • hate speech
  • hate speech detection
  • twitter
  • social media
  • social media analytics
  • natural language processing

Fingerprint Dive into the research topics of 'Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis'. Together they form a unique fingerprint.

Cite this