Comparing Automatic and Human Evaluation of Local Explanations for Text Classification

Dong Nguyen

doi:10.18653/v1/N18-1097

Comparing Automatic and Human Evaluation of Local Explanations for Text Classification

Dong Nguyen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for generating local explanations. While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable. This paper is a first step towards more robust evaluations of local explanations. We evaluate a variety of local explanation approaches using automatic measures based on word deletion. Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements.

Original language	English
Title of host publication	The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Place of Publication	New Orleans, Louisiana
Publisher	Association for Computational Linguistics
Pages	1069–1078
Number of pages	10
DOIs	https://doi.org/10.18653/v1/N18-1097
Publication status	Published - 6 Jun 2018
Event	16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Hyatt Regency New Orleans Hotel, New Orleans, United States Duration: 1 Jun 2018 → 6 Jun 2018 http://naacl2018.org/

Conference

Conference	16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Abbreviated title	NAACL HLT 2018
Country/Territory	United States
City	New Orleans
Period	1/06/18 → 6/06/18
Internet address	http://naacl2018.org/

Access to Document

10.18653/v1/N18-1097

nguyen-naacl2018Accepted author manuscript, 360 KB

https://aclanthology.coli.uni-saarland.de/events/naacl-2018

Dong Nguyen - ATI Research Fellow
Nguyen, D. (Principal Investigator)
UK-based charities
1/01/17 → 31/12/19
Project: Research

Cite this

@inproceedings{f85349d2c641449a8e42b08da7f49284,

title = "Comparing Automatic and Human Evaluation of Local Explanations for Text Classification",

abstract = "Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for generating local explanations. While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable. This paper is a first step towards more robust evaluations of local explanations. We evaluate a variety of local explanation approaches using automatic measures based on word deletion. Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements. ",

author = "Dong Nguyen",

year = "2018",

month = jun,

day = "6",

doi = "10.18653/v1/N18-1097",

language = "English",

pages = "1069–1078",

booktitle = "The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",

publisher = "Association for Computational Linguistics",

note = "16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 ; Conference date: 01-06-2018 Through 06-06-2018",

url = "http://naacl2018.org/",

}

Nguyen, D 2018, Comparing Automatic and Human Evaluation of Local Explanations for Text Classification. in The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, New Orleans, Louisiana, pp. 1069–1078, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, United States, 1/06/18. https://doi.org/10.18653/v1/N18-1097

Comparing Automatic and Human Evaluation of Local Explanations for Text Classification. / Nguyen, Dong.
The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New Orleans, Louisiana: Association for Computational Linguistics, 2018. p. 1069–1078.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Comparing Automatic and Human Evaluation of Local Explanations for Text Classification

AU - Nguyen, Dong

PY - 2018/6/6

Y1 - 2018/6/6

N2 - Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for generating local explanations. While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable. This paper is a first step towards more robust evaluations of local explanations. We evaluate a variety of local explanation approaches using automatic measures based on word deletion. Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements.

AB - Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for generating local explanations. While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable. This paper is a first step towards more robust evaluations of local explanations. We evaluate a variety of local explanation approaches using automatic measures based on word deletion. Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements.

U2 - 10.18653/v1/N18-1097

DO - 10.18653/v1/N18-1097

M3 - Conference contribution

SP - 1069

EP - 1078

BT - The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

PB - Association for Computational Linguistics

CY - New Orleans, Louisiana

T2 - 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Y2 - 1 June 2018 through 6 June 2018

ER -

Comparing Automatic and Human Evaluation of Local Explanations for Text Classification

Abstract

Conference

Access to Document

Fingerprint

Projects

Dong Nguyen - ATI Research Fellow

Cite this