I don't understand! Evaluation methods for natural language explanations

Miruna Clinciu; Arash Eshghi; Helen Hastie

I don't understand! Evaluation methods for natural language explanations

Miruna Clinciu, Arash Eshghi, Helen Hastie

Research output: Contribution to journal › Article › peer-review

Abstract

Explainability of intelligent systems is key for future adoption. While much work is ongoing with regards to developing methods of explaining complex opaque systems, there is little current work on evaluating how effective these explanations are, in particular with respect to the user’s understanding. Natural language (NL) explanations can be seen as an intuitive channel between humans and artificial intelligence systems, in particular for enhancing transparency. This paper presents existing work on how evaluation methods from the field of Natural Language Generation (NLG) can be mapped onto NL explanations. Also, we present a preliminary investigation into the relationship between linguistic features and human evaluation, using a dataset of NL explanations derived from Bayesian Networks.

Original language	English
Pages (from-to)	17-24
Number of pages	8
Journal	CEUR Workshop Proceedings
Volume	2894
Publication status	Published - 2 Jul 2021
Event	SICSA Workshop on eXplainable Artificial Intelligence 2021 - Aberdeen, United Kingdom Duration: 1 Jun 2021 → 1 Jun 2021 https://sites.google.com/view/sicsa-xai-workshop/

Keywords / Materials (for Non-textual outputs)

evaluation
explanations
natural language

Access to Document

I don't understand_CLINCIU_DOA20052021_VOR_CC BYFinal published version, 522 KBLicence: Creative Commons: Attribution (CC-BY)

https://ceur-ws.org/Vol-2894/short3.pdfLicence: Creative Commons: Attribution (CC-BY)

UK Robotics and Artificial Intelligence Hub for Offshore Energy Asset Integrity Management (ORCA)
Vijayakumar, S. (Principal Investigator), Mistry, M. (Co-investigator), Ramamoorthy, R. (Co-investigator) & Williams, C. (Co-investigator)
EPSRC
1/10/17 → 31/03/22
Project: Research

Cite this

@article{f5a9856c88a148a99e4532a3864f04fa,

title = "I don't understand! Evaluation methods for natural language explanations",

abstract = "Explainability of intelligent systems is key for future adoption. While much work is ongoing with regards to developing methods of explaining complex opaque systems, there is little current work on evaluating how effective these explanations are, in particular with respect to the user{\textquoteright}s understanding. Natural language (NL) explanations can be seen as an intuitive channel between humans and artificial intelligence systems, in particular for enhancing transparency. This paper presents existing work on how evaluation methods from the field of Natural Language Generation (NLG) can be mapped onto NL explanations. Also, we present a preliminary investigation into the relationship between linguistic features and human evaluation, using a dataset of NL explanations derived from Bayesian Networks.",

keywords = "evaluation, explanations, natural language",

author = "Miruna Clinciu and Arash Eshghi and Helen Hastie",

note = "Funding Information: This work was supported by the EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems at Heriot-Watt University and the University of Edinburgh. Clinciu{\textquoteright}s PhD is funded by Schlumberger Cambridge Research Limited (EP/L016834/1, 2018-2021). This work was also supported by the EPSRC ORCA Hub (EP/R026173/1, 2017-2021) and UKRI Trustworthy Autonomous Systems Node on Trust (EP/V026682/1, 2020-2024). ; SICSA Workshop on eXplainable Artificial Intelligence 2021, SISCA XAI 2021 ; Conference date: 01-06-2021 Through 01-06-2021",

year = "2021",

month = jul,

day = "2",

language = "English",

volume = "2894",

pages = "17--24",

journal = "CEUR Workshop Proceedings",

issn = "1613-0073",

publisher = "CEUR-WS",

url = "https://sites.google.com/view/sicsa-xai-workshop/",

}

TY - JOUR

T1 - I don't understand! Evaluation methods for natural language explanations

AU - Clinciu, Miruna

AU - Eshghi, Arash

AU - Hastie, Helen

N1 - Funding Information: This work was supported by the EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems at Heriot-Watt University and the University of Edinburgh. Clinciu’s PhD is funded by Schlumberger Cambridge Research Limited (EP/L016834/1, 2018-2021). This work was also supported by the EPSRC ORCA Hub (EP/R026173/1, 2017-2021) and UKRI Trustworthy Autonomous Systems Node on Trust (EP/V026682/1, 2020-2024).

PY - 2021/7/2

Y1 - 2021/7/2

N2 - Explainability of intelligent systems is key for future adoption. While much work is ongoing with regards to developing methods of explaining complex opaque systems, there is little current work on evaluating how effective these explanations are, in particular with respect to the user’s understanding. Natural language (NL) explanations can be seen as an intuitive channel between humans and artificial intelligence systems, in particular for enhancing transparency. This paper presents existing work on how evaluation methods from the field of Natural Language Generation (NLG) can be mapped onto NL explanations. Also, we present a preliminary investigation into the relationship between linguistic features and human evaluation, using a dataset of NL explanations derived from Bayesian Networks.

AB - Explainability of intelligent systems is key for future adoption. While much work is ongoing with regards to developing methods of explaining complex opaque systems, there is little current work on evaluating how effective these explanations are, in particular with respect to the user’s understanding. Natural language (NL) explanations can be seen as an intuitive channel between humans and artificial intelligence systems, in particular for enhancing transparency. This paper presents existing work on how evaluation methods from the field of Natural Language Generation (NLG) can be mapped onto NL explanations. Also, we present a preliminary investigation into the relationship between linguistic features and human evaluation, using a dataset of NL explanations derived from Bayesian Networks.

KW - evaluation

KW - explanations

KW - natural language

M3 - Article

SN - 1613-0073

VL - 2894

SP - 17

EP - 24

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

T2 - SICSA Workshop on eXplainable Artificial Intelligence 2021

Y2 - 1 June 2021 through 1 June 2021

ER -

I don't understand! Evaluation methods for natural language explanations

Abstract

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Projects

UK Robotics and Artificial Intelligence Hub for Offshore Energy Asset Integrity Management (ORCA)

Cite this