Hypothesis Only Baselines in Natural Language Inference

Adam Poliak; Jason Naradowsky; Aparajita Haldar; Rachel Rudinger; Benjamin Van Durme

doi:10.18653/v1/S18-2023

Hypothesis Only Baselines in Natural Language Inference

Adam Poliak
, Jason Naradowsky
, Aparajita Haldar
, Rachel Rudinger
, Benjamin Van Durme

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on 10 distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority-class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.

Original language	English
Title of host publication	Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics
Place of Publication	New Orleans, Louisiana
Publisher	Association for Computational Linguistics
Pages	180-191
Number of pages	12
DOIs	https://doi.org/10.18653/v1/S18-2023
Publication status	Published - 1 Jun 2018
Event	7th Seventh Joint Conference on Lexical and Computational Semantics - New Orleans, United States Duration: 5 Jun 2018 → 6 Jun 2018 Conference number: 7 https://sites.google.com/view/starsem2018/

Conference

Conference	7th Seventh Joint Conference on Lexical and Computational Semantics
Country/Territory	United States
City	New Orleans
Period	5/06/18 → 6/06/18
Internet address	https://sites.google.com/view/starsem2018/

Access to Document

10.18653/v1/S18-2023Licence: Creative Commons: Attribution (CC-BY)

https://aclanthology.org/S18-2023Licence: Creative Commons: Attribution (CC-BY)

Cite this

@inproceedings{cc6776aafac3496c8a8d3bd653e764e4,

title = "Hypothesis Only Baselines in Natural Language Inference",

abstract = "We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on 10 distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority-class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.",

author = "Adam Poliak and Jason Naradowsky and Aparajita Haldar and Rachel Rudinger and \{Van Durme\}, Benjamin",

year = "2018",

month = jun,

day = "1",

doi = "10.18653/v1/S18-2023",

language = "English",

pages = "180--191",

booktitle = "Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics",

publisher = "Association for Computational Linguistics",

note = "7th Seventh Joint Conference on Lexical and Computational Semantics ; Conference date: 05-06-2018 Through 06-06-2018",

url = "https://sites.google.com/view/starsem2018/",

}

Poliak, A, Naradowsky, J, Haldar, A, Rudinger, R & Van Durme, B 2018, Hypothesis Only Baselines in Natural Language Inference. in Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics, New Orleans, Louisiana, pp. 180-191, 7th Seventh Joint Conference on Lexical and Computational Semantics, New Orleans, United States, 5/06/18. https://doi.org/10.18653/v1/S18-2023

Hypothesis Only Baselines in Natural Language Inference. / Poliak, Adam; Naradowsky, Jason; Haldar, Aparajita et al.
Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. New Orleans, Louisiana: Association for Computational Linguistics, 2018. p. 180-191.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Hypothesis Only Baselines in Natural Language Inference

AU - Poliak, Adam

AU - Naradowsky, Jason

AU - Haldar, Aparajita

AU - Rudinger, Rachel

AU - Van Durme, Benjamin

N1 - Conference code: 7

PY - 2018/6/1

Y1 - 2018/6/1

N2 - We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on 10 distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority-class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.

AB - We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on 10 distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority-class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.

U2 - 10.18653/v1/S18-2023

DO - 10.18653/v1/S18-2023

M3 - Conference contribution

SP - 180

EP - 191

BT - Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics

PB - Association for Computational Linguistics

CY - New Orleans, Louisiana

T2 - 7th Seventh Joint Conference on Lexical and Computational Semantics

Y2 - 5 June 2018 through 6 June 2018

ER -

Hypothesis Only Baselines in Natural Language Inference

Abstract

Conference

Access to Document

Fingerprint

Cite this