Unbounded Dependency Recovery for Parser Evaluation

Laura Rimell; Stephen Clark; Mark Steedman

Unbounded Dependency Recovery for Parser Evaluation

Laura Rimell, Stephen Clark, Mark Steedman

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

This paper introduces a new parser evaluation corpus containing around 700 sentences annotated with unbounded dependencies, from seven different grammatical constructions. We run a series of off-the-shelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies. The overall results range from 25% accuracy to 59%. These low scores call into question the validity of using Parseval scores as a general measure of parsing capability. We discuss the importance of parsers being able to recover unbounded dependencies, given their relatively low frequency in corpora. We also analyse the various errors made on these constructions by one of the more successful parsers.

Original language	English
Title of host publication	Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL
Publisher	Association for Computational Linguistics
Pages	813-821
Number of pages	9
Publication status	Published - 2009

Access to Document

Rimell Clark et al 2009 Unbounded Dependency Recovery for Parser EvaluationFinal published version, 126 KB

http://www.aclweb.org/anthology/D09-1085

PACO-PLUS: Perception, Action & Cognition through learning of object-action complexes
Steedman, M.
EU government bodies
1/02/06 → 31/01/10
Project: Research

Cite this

Rimell, L., Clark, S., & Steedman, M. (2009). Unbounded Dependency Recovery for Parser Evaluation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL (pp. 813-821). Association for Computational Linguistics. http://www.aclweb.org/anthology/D09-1085

@inproceedings{db2f716cf99946d8af9a8dc91c229925,

title = "Unbounded Dependency Recovery for Parser Evaluation",

abstract = "This paper introduces a new parser evaluation corpus containing around 700 sentences annotated with unbounded dependencies, from seven different grammatical constructions. We run a series of off-the-shelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies. The overall results range from 25% accuracy to 59%. These low scores call into question the validity of using Parseval scores as a general measure of parsing capability. We discuss the importance of parsers being able to recover unbounded dependencies, given their relatively low frequency in corpora. We also analyse the various errors made on these constructions by one of the more successful parsers.",

author = "Laura Rimell and Stephen Clark and Mark Steedman",

year = "2009",

language = "English",

pages = "813--821",

booktitle = "Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL",

publisher = "Association for Computational Linguistics",

}

Rimell, L, Clark, S & Steedman, M 2009, Unbounded Dependency Recovery for Parser Evaluation. in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL. Association for Computational Linguistics, pp. 813-821. <http://www.aclweb.org/anthology/D09-1085>

Unbounded Dependency Recovery for Parser Evaluation. / Rimell, Laura; Clark, Stephen; Steedman, Mark.
Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL. Association for Computational Linguistics, 2009. p. 813-821.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Unbounded Dependency Recovery for Parser Evaluation

AU - Rimell, Laura

AU - Clark, Stephen

AU - Steedman, Mark

PY - 2009

Y1 - 2009

N2 - This paper introduces a new parser evaluation corpus containing around 700 sentences annotated with unbounded dependencies, from seven different grammatical constructions. We run a series of off-the-shelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies. The overall results range from 25% accuracy to 59%. These low scores call into question the validity of using Parseval scores as a general measure of parsing capability. We discuss the importance of parsers being able to recover unbounded dependencies, given their relatively low frequency in corpora. We also analyse the various errors made on these constructions by one of the more successful parsers.

AB - This paper introduces a new parser evaluation corpus containing around 700 sentences annotated with unbounded dependencies, from seven different grammatical constructions. We run a series of off-the-shelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies. The overall results range from 25% accuracy to 59%. These low scores call into question the validity of using Parseval scores as a general measure of parsing capability. We discuss the importance of parsers being able to recover unbounded dependencies, given their relatively low frequency in corpora. We also analyse the various errors made on these constructions by one of the more successful parsers.

M3 - Conference contribution

SP - 813

EP - 821

BT - Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL

PB - Association for Computational Linguistics

ER -

Unbounded Dependency Recovery for Parser Evaluation

Abstract / Description of output

Access to Document

Fingerprint

Projects

PACO-PLUS: Perception, Action & Cognition through learning of object-action complexes

Cite this