TY - GEN
T1 - Validating the Web-based Evaluation of NLG Systems
AU - Koller, Alexander
AU - Striegnitz, Kristina
AU - Byron, Donna
AU - Cassell, Justine
AU - Dale, Robert
AU - Dalzel-Job, Sara
AU - Oberlander, Jon
AU - Moore, Johanna
PY - 2009/8/4
Y1 - 2009/8/4
N2 - The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internet-based approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.
AB - The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internet-based approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.
M3 - Conference contribution
T3 - ACLShort '09
SP - 301
EP - 304
BT - Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
PB - Association for Computational Linguistics
CY - Stroudsburg, PA, USA
T2 - ACL-IJCNLP 2009 Conference
Y2 - 4 August 2009 through 4 August 2009
ER -