Passing a USA National Bar Exam: a First Corpus for Experimentation

Biralatei Fawei, Adam Wyner, Jeff Pan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Bar exams provide a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides an excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper provides a corpus and experimental results with material derived from a real bar exam, treating the problem as a form of textual entailment from the question to an answer. The providers of the bar exam material set the Gold Standard, which is the answer key. The experiments carried out using the `out of the box' the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, but it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future. The corpus facilitates experimentation by other researchers.
Original languageEnglish
Title of host publicationProceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
Place of PublicationPortorož, Slovenia
PublisherEuropean Language Resources Association (ELRA)
Number of pages6
Publication statusPublished - 1 May 2016
Event10th International Conference on Language Resources and Evaluation, LREC 2016 - Portoroz, Slovenia
Duration: 23 May 201628 May 2016


Conference10th International Conference on Language Resources and Evaluation, LREC 2016

Keywords / Materials (for Non-textual outputs)

  • textual entailment
  • bar exam
  • legal reasoning
  • Natural Language Processing


Dive into the research topics of 'Passing a USA National Bar Exam: a First Corpus for Experimentation'. Together they form a unique fingerprint.

Cite this