The PASCAL Visual Object Classes Challenge: A Retrospective

Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, Andrew Zisserman

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

The PASCAL Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008–2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community’s progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, 2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges.
Original languageEnglish
Pages (from-to)98-136
Number of pages39
JournalInternational Journal of Computer Vision
Volume111
Issue number1
Early online date25 Jun 2014
DOIs
Publication statusPublished - 1 Jan 2015

Keywords / Materials (for Non-textual outputs)

  • Database
  • Benchmark
  • Object recognition
  • Object detection
  • Segmentation

Fingerprint

Dive into the research topics of 'The PASCAL Visual Object Classes Challenge: A Retrospective'. Together they form a unique fingerprint.

Cite this