Visual Knowledge Tracing

Neehar Kondapaneni, Pietro Perona, Oisin Mac Aodha

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.
Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
PublisherSpringer, Cham
Number of pages17
ISBN (Electronic)978-3-031-19806-9
ISBN (Print)978-3-031-19805-2
Publication statusPublished - 20 Oct 2022
EventEuropean Conference on Computer Vision 2022 - Israel, Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Cham
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceEuropean Conference on Computer Vision 2022
Abbreviated titleECCV 2022
CityTel Aviv
Internet address

Keywords / Materials (for Non-textual outputs)

  • visual classification
  • knowledge tracing
  • human learning


Dive into the research topics of 'Visual Knowledge Tracing'. Together they form a unique fingerprint.

Cite this