Visual Knowledge Tracing

Neehar Kondapaneni; Pietro Perona; Oisin Mac Aodha

doi:10.1007/978-3-031-19806-9_24

Visual Knowledge Tracing

Neehar Kondapaneni, Pietro Perona, Oisin Mac Aodha

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.

Original language	English
Title of host publication	Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Publisher	Springer
Pages	415-431
Number of pages	17
ISBN (Electronic)	978-3-031-19806-9
ISBN (Print)	978-3-031-19805-2
DOIs	https://doi.org/10.1007/978-3-031-19806-9_24
Publication status	Published - 20 Oct 2022
Event	European Conference on Computer Vision 2022 - Israel, Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022 https://eccv2022.ecva.net/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Cham
Volume	13685
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Computer Vision 2022
Abbreviated title	ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22
Internet address	https://eccv2022.ecva.net/

Keywords / Materials (for Non-textual outputs)

visual classification
knowledge tracing
human learning

Access to Document

10.1007/978-3-031-19806-9_24Licence: All Rights Reserved

Visual Knowledge_KONDAPANENI_DOA03072022_AFVAccepted author manuscript, 8.28 MBLicence: Creative Commons: Attribution (CC-BY)

https://link.springer.com/chapter/10.1007/978-3-031-19806-9_24Licence: All Rights Reserved

Enabling advanced autonomy through human-AI collaboration
Fisher, B. (Principal Investigator), Bilen, H. (Co-investigator), Keller, F. (Co-investigator), Lascarides, A. (Co-investigator), Mac Aodha, O. (Co-investigator), Mollica, F. (Co-investigator), N, S. (Co-investigator), Ramamoorthy, R. (Co-investigator), Rovatsos, M. (Co-investigator) & Sevilla-Lara, L. (Co-investigator)
UK-based charities
1/10/21 → 30/06/22
Project: Research

Cite this

Kondapaneni, N., Perona, P., & Mac Aodha, O. (2022). Visual Knowledge Tracing. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV (pp. 415-431). (Lecture Notes in Computer Science; Vol. 13685). Springer. https://doi.org/10.1007/978-3-031-19806-9_24

Kondapaneni, Neehar ; Perona, Pietro ; Mac Aodha, Oisin. / Visual Knowledge Tracing. Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Springer, 2022. pp. 415-431 (Lecture Notes in Computer Science).

@inproceedings{59aab66af56d492b9d378a56d56ec477,

title = "Visual Knowledge Tracing",

abstract = "Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.",

keywords = "visual classification, knowledge tracing, human learning",

author = "Neehar Kondapaneni and Pietro Perona and {Mac Aodha}, Oisin",

note = "Funding Information: Acknowledgements. Thanks to the anonymous reviews for their valuable feedback. This work was in part supported by the Turing 2.0 {\textquoteleft}Enabling Advanced Autonomy{\textquoteright} project funded by the EPSRC and the Alan Turing Institute and also by the Simons Collaboration on the Global Brain. Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Computer Vision 2022, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

month = oct,

day = "20",

doi = "10.1007/978-3-031-19806-9_24",

language = "English",

isbn = "978-3-031-19805-2",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "415--431",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, {Giovanni Maria} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV",

address = "United Kingdom",

url = "https://eccv2022.ecva.net/",

}

Kondapaneni, N, Perona, P & Mac Aodha, O 2022, Visual Knowledge Tracing. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV. Lecture Notes in Computer Science, vol. 13685, Springer, pp. 415-431, European Conference on Computer Vision 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-19806-9_24

Visual Knowledge Tracing. / Kondapaneni, Neehar; Perona, Pietro; Mac Aodha, Oisin.
Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Springer, 2022. p. 415-431 (Lecture Notes in Computer Science; Vol. 13685).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Visual Knowledge Tracing

AU - Kondapaneni, Neehar

AU - Perona, Pietro

AU - Mac Aodha, Oisin

N1 - Funding Information: Acknowledgements. Thanks to the anonymous reviews for their valuable feedback. This work was in part supported by the Turing 2.0 ‘Enabling Advanced Autonomy’ project funded by the EPSRC and the Alan Turing Institute and also by the Simons Collaboration on the Global Brain. Publisher Copyright: © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

PY - 2022/10/20

Y1 - 2022/10/20

N2 - Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.

AB - Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.

KW - visual classification

KW - knowledge tracing

KW - human learning

U2 - 10.1007/978-3-031-19806-9_24

DO - 10.1007/978-3-031-19806-9_24

M3 - Conference contribution

SN - 978-3-031-19805-2

T3 - Lecture Notes in Computer Science

SP - 415

EP - 431

BT - Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer

T2 - European Conference on Computer Vision 2022

Y2 - 23 October 2022 through 27 October 2022

ER -

Visual Knowledge Tracing

Abstract

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Projects

Enabling advanced autonomy through human-AI collaboration

Cite this