Demystifying Unsupervised Semantic Correspondence Estimation

Mehmet Aygun; Oisin Mac Aodha

doi:10.1007/978-3-031-20056-4_8

Demystifying Unsupervised Semantic Correspondence Estimation

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We explore semantic correspondence estimation through the lens of unsupervised learning. We thoroughly evaluate several recently proposed unsupervised methods across multiple challenging datasets using a standardized evaluation protocol where we vary factors such as the backbone architecture, the pre-training strategy, and the pre-training and finetuning datasets. To better understand the failure modes of these methods, and in order to provide a clearer path for improvement, we provide a new diagnostic framework along with a new performance metric that is better suited to the semantic matching task. Finally, we introduce a new unsupervised correspondence approach which utilizes the strength of pre-trained features while encouraging better matches during training. This results in significantly better matching performance compared to current state-of-the-art methods.

Original language	English
Title of host publication	Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé
Publisher	Springer
Pages	125-142
Number of pages	31
ISBN (Electronic)	978-3-031-20080-9
ISBN (Print)	978-3-031-20079-3
DOIs	https://doi.org/10.1007/978-3-031-20056-4_8
Publication status	Published - 23 Oct 2022
Event	European Conference on Computer Vision 2022 - Israel, Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022 https://eccv2022.ecva.net/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Cham
Volume	13690
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Computer Vision 2022
Abbreviated title	ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22
Internet address	https://eccv2022.ecva.net/

Keywords / Materials (for Non-textual outputs)

semantic correspondence
self-supervised learning

Access to Document

10.1007/978-3-031-20056-4_8Licence: All Rights Reserved

Demystifying Unsupervised_AYGUN_DOA03072022_AFVAccepted author manuscript, 18.5 MBLicence: Creative Commons: Attribution (CC-BY)

https://www.ecva.net/papers/eccv_2022/papers_ECCV/html/499_ECCV_2022_paper.phpLicence: All Rights Reserved
https://link.springer.com/chapter/10.1007/978-3-031-20056-4_8Licence: All Rights Reserved

Cite this

Aygun, M., & Mac Aodha, O. (2022). Demystifying Unsupervised Semantic Correspondence Estimation. In S. Avidan, G. Brostow, & M. Cissé (Eds.), Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X (pp. 125-142). (Lecture Notes in Computer Science; Vol. 13690). Springer. https://doi.org/10.1007/978-3-031-20056-4_8

@inproceedings{89f29a91f6624be5a87fdef15af597dd,

title = "Demystifying Unsupervised Semantic Correspondence Estimation",

abstract = "We explore semantic correspondence estimation through the lens of unsupervised learning. We thoroughly evaluate several recently proposed unsupervised methods across multiple challenging datasets using a standardized evaluation protocol where we vary factors such as the backbone architecture, the pre-training strategy, and the pre-training and finetuning datasets. To better understand the failure modes of these methods, and in order to provide a clearer path for improvement, we provide a new diagnostic framework along with a new performance metric that is better suited to the semantic matching task. Finally, we introduce a new unsupervised correspondence approach which utilizes the strength of pre-trained features while encouraging better matches during training. This results in significantly better matching performance compared to current state-of-the-art methods. ",

keywords = "semantic correspondence, self-supervised learning",

author = "Mehmet Aygun and \{Mac Aodha\}, Oisin",

note = "Funding Information: Acknowledgements. Thanks to Hakan Bilen and Omiros Pantazis for their valuable feedback. This work was in part supported by the Turing 2.0 {\textquoteleft}Enabling Advanced Autonomy{\textquoteright} project funded by the EPSRC and the Alan Turing Institute. Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Computer Vision 2022, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

month = oct,

day = "23",

doi = "10.1007/978-3-031-20056-4\_8",

language = "English",

isbn = "978-3-031-20079-3",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "125--142",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e}",

booktitle = "Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X",

address = "United Kingdom",

url = "https://eccv2022.ecva.net/",

}

Aygun, M & Mac Aodha, O 2022, Demystifying Unsupervised Semantic Correspondence Estimation. in S Avidan, G Brostow & M Cissé (eds), Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X. Lecture Notes in Computer Science, vol. 13690, Springer, pp. 125-142, European Conference on Computer Vision 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-20056-4_8

Demystifying Unsupervised Semantic Correspondence Estimation. / Aygun, Mehmet; Mac Aodha, Oisin.
Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé. Springer, 2022. p. 125-142 (Lecture Notes in Computer Science; Vol. 13690).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Demystifying Unsupervised Semantic Correspondence Estimation

AU - Aygun, Mehmet

AU - Mac Aodha, Oisin

N1 - Funding Information: Acknowledgements. Thanks to Hakan Bilen and Omiros Pantazis for their valuable feedback. This work was in part supported by the Turing 2.0 ‘Enabling Advanced Autonomy’ project funded by the EPSRC and the Alan Turing Institute. Publisher Copyright: © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

PY - 2022/10/23

Y1 - 2022/10/23

N2 - We explore semantic correspondence estimation through the lens of unsupervised learning. We thoroughly evaluate several recently proposed unsupervised methods across multiple challenging datasets using a standardized evaluation protocol where we vary factors such as the backbone architecture, the pre-training strategy, and the pre-training and finetuning datasets. To better understand the failure modes of these methods, and in order to provide a clearer path for improvement, we provide a new diagnostic framework along with a new performance metric that is better suited to the semantic matching task. Finally, we introduce a new unsupervised correspondence approach which utilizes the strength of pre-trained features while encouraging better matches during training. This results in significantly better matching performance compared to current state-of-the-art methods.

AB - We explore semantic correspondence estimation through the lens of unsupervised learning. We thoroughly evaluate several recently proposed unsupervised methods across multiple challenging datasets using a standardized evaluation protocol where we vary factors such as the backbone architecture, the pre-training strategy, and the pre-training and finetuning datasets. To better understand the failure modes of these methods, and in order to provide a clearer path for improvement, we provide a new diagnostic framework along with a new performance metric that is better suited to the semantic matching task. Finally, we introduce a new unsupervised correspondence approach which utilizes the strength of pre-trained features while encouraging better matches during training. This results in significantly better matching performance compared to current state-of-the-art methods.

KW - semantic correspondence

KW - self-supervised learning

U2 - 10.1007/978-3-031-20056-4_8

DO - 10.1007/978-3-031-20056-4_8

M3 - Conference contribution

SN - 978-3-031-20079-3

T3 - Lecture Notes in Computer Science

SP - 125

EP - 142

BT - Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

PB - Springer

T2 - European Conference on Computer Vision 2022

Y2 - 23 October 2022 through 27 October 2022

ER -

Demystifying Unsupervised Semantic Correspondence Estimation

Abstract

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this