DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Models

Nick Louloudakis, Perry Gibson, Jose Cano, Ajitha Rajan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Image recognition tasks typically use deep learning and require enormous processing power, thus relying on hardware accelerators like GPUs and TPUs for fast, timely processing. Failure in real-time image recognition tasks can occur due to sub-optimal mapping on hardware accelerators during model deployment, which may lead to timing uncertainty and erroneous behavior. Mapping on hardware accelerators is done using multiple software components like deep learning frameworks, compilers, and device libraries, that we refer to as the computational environment. Owing to the increased use of image recognition tasks in safety-critical applications like autonomous driving and medical imaging, it is imperative to assess their robustness to changes in the computational environment, as the impact of parameters like deep learning frameworks, compiler optimizations, and hardware devices on model performance and correctness is not yet well understood. In this paper we present a differential testing framework, DeltaNN, that allows us to assess the impact of different computational environment parameters on the performance of image recognition models during deployment, post training. DeltaNN generates different implementations of a given image recognition model for variations in environment parameters, namely, deep learning frameworks, compiler optimizations and hardware devices and analyzes differences in model performance as a result. Using DeltaNN, we conduct an empirical study of robustness analysis of three popular image recognition models using the ImageNet dataset. We report the impact in terms of misclassifications and inference time differences across different settings. In total, we observed up to 72% output label differences across deep learning frameworks, and up to 81% unexpected performance degradation in terms of inference time, when applying compiler optimizations.
Original languageEnglish
Title of host publication2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)
PublisherIEEE
Pages414-424
Number of pages11
ISBN (Electronic)979-8-3503-2783-0
ISBN (Print)979-8-3503-2784-7
DOIs
Publication statusPublished - 11 Dec 2023
Event39th IEEE International Conference on Software Maintenance and Evolution - Bogota, Colombia
Duration: 1 Oct 20236 Oct 2023
Conference number: 39
https://conf.researchr.org/home/icsme-2023

Publication series

NameIEEE International Conference on Software Maintenance and Evolution (ICSME)
PublisherIEEE
ISSN (Print)1063-6773
ISSN (Electronic)2576-3148

Conference

Conference39th IEEE International Conference on Software Maintenance and Evolution
Abbreviated titleICSME 2023
Country/TerritoryColombia
CityBogota
Period1/10/236/10/23
Internet address

Fingerprint

Dive into the research topics of 'DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Models'. Together they form a unique fingerprint.

Cite this