Selecting pre-trained models for transfer learning with data-centric Meta-features

When applying a neural network to address a new learning problem, it is common to not train the network from scratch, but instead start with a neural network that has already been trained on a related dataset, and then fine-tune this on the data of the target task. This poses the question: which pre-trained network should be selected? In this work, we investigate this problem in the context of three different dataset relationships: same-source, same-domain, and cross-domain. We utilize Meta-Album, which offers an extensive collection of datasets from various unrelated domains. We first split each of the 30 datasets of Meta-Album into a meta-train dataset and meta-test dataset, then create pre-trained models for each meta-train dataset, and finally compare the performances of the pre-trained models in a fine-tuning context when applied to meta-test tasks. We categorize the performances into the three dataset relationship groups and find that the same-source category has the best performance. Then, using meta-features of the meta-train dataset and meta-test tasks, we train statistical meta-models that are employed to select the best pre-trained model for a given meta-test task. Our best meta-model identifies the best-performing model in ~25% of cases. It improves upon a baseline that always selects the best average model by more than 30%.