Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance

Given a set of pre-trained models, how can we quickly and accurately find the most useful pre-trained model for a downstream task? Transferability measurement is to quantify how transferable is a pre-trained model learned on a source task to a target task. It is used for quickly ranking pre-trained models for a given task and thus becomes a crucial step for transfer learning. Existing methods measure transferability as the discrimination ability of a source model for a target data before transfer learning, which cannot accurately estimate the fine-tuning performance. Some of them restrict the application of transferability measurement in selecting the best supervised pre-trained models that have classifiers. It is important to have a general method for measuring transferability that can be applied in a variety of situations, such as selecting the best self-supervised pre-trained models that do not have classifiers, and selecting the best transferring layer for a target task. In this work, we propose TMI (TRANSFERABILITY MEASUREMENT WITH INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure transferability. We view transferability as the generalization of a pre-trained model on a target task by measuring intra-class feature variance. Intra-class variance evaluates the adaptability of the model to a new task, which measures how transferable the model is. Compared to previous studies that estimate how discriminative the models are, intra-class variance is more accurate than those as it does not require an optimal feature extractor and classifier. Extensive experiments on real-world datasets show that TMI outperforms competitors for selecting the top-5 best models, and exhibits consistently better correlation in 13 out of 17 cases.

How to Pick the Best Source Data? Measuring Transferability for Heterogeneous Domains

Given a set of source data with pre-trained classification models, how can we fast and accurately select the most useful source data to improve the performance of a target task? We address the problem of measuring transferability for heterogeneous domains, where the source and the target data have different feature spaces and distributions. We propose Transmeter, a novel method to efficiently and accurately measure transferability of two datasets. Transmeter utilizes a pre-trained source classifier and a reconstruction loss to increase its efficiency and performance. Furthermore, Transmeter uses feature transformation layers, label-wise discriminators, and a mean distance loss to learn common representations for source and target domains. As a result, Transmeter and its variant give the most accurate performance in measuring transferability, while giving comparable running times compared to those of competitors.