A metric that estimates how difficult a test image dataset will be to classify by measuring the distance, in the low-dimensional embedding space of a pre-trained image encoder, between the training-domain distribution and the test-domain distribution. A large discriminative difficulty distance signals a domain gap and insufficient training-data diversity, and the metric correlates strongly with the actual error of an independently trained classifier, so it can flag when a model's benchmark accuracy is inflated by same-domain train/test splits and guide the construction of more diverse datasets.
Introduces Discriminative Difficulty Distance (DDD), a metric that quantifies the domain gap between a training dataset and a test dataset and predicts how hard the test data will be to classify. DDD…