To understand double descent, we need to understand VC theory

We analyze generalization performance of over-parameterized learning methods for classification, under VC-theoretical framework. Recently, practitioners in Deep Learning discovered ‘double descent’ phenomenon, when large networks can fit perfectly available training data, and at the same time, achieve good generalization for future (test) data. The current consensus view is that VC-theoretical results cannot account for good generalization performance of Deep Learning networks. In contrast, this paper shows that double descent can be explained by VC-theoretical concepts, such as VC-dimension and Structural Risk Minimization. We also present empirical results showing that double descent generalization curves can be accurately modeled using classical VC-generalization bounds. Proposed VC-theoretical analysis enables better understanding of generalization curves for data sets with different statistical characteristics, such as low vs high-dimensional data and noisy data. In addition, we analyze generalization performance of transfer learning using pre-trained Deep Learning networks.