Model Selection and Error Estimation in a Nutshell

Statistical learning theory answers these questions by deriving non-asymptotic bounds on the generalization error of a model or, in other words, by upper bounding the true error of the learned model based just on quantities computed on the available data.