Should we optimize hyper-parameters of our models separately for each training set of a LOOCV experiment (this is a subset of a Challenge training set excluding the test example) or are we allowed to optimize on the whole dataset and then do LOOCV using a single optimum? In a standard setting we should do the former but I just wanted to make sure if you are interested in how much the models can overfit the data.
Created by Zafer Aydin zaferaydin Please use the first approach. As you rightly said, we are asking for this to reduce overfitting. Thanks.