For round 1 three submissions were allowed. Will this also be the case for round 2?

Created by Jeremy Jacobsen jjacob_cub
We opted to have a 3 submission limit for Round 1 to - as mentioned by @XiaokangWang - avoid overfitting to the testing set. Similarly, this is why round 1 is unranked and has 6 metrics. I'll certainly bring your concerns to the attention of the Challenge co-organizers for further discussion. Cheers, Robert
My thoughts as well. Is there a good reason why Dream challenges aren't administered more like Kaggle (n submissions allowed per day)? I understand not releasing the round one answers, but the strict restriction on the number of submissions is likely discouraging. If you're lucky and your model doesn't suffer from any major logical errors then you can use the first round to tune. On the other hand if your model performs much worse than your cross validation based expectations you'll likely spend the whole first round submitting to verify hunches about errors in your code. With so few submission opportunities it becomes quite possible that one of the low scoring teams is in possession of a winning model that contains a logical error.
I understand that it is useless in practice to overfit a model to the testing set. But if the organizer allows one or two more submissions, I think we are far away from overfitting. It is a research challenge, so we search and research and three submissions might not be sufficient. Besides, if more submissions are allowed, we would see more decent results. I hope the organizer could consider this.

Number of allowed round 2 submissions. page is loading…