Hello, I wonder is there a difference between evaluating SC1 in the leaderboard phase and the final submission phase? If any, would you explain what is the difference? (ex. hidden target) Thank you!

Created by Jungsoo Park JJumSSu
Dear @allawayr Thank you for your reply. I appreciate it. Best, Park
Hi there, There is no difference in the data - unfortunately, we don't have enough to use independent datasets for the leaderboard and final phases. The difference is the scoring approach - we scored on a subset of null models in the leaderboard round, effectively adding a small bit of noise to the scores. The goal of this is to prevent data leakage. The final round is scored on a full set of null models to calculate the most accurate p-value - this is your final score. The prediction question, is however, exactly the same for both rounds. Hope this helps! Best, Robert

Difference between evaluating SC1 in the leaderboard phase and the final submission phase? page is loading…