Dear organizers,
I'd like to clear my confusion regarding the scoring on the validation phase:
1. Is my understanding correct that for Ch question 1 & 2, no scores will be returned during the final round, and only during the scoring round will we receive the scores (but then no further submission will be allowed)?
" Each team will be allowed one final submissions per Challenge question 1 and 2. For teams submitting more than two predictions, only the last two submissions will be scored."
Do you mean "..., only the last submission for each challenge question 1 and 2 will be scored"?
2. How does the scoring work for Ch question 3? We have 3 submissions during the final round of this challenge, but will we receive scores of the smaller validation sizes (15% and 30%) BEFORE the submission period closes (to refine our model for the final-final submission on the 100% validation size)?
Also, I think Rules 3 and 4 on the Final submission page still use the old deadlines.
Thanks again!
Mushthofa
Created by Mushthofa Mushthofa mushthofa Challenge Question 3 timing is rather non standard. We'll post leader boards for each validation round in that challenge so you'll be able to see your scores. Thanks for the response Mike.
How about for Challenge question 3? It wouldn't make sense to have all the scores for the sub-sampled validation set (15% and 30%) also AFTER the final phase is over, because it wouldn't matter anyway, or am I missing something here?
Dear Mushtofa,
Thank you for pointing this out. I have updated wiki to correct these errors. You are correct that you will not receive e-mails with your score in it but you should receive an email saying it was correctly scored. Once the validation round closes the leaderboard will be posted with the finalized scores in it.