Hi,
How will the validation phase be organized?
* Will there be a new set of validation data?
* Will the leaderboard phase data be released?
* Is there a limit on the number of submissions for the validation phase, or do you consider the latest/best during the period?
Best, Matthias
Created by Matthias Lienhard mats2704 @Jacoberts
Thanks,
Is the standard of the day based on the pacific time?
and can I submit both subchallenge 1 and 2 per day?
thanks. Thanks, @Jacoberts for the clarifications. It seems that we got confused because we could not think of the logical purpose of 1-day submission limitation to validation queue since only the last submission is used. It is a low possibility that someone will make hundreds of submission to the validation queue if only last is used. Dear Sanghoon,
You may make one submission each day, but only the final submission will be counted in the final score. If you submit model A on the 14th, model B on the 15th, and model C on the 16th, we will take the score of model C as your "final score," and ignore the scores of models A and B.
Best,
Jacob @Jacoberts
I'm sorry but I'm a little confused.
Can I submit a final submission by the 16th, one day each day during the validation phase, and is this all recorded in the final score?
thanks
sanghoon Yes that helped! Thank you!
Best,
Joshua Dear Sanghoon: We do not take the best score. We take the score of the final submission.
Dear Joshua: From 3/2 to 3/16, the validation round, you will be able to submit your models to different "queues:"
- *validation queue*: you may submit to the validation queue once per day, you will not get your score or logs, and your scores will not be posted anywhere. We will take your final submission as your final score.
- *leaderboard queue*: you may continue submitting to the leaderboard queue, for testing purposes. You'll get your scores in email, but the leaderboard will not be updated.
I hope that helps!
Best,
Jacob Sry one more question: Will it still be possible to submit for testing dataset of leaderboard phase? Would you mind clarifying once again: Is it possible to submit once a day in validation phase and will the score be published on leaderboard?
Best
Joshua @Jacoberts
Dear Jacoberts,
Does that mean our team can submit once a day during the final round, and you pick the best performance of final round submissions?
thanks
sanghoon Dear Sanghoon and Matthias,
Correction to the above answer: you may make at most one (successful) submission to the validation queue *per day* in the validation round. We will use the final submission for your score.
Sorry for the confusion.
Best,
Jacob @Jacoberts
hi jacoberts,
What do you mean by that last word? Can I submit an unlimited number of VALIDATION codes?
thanks Hi Matthias,
There is going to be an announcement later today about the end of the leaderboard phase and the beginning of the validation phase, which should address these questions. But in short:
> Will there be a new set of validation data?
Yes, submissions will be evaluated against a never-before-seen dataset from the BeatAML group. That brings us to a total of three datasets - training, leaderboard, and validation.
> Will the leaderboard phase data be released?
Yep, it'll be available in the Files tab.
> Is there a limit on the number of submissions for the validation phase, or do you consider the latest/best during the period?
**Updated - see below** I believe we use the final submission during the validation phase. Note that, although you may technically be able to submit multiple times, you cannot view your logs or your score - all you hear back is whether the model completed successfully or failed.
Best,
Jacob