In the final submission guideline webpage (https://www.synapse.org/#!Synapse:syn6187098/wiki/463671), it is said:
1.B. Each team needs to submit three final submissions for Challenge 3.
So does it mean that every team should participate SC3 and do exactly 3 submissions?
Thanks,
Yichao
Created by yichao li unfashionable Dear All,
Thank you for brining up these points. The validation round structure of challenge question 3 is somewhat non-standard and especially confusing when compared to the round structure of questions 1 and 2.
For challenge question 3, a team's **last successful submission in the last internal validation round will be used for determining top performing teams**. The first two internal validation rounds will **not** be used for determining top performers. In each of these 3 internal validation rounds, a team can submit multiple times; a team's **last not best** score will be posted. Again, only the 3rd internal round with 100% of the data will be used to determine top performing team(s).
Additionally, please remember:
1. it is a violation of challenge rules for a team to have multiple accounts submitting to a challenge question.
2. it is a violation of challenge rules to return patient id's in log files.
Regards,
Mike i am absolutely against release score, @Michael.Mason unless the 30% you release will not be counted towards final. As a matter of fact, the IDs overlapping with sub1 and sub2 leaderboard rounds should also be excluded.
The reason is that one can easily have 2-3 accounts and basically determine which model is the best from the 30%.
I'm not sure what you mean. What is the point of having rounds unless we get a score? Hi Tom,
You mean removing the quota of MAIN queue in EACH internal round of challenge question 3 ?
Implying one can only use Main queue nearly (sub-ch3)???
Many thanks,
Bruce Dear all,
We have decided to remove the quota. Scores are not returned to participants so we will take your LAST submission (NOT BEST) as your final submission.
Best,
Tom Hi,
My responses above only relate to challenge question 3. are you sure?
(for sub1 and sub2) The quota is 1 for this first internal round. After this Wednesday's deadline we will post scores based on the 15% of data, and the second internal round will open and with a quota of 1 and scoring based on 30% of the data. That round goes through November 15th, then we will post those scores and the final internal validation round will open with a quota of 1 and scores based on 100% of the data. This final submission round's scores will be used for determining top performers. I have the same issue described above.
I submitted a final model like previous round, but when trying to submit the final write-up here (https://www.synapse.org/#!Synapse:syn6187098/wiki/463671), it said that I reached my quota.
Thanks,
Jincheng I don't get it. it says only the last submission is counted towards scoring. https://www.synapse.org/#!Synapse:syn6187098/wiki/463671
So I submitted my original one just in case I won't have chance to submit a new one. Now when I want to submit, it says the quota is done.
Dear Yichao,
I have updated the challenge timeline wiki to say there there are 3 internal validation rounds the main validation phase of question 3. Each round allow for **one** successful submission (invalid submissions do not count towards the quota).
Regards,
Mike Hi Mike,
For the 3 submissions for SC3, are there 3 corresponding time slots for them? In other words, can we just submit a docker model 3 times in one day?
Thanks,
Yichao Hi Yichao,
Teams have to make the last submission to participate. I'd recommend checking out the [timeline](https://www.synapse.org/#!Synapse:syn6187098/wiki/449433). If you miss the 1st and 2nd that is fine but you won't be allow to squeeze 3 submissions into the last week.