Hi organizers, I have a recently completed Round 3 Inference Submission (id=8507961, rep=syn7835091). It was submitted on 03/21/2017 08:22:57PM. However, I received email this morning informing me that this submission is counted towards Validation Round. This is not our intention. Is there a way to revert this inference submission so that we have a chance to submit our model trained in validation round? Thanks for your time! A.S.

Created by Yiqiu Shen ashen
@alalbiol We understand your concern and would like to support you. Let me clarify that although the score in the validation round is not shared, if your model encounters an error the system **will** notify you as usual (and provide the tail of the log file to help you locate the error). If you send your final submission at the very end of the validation round and an error occurs after the challenge closes, then there is no chance to use this information to correct the error. If you submit your final submission (with the updated weights) well in advance of the end of the validation round, then there is time to learn about any hidden errors and submit again before the challenge closes.
I am sorry to insist but I do not fully understand (probably is my problem with english) I understand that the model WILL be scored and we will not get the score. For this reason my suggestion is if it is possible to submit a model (with the weights of previous rounds), so we can FULLY check that everything works, although we do not get a score, and when we finish the fine-tuning replace the weights and resubmit the model. So we avoid "unpleasant" final surprises Thanks for your patience Regards
Dear Alberto, Apologies for the confusion, but your model WILL be scored, its just that you wont get your scores back. If you get that email that means your submission has been scored. Best, Tom
Hello Our largest worry is that by any kind of error our submission doesn't get scored (it has happened to us in one submission to sc2) and since in the validation round we only have a shot, (and unfortunately "things happens"), we get automatically disqualified Right now we are fine-tuning our model with the new data, but it could be great if we could send a submission with our best model of the previous rounds, so we can check if it can at least finish with no issues. The final submission would be exactly the same code but with the new weights, so we have more changes to at least get scored. For this reason, I would like to ask for your permission to do so, and if everything goes fine, we will submit the final model after this last fine-tuning with the new weights Thanks for your response Alberto
Dear Yaroslav, You are only allowed one submission to the validation round. So lets say you accidentally submit to the validation queue, then we can make your submission invalid because you won't know your score. Once you get your score, that is the score that you will have for the validation round. This way if you decide you want to change your model, we can make your submission invalid and you can submit again. Best, Tom
Hello @thomas.yu Do I understand right that no one will receive his scores before the end of the Validation Round? Is it an already taken decision or still discussed? Personally, I'd like to receive my scores as soon as my model is evaluated for two reasons: not to worry too much that something went wrong and IF something went wrong to be able to resubmit. Thanks, Yaroslav
Hi @thomas.yu, Thanks for your help. I really appreciate it. A.S.
Dear A.S, I emailed you your scores. Best, Tom
Hi @thomas.yu, Thanks for looking into this issue. Is there a way for us to get the score for this submission. It would affect our modeling decision in Validation Round. Thanks, A.S.
Dear Yiqiu, Sorry for the confusion. Your submission is counted towards round 3. I updated the automated emails being sent out to participants to not include scores in case there are mistakes in final submission. Best, Tom
@thomas.yu Can you comment?
Hi @brucehoff, Thanks for taking time look into this. Here is the email: Hello Yiqiu Shen, Your submission "inference" (ID: 8507961) to the Digital Mammography Prediction - Sub-Challenge 1 has been scored: Your scores will be held in private until the leaderboard is released at the end of the Validation Round. Please direct any questions to the challenge forum https://www.synapse.org/#!Synapse:syn4224222/discussion. Sincerely, DM Challenge Scoring Script
> However, I received email this morning informing me that this submission is counted towards Validation Round. Our records show that 8507961 was part of Round 3. Would you please post the email saying that 8507961 is part of the Validation Round?
@thomas.yu @brucehoff @tschaffter

Round 3 Inference Submission was Mistakenly Counted Towards Validation Round page is loading…