Dear Organizers:   Can we have more submissions in the Final Round? According to earlier response, only one submission is allowed for every sub-challenge in the Final Round. I think compare to the leaderboard rounds (3 submission per round), it is too small. The goal of this challenge is to find the best prediction model. But since each submission is complex and length, and many things can go wrong, so we always have to balance the risk (not getting any score) and benefit (more complex model and better prediction). If there is only one submission allowed, we have to tilted heavily toward risk-control mode, which is contradict to competition's ultimate goal: find the best prediction. Especially the final round has much more data than the previous rounds, allow only one shot seems to be a waste to this gold opportunities. Please consider my suggestion of increasing the number of submission in the final round. Thanks!

Created by vacuum
I agree with Vacuum and Alberto as well. Considering there are more data in the final round, we do need more submissions to make sure that we can at least have a score in the end. Sometimes the preprocessing may take a very long time in the scoring phase and having extra submissions will allow us to modify the algorithm according to the result of the first submission.
I totally agree with Vacuum, someone can get totally out of the competition just because the servers are running slower than usual in the final round due to heavy load and the algorithm lasts one more day... We all have seen huge differences in the computers performance these previous days and only this little detail makes difficult to adjust the complexity of the algorithms....
Dear Thomas, thank you very much for forwarding the message!   Please also note that the uniqueness of this competition when discuss my suggestion. It is the first time for a major competition that all training/inference are done in a fully controlled automatic environment, so no human interaction is possible once the job file is submitted. In addition, the amount of data involved are huge and the inference stage can take literally weeks. That means jobs have to be run weeks without any hardware/software glitch/interruption. The vulnerable window caused by hardware/software bug is big, and the consequence is huge: a team might completely lose scoring opportunity because something out of their control and waste months of effort. Thanks!
Dear Vacuum, It is conventional for most challenges to only have one submission in the final round, but I have forwarded your message to other challenge organizers to see what they say. Best, Tom

Can we have more submissions in the Final Round? page is loading…