Firstly, Hoping that the leader board will be open for submissions.
Second, If i am not mistaken,We are only given 5 submissions to validate our models.Could this cap be removed , and maybe restrictions could be placed on the number of submissions that can be made on a single day.Maybe 1 or 2 per day?
This would really help in validating our models.
Thoughts?
Created by decentmakeover Please do see our reply to requests about increasing the cap on submissions [here](https://www.synapse.org/#!Synapse:syn20545111/discussion/threadId=6550&replyId=20972)  I concur.
There should be more submission attempts.
The lack of clarity around the scoring metrics makes it even more difficult (impossible) to evaluate solutions ourselves. This challenge is very complex and the limit of 5 leader-board submissions does not do justification to the complexity of the task.Requesting the organisers to loosen the restriction on the number of submissions.
Thanks. Hi Lars, we are still doing some test runs and getting a baseline model established. I am hopeful that we will have it open by the end of the month. Should we be delayed, we can extend the deadline for the challenge as well, but we will notify the group with an email when the leaderboards are open.
Thanks,
jim What's the current ETA for opening the Leaderboard? Still blank: https://www.synapse.org/#!Synapse:syn20545111/wiki/597246 @lars.ericson Thank you for your suggestion , i will contact a few people who might be interested in forming a team.
Thanks @decentmakeover thank you for the invitation. I already have a partner for this challenge. You can post a teaming thread on the Discussion board to find other solvers who are looking to collaborate. There are 187 people registered, you can see the list here: https://www.synapse.org/#!Team:3392643 I expect that quite a few of those will be unteamed. The 187 people should be quite diverse in terms of goals and strengths. You can state your goals and strengths in a post and look for people with complementary skills. [Irrelevant to the current discussion] Also @lars.ericson would you like to team up? I agree with @decentmakeover. It's a lot more fun and motivating to see the Leaderboard move and have a chance to work harder and resubmit. Sponsors may see initial scores in the 30's and solvers could push that to 80's or 90's with unlimited submissions. If it's 5 and done then sponsors are likely to see much lower final performance. Since retraining and code rework is involved in each submission, the scoring engine will not be overloaded. You can reduce load in this case by setting a daily limit like 4 submissions per day. I understand the organiser's concerns, Agreed, we can cross validate on our set.But it is more motivating when we can submit our results and also see the progress made by other participants.Please do consider removing the cap and even if it may not be unlimited submissions, at the least it could be increased to some number in the double digits.
Thanks Hi @decentmakeover and @lars.ericson, regarding limiting submissions. We will be limiting the number of submission as we do not want data leakage and risk the overall challenge. The current limit of 5 submission is an arbitrary threshold that we have set and we are open to revisiting this cap. We will discuss this on our next organizers call, though I would expect the 5 submission cap for the near future and depending on performance and how teams are feeling about the challenge, we may raise it. We have supplied the full set of training data so you can do training and cross validation on this subset of data, and obviously you are welcome to do as much cross validation on this dataset. I hope this addresses your question and we will revisit this question in the future.
Kind Regards,
JIm
I vote in favor of unlimited Leaderboard submissions. It's up to solvers to assume the risk of overfitting. Hi there,
The leaderboard is not yet open. You can submit your Docker container to the "RA2 Challenge Submission" queue but it will not be run/scored until we work out some technical issues on the compute side.
Regarding your second question, I am tagging @james.costello for his thoughts. Too many submissions can result in overfitting to the leaderboard data, which will negatively affect your score when it comes to the final submission on the new test dataset.