Hi,
I have two concern over submission evaluation:
1. Will you consider the leaderboard submission in final ranking of the team?
2. How, it will affect, if we submit binary (0 and 1) value for each bin in final round?
Please clarify
Thanks,
Sandeep
Created by Sandeep Kumar Dhanda skdhanda Please note we've extended the deadline to Sunday Oct 2 midnight Pacific time to account for all these technical issues with submissions.
Please hold off on submissions for the next 5 hours as we try to reset the queues and fix these various issues.
Thanks,
Anshukl. Yes. We are looking into what's causing the stall. I was just clarifying how often the system is expected to score and rank. Everything should be back up soon. Thank you Anshul and Jan for the discussion. I know that ranking is done only once a day, of course.
However, several submissions submitted on September 27 (e.g. 7294552 for ARID3A), i.e., two days ago, to the leaderboard still haven't been ranked.
In addition, as I explained in the previous two posts, the complete leaderboard ranking is empty since yesterday (at least).
I attach a screenshot for clarification.
${imageLink?synapseId=syn7320823&align=None&responsive=true}
No UserIDs, no scores or ranks listed.
Maybe, both problems are related. Just a guess.
Best,
Jan The ranking is only run once a day since it depends on everyone's submissions. But the submissions are scored ASAP through a queue in the order that it arrives. Hi Anshul,
thanks a lot for looking into this.
Please note that my problem is not about the scoring of submission but the ranking. This applies to the ranking for individual factors. But it also applies to the complete ranking across all leaderboard submissions behind the link I supplied.
Nonetheless, there also seems to be a more general problem with the scoring and validation as mentioned by others in another thread (https://www.synapse.org/#!Synapse:syn6131484/discussion/threadId=972).
Best,
Jan Will ask Nathan to look into your submission. Sure. You can certainly submit 0/1 predictions. I just meant they are unlikely to score as well as providing unthresholded probabilities. So might as well not threshold them. Hi Anshul,
regarding your second answer, I do not really see your point. Submitting just values 0 or 1 perfectly fulfils the requirement of scores between 0 and 1. For any submission, it might happen (and most likely will) that two or multiple bins are scores with identical values. And having only two values (0 and 1) is just an extreme case in this regard.
Besides, curves and performance measures may be computed for binary predictions as well. Specifically, the ROC and PR curves will only have one intermediate point, but interpolation is possible even in this case, although the resulting performance is typically worse than for continuous prediction values.
And finally, one could easily circumvent this restriction, for instance, by replacing the prediction values of the first ten bins by random numbers in [0,1], or adding a (very small) random number to all zeros and subtracting a (very small) random number from all ones.
BTW: For me, the leaderboard ranking is currently not working. The wiki page at https://www.synapse.org/#!Synapse:syn6131484/wiki/402030 just displays an empty list. 1. No we dont use leaderboard submissions in the final ranking. Only the final round datasets are evaluated to determine winners.
2. We need real valued predictions between 0 and 1 as stated on the wiki. We cannot compute the prediction performance measures meaningfully with pre-thresholded binary predictions.