We find out there is one extraordinary participant, Stephen Capuzzi, University of North Carolina, with an extremely high CI 0.99, but not as high as other evaluations (rmse 1.0734, pearson 0.5363, spearman 0.5353, f1 0.5569, average_AUC 0.7636). Is it realistic? Tsai-Min

Created by TSAI-MIN (??) CHEN (?) chentsaimin
Hi @unc2015, We do have a fix for this; it turned out to be a little more nuanced of an issue than I initially thought. Given the proximity to the end of the Round 1, and the fact that many teams have already submitted, we are going to modify the scoring harness for Round 2 and leave the Round 1 scoring harness as-is. For Round 1, I will be verifying/recalculating the CIs for all submissions after submissions are closed tomorrow. The other 5 metrics on the leaderboard are not affected by this issue. For the time being, please submit your prediction file in the same row order as the template file. Apologies for any inconvenience, and thank you for participating in the Challenge! Cheers, Robert
Has this logic issue been fixed? Or do I always have to submit my results in that same format?
Hi @unc2015, this score (CI = 0.999) is incorrect and is caused by the ordering if your submission file; the other 5 metrics are unaffected by this, but the CI will be incorrect if the submission file order doesn't match the template order (you can see more about this on the wiki section [3.4.1](https://www.synapse.org/#!Synapse:syn15667962/wiki/583675)). Our submission validation logic didn't test for this scenario and allowed your file to proceed to scoring - apologies for the inconvenience. If you resubmit your prediction file in the same order as the round 1 template file, you will get an accurate score for CI. Thanks for your understanding, Robert
Thank you for bringing this to our attention. I ran the scoring functions on my computer with your prediction submission and was unable to reproduce the CI score for your prediction that came up on the leaderboard. The other 5 scores were the same on my computer as they were on the leaderboard. We'll look into this further! Apologies for any inconvenience.
CI, to my understanding, is a ranking index. So if my compounds are ranked in the absolute right order, this would make sense. But, yes, even I was perplexed by such a high score.
Hi, Thanks for pointing it out, we will look into it.

[CI 0.99]Are there anythings wrong in the scoring system? page is loading…