On a set of features which should be fairly uninformative, the subchallenge two scoring code seems to achieve unrealistic results (AUPRC > 0.99) for most of the tasks.
I'm not too certain why this is the case -- it looks like it could be from overfitting. Is anyone else seeing similar behaviour with the scoring code?
On an unrelated note, when will subchallenge 1 results be released?
Created by Max Wang maxwg Thanks Larsson,
I've created a project with the file here and shared it with you and Phil
https://www.synapse.org/#!Synapse:syn11238078/wiki/484445
Hi Max:
Max, can you upload and share your features that are causing this issue with with me @larssono
and @phil on Synapse and we will take a look.
@yuanfang.guan, thank you for your help. We will have preliminary results sooner to let top performers plan their trips to the DREAM conference next month.
Thanks,
Larsson
I don't see such behavior, unless it is from the tied example issue, which sometimes does exaggerate auprc by tens. As a matter of fact, due to the multi-category in sub2.1, even perfect ranking prediction values/or features should not get perfect score. So I am >99% sure there is something wrong with what you did.
a good sanity check on your side is, if you print a set of random features, do you see the same performance?
Sub 1 final result will be released when all evaluations are finalized and winning solution(s) reproduced, which usually takes abut 4-6 weeks, but could also be as long as 3-6 months.
Drop files to upload
Unexpected Subchallenge 2 Scoring Code Behaviour page is loading…