Hi Thomas,
I see on the leaderboard 2, many submissions got AUROC > AUPR. I think it is strange because scores on leaderboard 1 look normal as all AUROC < AUPR. Precision-Recall curve usually apply to imbalanced databases with very few positive cases. In such cases AUROC values are close to 0.5, if problem is not easy like our questions here, and evaluation of models is not easy. I personally never experienced AUPR > AUROC!
Thanks,
Samad
Created by S J SAJA I don't think we disagree on the null expectations, however I don't understand why you think the scores are incorrect. If you are referring to the p-value calculations, these have been removed in order to identify the source of a bug in the computation of the permutation p-values. However, I am confident in the actual scores themselves. These have been checked through independent validation, using separate code in a separate language written by a separate individual. Solly,
Yes, The expected AUPR under the null is the proportion of true positives. However, the AUPR for a random model is not 0! Depends on the portion of positive samples in database AUPR for a random model will change. For example for a balanced database AUPR for a random predictor is 0.5 not 0! So, I think scores on leaderboard 1 looks correct but leaderboard 2 may not.
Best,
Samad Samad-
The expected AUPR under the null is the proportion of true positives. As such, can be > 0.5 (the null expectation for the AUROC) when the proportion of true positives is > 0.5.
Solly