Dear organizers, Can you please tell me if it is CTRP or GDSC datasets that you use to score the submissions? Thanks. Denise

Created by Denise Duma denise.duma
Hi Chuyen, There are no criteria - anyone (even teams that did not participate in the leaderboard round) - can participate in the final round. The only difference is that the 15 compounds are different, and there is only one submission opportunity, so there is no opportunity in this round to optimize a method to a new test dataset. Cheers, Robert
Dear organizers, I just wanted to follow up with a related question about the scoring. I was wondering if we could get any information about how advancing to the final round is determined? Are there a set number of participants in the next round, or do we need to get a certain prediction score? Thank you!
Thank you Robert!
Hi Chuyen, The two scoring metrics are as follows: > AUROC: We have used a mixture-modeling approach to classify "sensitive" and "non-sensitive" cell lines for each compound with data obtained from the Cancer Therapeutics Response Portal (CTRP). The submitted prediction values will be compared to the classification of each cell line, and we will calculate the Area Under the Receiver Operating Characteristics curve (AUROC) for each compound. For more on the AUROC, read here. The total score for this metric will be the average AUROC across all 15 leaderboard or final round compounds. > Spearman correlation: The submitted prediction values will be inversely correlated to the CTRP Area Under the dose-response Curve (AUC) values for all cell lines. A higher Spearman correlation indicates a more accurate correlation of increasing confidence values to decreasing AUC values. This correlation will be calculated for each compound, and then averaged across all compounds within the leaderboard or final phase to generate an average Spearman correlation value. For AUROC, we are not using the raw AUC values but rather the classifications as "sensitive" or "resistant". For Spearman, we are correlating the predicted sensitivities (0-1) from you to the "raw" CTRP AUC values (0-22). The CTRP AUC range doesn't really matter that much in this case, as the rank will be unchanged no matter how it is scaled. Best, Robert
Dear organizer @allawayr , Could you please kindly provide some hints to normalize AUC scores from CTRP (https://ocg.cancer.gov/programs/ctd2/data-portal/) to between 0 to 1? From the link, the range of AUC is 0 to 22, which is inconsistent with AUC from GDSC (0 to 1). Since final evaluation will be more close to AUC from CTRP, I wonder what's the suggested approach to normalize? Is it a regular scaling process (scale 0-22 to 0-1)? Or where can I find the normalized AUC from CTRP? It's my first time to attend the challenge, perhaps I missed something obvious. Thank you, Chuyen
Ok, thank you.
Hi Denise, The gold standard is obtained from CTRP. You can read more about the scoring dataset and process under "Assessment," [here](https://www.synapse.org/#!Synapse:syn21763589/wiki/602313). Best, Robert

What data do you use to score the submissions page is loading…