Hello! Perhaps I missed this somewhere or it is taken into account in some other way, but what are the binary labels for non-responders (iRecist PD) vs responders (everything else besides NE) in the test data for sub challenge 3? i.e. Are non-responders labeled 0 and responders labeled 1?
Created by Adam Klie adamklie Okay, good. The examples above got me a bit confused. Thank you for the answer.
Br,
Anni Dear @nabuliini ,
Thanks for your question. No, you should **not** submit binary predictions. Please submit continuous values.
Kind regards,
Mike Dear @Michael.Mason,
just to make sure, I am following. If my model gives continuous prediction values for SC3, I should cast the predictions to 0 and 1 for the prediction.cvs returned in the Docker?
Best regards,
Anni Halkola Dear @adamklie , @oski and @WenyuWang,
You have all brought good points. I have reviewed our metric code and Wenyu Wang is correct and we are flipping the predictions for SC3. So if you are correctly submitting a 1 for patients that will have BOR == PD and 0 for others, it will result in an auc of 0 (-1 once we rescale it). I will bring this up for discussion with our committee today. Because it does not affect the primary metric, I will suggest that we simply take the abs( AUC - 0.5) for the secondary metric since we are more than halfway through the validation phase. I will post here on the resolution .
I have checked and the SC1 and SC2 code is good. I apologize for the confusion this has caused any of you and will do my best to ensure that this does not affect any participant rankings for SC3.
Kind regards,
Mike Dear @Michael.Mason ,
Could you help confirmed the logic between the code and your explanation?
If we participants are expected to provide a prediction of "BOF = progressive disease", then why in the code the first argument to AUC should be -(prediction of NIVO response)?
Say if I submitted a perfect prediction for the Nivo arm, where for all real PD patients my predicted values are 1 (and for all the Non PD my prediction values is -1). According to your explanation, since my prediction perfectly prioritized PD patients over non-PD patients, I should get a nivo_auc of 1, right?
But according to your code, it seems I will get a nivo_auc = -1 :
nivo_auc <- AUC(-preds_df$prediction[nivo_arm], as.numeric(gs_df$BOR == "PD")[nivo_arm])
Thanks in advance! Dear @oski ,
For SC3 participants need to provide a prediction of "BOF = progressive disease" which can be thought of at `- prediction of Nivo response`. Challenge organizers *will not* change the sign of of what is submitted.
We hope this helps,
Mike Dear @Michael.Mason ,
Following up on this clarification, does it mean that were are expected to provide ```- prediction of NIVO response``` for sub-challenge 3? Or do we just provide ```prediction of NIVO response``` and you will apply the sign change?
Thanks in advance,
Óscar
Sorry for the late reply @adamklie,
You are correct. The opposite is true for the SC1 and SC2 though.
Regards,
Mike So just to clarify, if I have. a test example that's a non-responder (aka "PD"), I would want to output a 1 to the `predictions.csv` file for Sub-challenge 3. And if that individual was a responder, I would want to output a 0?
Dear @adamklie
In reviewing our code we noticed that the first argument to AUC should be `-(prediction of NIVO response)`. This is due to the fact that we are asking participants to predict response to Nivo and that would mean better disease outcome but `BOR == "PD"` is actually worse disease outcome. The sign actually has no impact on the primary metric but does affect the tie-breaking metric.
```r
diff_in_auc_wrapper_sc3 <- function(preds_df, gs_df)
{
gs_df <- gs_df[gs_df$BOR != "NE",]
preds_df <- preds_df[match(gs_df$patientID, preds_df$patientID),]
nivo_arm <- gs_df$ARM == "NIVOLUMAB 3 mg/kg"
chemo_arm <- gs_df$ARM == "INVESTIGATOR CHOICE"
nivo_auc <- AUC(-preds_df$prediction[nivo_arm], as.numeric(gs_df$BOR == "PD")[nivo_arm])
chemo_auc <- AUC(-preds_df$prediction[chemo_arm], as.numeric(gs_df$BOR == "PD")[chemo_arm])
score <- diff_in_sqrs(nivo_auc, chemo_auc)
return(score)
}
```
Kind Regards,
Mike Dear @adamklie,
Thanks for participating in this DREAM Challenge. Participants are trying to predict whether BOR=="PD" for each patient (omitting patients that are NE). If you are thinking of this in terms of logistic regression then BOR="PD" would be coded as 1 (or TRUE) and all other BOR's would be coded as 0 (or FALSE). Below is the R wrapper code that computes AUC in each arm before passing to the difference in squares metric. There you can see that `as.numeric(gs_df$BOR == "PD")[nivo_arm]` codifies BOR= "PD" as 1.
```r
diff_in_auc_wrapper_sc3 <- function(preds_df, gs_df)
{
gs_df <- gs_df[gs_df$BOR != "NE",]
preds_df <- preds_df[match(gs_df$patientID, preds_df$patientID),]
nivo_arm <- gs_df$ARM == "NIVOLUMAB 3 mg/kg"
chemo_arm <- gs_df$ARM == "INVESTIGATOR CHOICE"
nivo_auc <- AUC(preds_df$prediction[nivo_arm], as.numeric(gs_df$BOR == "PD")[nivo_arm])
chemo_auc <- AUC(preds_df$prediction[chemo_arm], as.numeric(gs_df$BOR == "PD")[chemo_arm])
score <- diff_in_sqrs(nivo_auc, chemo_auc)
return(score)
}
```
Hope this helps,
Mike