1. Hello. Why is the leaderboard empty, i.e. without any scores? (Torbjorn Nordling)
- There was an issue with view permissions on the table, which has now been corrected.
2. Is there a maximum for the number of predictors in the submission? (Qian Li)
- There is no limit on the number of predictors.
3. For example, is it okay if we submit a list of ensemble predictors? (Quian LI)
- Yes.
4. Should the writeups be public or is it enough to be shared with organizers (Mehrad Mahmoudian)
- For the time being, it is ok to only share the write-ups with organizers. At the end of the challenge, after models have been evaluated on independent test data, we will ask that they be made fully public.
5. Using data up to time 0 and 24 to predict what time? (Samad Jahandideh)
- The outcomes are not time-dependent. For the `Up to time 0` and `Up to time 24` models, the predictors should be restricted to the described time range.
6. Test data has been released for phases 1 to 4. Are we supposed to make predictions for all 4 phases or just phases 1 and 3, which correspond to up to hour 0 and 24 respectively? For instance which time points are we going to use for phase 4 test data? (Zafer Aydin)
- `Up to time 0` corresponds to the data in the ?Phase 1? folder and `Up to time 24` corresponds to data in the ?Phase 1?, ?Phase 2? and ?Phase 3? folders.
7. The type of virus type in the test set is smaller than the virus type in the training data - was it intentional? Also in the test set there were some patients who were given early treatment - is it possible to know what effect will that have on the patients - is the effect significant? Thanks. (Mohammad Rahman)
- The test data was selected among data sets which were previously unreleased, so not all viruses were represented. It is possible that early treatment affects the outcome. You may include this in your model if you wish.
8. Also, there is 4 different datasets, we have to put all predictions in a single file? and how to specify predictions at 0 and 24 when we submit to the leaderboard? (Samad Jahandideh)
- Submission templates have been provided for your convenience. To distinguish between the two submission sets, please name your submissions in a way that distinguishes them, and indicate the Synapse IDs in your writeup.
9. I thought the monte carlo permutation is going to be used to assess that the models don?t overfit the data, are the models from each group going to be compared on the permutated data? (Slim Fourati)
- There will be some analysis of each team?s score relative to it?s ?null? distribution to assess the amount of signal in each model. Ultimately, each submission will be evaluated on independent test data.
10. I worry that having everyone permute a certain way will give some models an unfair advantage over others (Erika Bongen)
- We have chosen the permutation method to be fair among all models and to address the question of what the estimated model performance would be in an independent test data set.
11. Since the permutation is optional, how much weight will it count towards the final contribution of each team? (Di He)
- Teams will ultimately be scored on an independent test data set. However, permutation may be a consideration in travel awards.
12. My question is about time. We will use data up to time 0 and 24. I am wondering our predictor model would be a longitudinal model or not? We can use time as a input model to make a time series prediction. But it looks the questions are not time series because we don't have time in test sets?! (Samad Jahandideh)
- The outcomes variables are not time dependent, however you may wish to use a model that leverages the longitudinal nature of the predictor variables. The predictors in the test data are also available at various timepoints.