Dear @CTDsquaredPancancerChemosensitivityDREAMChallengeParticipants,
We received a request last night to extend the leaderboard round of the Chemosensitivity challenge by one week. I am writing to request feedback from participants about this idea.
Looking at the leaderboard, which will close in a few hours (5pm PT), it appears as if most, if not all, teams have not yet exhausted their 10 submissions for this round, and may benefit from some extra time. If we did this, we would shorten the final round by one week (it's currently one month long), to keep the overall challenge on track.
If you have strong feelings about this, can you please comment below? We hope to make a decision within the next couple of hours.
Best,
Robert
Created by Robert Allaway allawayr Dear @CTDsquaredPancancerChemosensitivityDREAMChallengeParticipants,
As a reminder the leaderboard phase will end today at 5PM PT. and we'll be opening up the final round, which will close in three weeks (Final Round Closes: July 27, 2020, 5PM Pacific Time).
Best of luck!
Robert
Go for it! Dear @CTDsquaredPancancerChemosensitivityDREAMChallengeParticipants,
Thank you all for your feedback. In considering the comments in this thread and email correspondence - and recognizing that this challenge does not exist in a vacuum and is therefore subject to the multiple global crises that are currently occurring - the organizers would like to extend the leaderboard phase by one week.
Here is an updated timeline for the remainder of the challenge:
**Leaderboard Round Closes: **--June 29, 2020-- July 6, 2020, 5PM Pacific Time ^ \*new date\* ^
**Final Round Opens: ** --June 29, 2020-- July 6, 2020, 5PM Pacific Time ^ \*new date\* ^
**Final Round Closes: **July 27, 2020, 5PM Pacific Time
Please note that the final round length is reduced by one week - we'll be ending this challenge at the same time as initially planned - but as noted above, since the predictive problem is the same for the final round (just with different molecules) - this should not be a major impediment.
Thanks again all for your patience and feedback on this change!
All the best,
Robert
Hi all,
given the overall response to this thread and given the feedback of my students working on this specific challenge ("time-intensive preprocessing, gene and drug matching by alias names instead of IDs, batch effects..."), I think a maybe even longer extension could help getting in more models that are better than baseline. This might also be important for any subsequent paper, I think.
Let me convey a general spirit from the current Lindau Online Science Days 2020: As scientists, we love to challenge problems, not timelines. ;) Also for future similar challenges, I think that less tight timelines could see more and better models.
You also have to recall that students usually cannot work 24/7 on one challenge alone, but have many parallel obligations... (And PIs have usually even more parallel projects ongoing and even less time ;) Of course, this doesn't mean that it should be open end. But if all challenges kept their start date and were just designed longer, the challenge throughput would still remain the same...
Best,
Michael.
Hi Frank,
Yes, there are a backlog of submissions from this morning that have not yet been scored - we're looking into this issue on our end. Hope to have it resolved soon!
Best,
Robert
Hello,
Our team would be okay and appreciative of the extension/modified time line.
Related, we have submitted a couple leaderboard submissions this morning, but it appears our models have not been scored (submitted over an hour ago). Is anyone else having this issue?
Thanks to the everyone involved!
Best,
Frank I would be happy with an extension.
Best regards,
Evelyn Hi, yes I think an extension is a good idea too!
Given the final phase is only one submission using one's best model from the leaderboard phase, 3 weeks for the final phase is plenty!
Best,
Denise Our team would appreciate the extension. Hi All
I would be excited to have an extension.
Regards
Suyash Hi,
From my understanding, the methodology someone developed in the leaderboard round will be used for the final round. So, leaderboard round is the most important phase for the development and improvement of the methodology. In my opinion time extension would be a great idea.
Best Regards
Nahim Adnan Hello Robert,
we would very much welcome an extension.
Best Regards,
Benjamin Lotter (Team GrauLab) Thank you both for the quick feedback!
@andrea.degasperi; thank you for the heads up about your submissions, it looks like there are a few submissions from this morning stuck in the queue - we will fix this ASAP. Thanks for the proposal, we fully support the extension of the leaderboard phase considering how challenging the question is.
BR, Wenyu I would be happy with an extension, as I have only started to work on this yesterday. Also, I have submitted a couple of attempts to the leaderboard more than an hour ago, but the submissions haven't been evaluated yet, so I am not receiving feedback on how I am doing yet.
Best Wishes,
Andrea