day 26 opens the final submission round and are so few predictions on leaderboard. Only one team has predictions in all Tfs. It was expected?

Created by RPS PS maximus
Hi Stackd, we will soon open the final submission queue, and make a corresponding announcement. I apologise for the delay. Cheers, Robert
Is it possible to submit to the final cell line benchmark at the moment? I only see the Leaderboard cell types.
I am working as a one man team, and I can say I could use more time. The data set involved is pretty big. Even extracting DNA shape data from the human genome in absence of any other data takes up at least 100 GB.
I think we've all had enough of distracting tangential discussions. I can safely say that 99.9% of us have no idea where all this coming from. A discussion board full of animosity and venom is not a welcoming place for scientific discussion. In the interest of avoiding alienating participants from the discussion boards, I am going to take the liberty of being a moderator of these discussion boards and start deleting posts that are not directly related to the TF binding challenge. Thanks, Anshul.
Hi Rahul, That is indeed a concern. We are going to dis cuss options this Fri and get back to everyone. Thanks Anshul.
Dear Yuanfang, we are indeed concerned about low participation. To summarise the above, and if I didn't miss anything (everyone, please add) we have as possible reasons - difficult data and challenge - not ideal target for standard machine learning - short time - early good submission on the leaderboard might discourage people - dream principles Cheers, Robert
robert: i absolutely agree with you that whether a specific person participates or not doesn't say anything on the validity of the challenge results at any point. --- but less than 10 or even 5 final submissions is sufficient to say the challenge is not valid. please note that there will be a significant drop from leaderboard phase to final test. especially the leaderboard is so short that the team and who they are obvious. for people doing this for a living no one would risk to bottom the leaderboard, right? if you have a longer leaderboard, then no one cares about the second page, right? ---- in the current situation, you should thank sky and earth if you have 5 submissions. ---- don't you think it is really weird that no past participants, and NO past best-performers participating in this challenge? if you look at the the neighboring ones, it is much more vibrate and healthy... i have been in about a dozen challenges, and you know i am quite active on all past ones and on all the neighboring ones that are running right now. this is the only one has this problem, and this is the one with the lowest entry. don't you think there are reasons beyond data, timeline, etc? --- if you say this challenge is too difficult, i am sure challenges running on a cloud is much more difficult, then how do you explain ALS, DM, and even the SMC-HET, which dataset had bugs until the last day of the challenge and which was without any incentives (but you have to pay out of pocket to google cloud once in a while) had more submissions? i think this is really something the organizers want to think about. -- (update: just to clarify by no means i am complaining the SMC-HET challenge organizers here. our team truly enjoyed the problem and working with them).
The leaderboard timeline extends all the way to the final submission timeline - so it can't be extended without extending the final submission timeline too, which would then impact the benchmarking deadline, and it would all get uncomfortably close to the conference date, right? With the current deadline there are over 5 weeks to go for final submission and, speaking for my team, we feel we can realistically make a submission in that timeframe. More time lets you optimise things better, explore other methods, etc, but only up to a point - we are all busy with other stuff too!
Dear Yuanfang, thanks for suggesting folks to notify about the challenge. However, whether a specific person participates or not doesn't say anything on the validity of the challenge results at this point. Cheers, Robert
Dear Maximus, thanks a lot for your comments, we are taking this into account. As Anshul said, we are pooling opinions to see how we could give you more time. You mention the leaderboards: do you have suggestions there? Cheers, Robert
I am enjoying this DREAM initiative. I have been working hard writing, running and improving codes. And yes, i think is enlightful to submit all the TFs, to see the differences and limits that are there between the TFs and what the algorithm can do. The task of this challenge is complex and very computer intensive. Maybe the competition would have some benefit in increase the Leaderboard Round, because is when we can compare how good is the strategy/algorithm chosen and improve them. This process take a lot of time and it is possible that new good ideas do not be tested because of the shortness of time. It is just my opinion. thank you.
https://www.synapse.org/#!Synapse:syn5759485/wiki/394859 russ and bence/gabor are the top performers of AZ, olfaction and prostate cancer challenges, maybe a bunch more that i am not aware of. they are usually separate teams
Hi Yuanfang, I am not sure which Russ and Bence you are referring to. Can you point to their lab webpages or contact info. Thanks, Anshul.
to archul-- but i do not have problems with the organizers of other challenges... --- but i take your words to focus on the current problem. there are about 3-4 teams who perform in the top 3 in every challenge no matter of the topic. i was hoping to at least see russ or bence's team or several teams partitioned from the big finn group here; otherwise the challenge result isn't even convincing. if i were you i would approach them, because they are incredibly smart and they will provide very unconventional top solutions in a short period of time.
Please avoid tangential commentary and let this discussion board stay focused on this particular challenge. If anyone has had problems with DREAM in general or other challenges (rather than this specific challenge), please take it up offline with the DREAM management team. This is really not the right forum to air all possible grievances one may have unrelated to this particular challenge. I'd just like to clarify that nowhere in the DREAM rules does it state that a participant is not allowed to publish their methods for 2-3 years or that they will be relegated to some sundry position on some DREAM paper. As one of the primary organizers of this challenge, I can guarantee you that these statements have no relation to this challenge. The small number of submissions is indeed disappointing. We are reaching out to the community and participants to figure out what may be resulting in difficulties with participation. We might modify challenge deadlines etc. depending on participation and feedback. Please continue working on the challenge and making submissions. From our side, we are working hard to improve the experience, release and activate the unified final round scoring code and get everything as streamlined as possible. We appreciate your patience with the few glitches we have had. For registered participants and others who have not yet submitted, please let us know what is impeding your progress and we can try to find ways to help. Thanks, Anshul.
Well, maybe!
it is great to have new people. but it is equally important to keep existing ones. good luck for your and ivan's submissions! is it possible that less participation might make it easier to be the best-performer? that actually would become a good thing they did...
Yuanfang, yes, first time submitting. But I have followed for some years and attended a couple of previous conferences. Isn't it good if DREAM attacts new people? :) I agree that the question of why so few submissions is a question, but there are 25 registered teams and many more un-teamed participants, and there remains time. But maybe you are right in your diagnosis!
rahul, and ivan, i believe both of you are new to dream. and it is obviously still the honey moon period. i am sure a big fraction of the dream community understands where i come from, because i have received quite some comforting emails this year, and because my write-ups in dream are closely followed by the dream community (when i updated my write ups for the AZ challenge, in less than one hour i received two congratulation emails). now i am just trying to help Costello understand why no participants; i believe (not i hope) if they don't change it, this challenge will have a history low in final submissions.
Ok.. having uploaded two leaderboard submissions (plus one problematic one) let me chime in, because I find some of the negativity above unjustified. I agree with Ivan that ladderboard participants are unlikely to bother uploading for *every* cell type (though we intend to; we were convinced though that 10 submissions per tf is very generous and we are never going to max that out). Also with Yuanfang that this is a difficult challenge, which is why we made our first ladderboard attempt only yesterday and still have many improvements to make. I partly see Yuanfang's point about the publishing rules. I don't see the "two years" thing written anywhere -- maybe I missed that bit. Or the bit about not publishing the method elsewhere, but our current method and other methods being contemplated have been published long ago and are in textbooks, only the application is (possibly) new! I do regard membership of the the "Dream community" on that cell paper as a rather minor incentive, though obviously I wouldn't say no. Basically, like Ivan, I see it as a fun exercise and a chance to explore machine learning techniques (not my speciality). Finally regarding this «"Use of sarcasm, threats, blackmailing, foul or offensive language, belittling/taunting others, or making them feel unwelcome may be cause of dismissal from DREAM." i felt quite unwelcome when i read the dream principles, will the person who wrote it be dismissed from DREAM?» Not all forms of "feeling unwelcome" are equal. The question is what caused them to feel unwelcome. I fully support the policy quoted.
Dear Everybody, Regarding the submissions count (2maximus): I personally consider ladder as a nice playground to check if our models technically work and if we do everything in technically correct way (it might be a bit sad to submit a final model and realize it has 0 scores due to missed tab). So (if it is not a mandatory step to enter the final evaluation) it seems that we are not forced to submit a ladder prediction for every TF. _a side note_ If only there'd be an intermediate deadline for ladder submissions when a "ladder-round-winner" would be announced... Regarding the other 'big stuff' (2Yuanfang): It's an interesting challenge, a fun game and a nice motivation to have a deeper look on various data sets provided by ENCODE. I am not sure if it makes sense in terms of machine learning, but for people in regulatory genomics it's 'just interesting' beside ratings-winning-and other cool stuff.
Hi Maximus, We are a bit surprised by such few submissions to the leaderboard. We will be sending a note to the community inquiring about this soon if the submissions do not pick up a bit. Any thoughts you have on the challenge would be appreciated. Feel free to contact the organizers via personal email if you would like to communicate privately. Kind Regards, Jim

The final submission round is ahead and just page is loading…