Dear Organizers, as requested, below are my comments about the idea challenge. This thread is for anyone interested in exploring the concept of Idea in more depth.
1) The traditional form of challenge is for the challengers to solve a problem for the organizers (often a company or a research institution). The combined power of the crowd to dig very deep into a difficult question. And this question if solved, creating value for some of the stakeholders (which gives them incentives to finance the challenge).
In the Idea challenge, it is exactly the opposite: the organizers are working for the challengers who are only doing what they know to do the best : which is to think. Isn't it going to be a bit difficult for the organizers ? that's a huge amount of work.
2) By allowing the challengers to think about their own questions and building their own model. You might have 200 different teams and different models for different questions. How to ensure that each problem is deeply cogitated ? For each question, only 1 team thinks about it. Isn't it the opposite of the whole concept of "collaborative thinking" ?
3) Here is an example of concept I envisioned (close to the idea challenge but different): a ?Reverse DREAM challenge?
Subchallenge 1: We propose a set of machine learning algorithms. The participants cannot change them. Their role is to think about what could be predictive of the response (drug response/patient survival....), and search for data sets on the web. They can even build their own data sets according to their own theory. They have to submit clean and ready to use feature sets. This challenge is therefore entirely about biology and intuition: probably the most important step. And it is designed for those who really care about mechanism.
Subchallenge 2: Once the most important feature sets are selected from subchallenge1 with our own ML algorithms. We can use those selected data to do a pure machine learning challenge. This time, it?s for data scientists. Time for the participants to use their own algorithms !
Of course this is only an intuition which in its current form is very difficult to implement. But the main issue I wish to tackle is: validate new biological insights from very different perspectives in a objective way (every model should be tested by algorithms).
If you have any ideas or concept of "smart ML challenge", here is the place to discuss about them.
Created by Mi YANG MI_YANG Dear Mi and Yuanfang, I am very excited that you are interested in our challenge even modulo some concerns. I agree that this challenge is quite unlike the others, but that's the whole point, as Chloe said. It serves two purposes: 1) to try to bring about the creation of datasets that are not necessarily currently in existence and give an opportunity for pure ML models, as Mi said, a chance to drive this step; 2) bring about ideas for generation of new challenges, perhaps ones that Yuanfang would like to see more of =) I agree with Mi, step 1 is hard on the challenge organizers but we are aware of that and are willing to carry the load if it means bringing about a revolution or at least a new direction to how research in our field is done. From the participants, this challenge will of course require to bring in not just the new methodology but bigger ideas about our field of application (biology or medicine), which I think most of us accumulate having spent a bit of time in the field. It gives a chance to work on those pet projects that might be hard to fund because they are too risky. We believe that this challenge constitutes a kind of a leap in terms of ML types of models used for challenges as well, because there is no constraint to a simple classification task, allowing for example, to give generative models a chance to shine.
To Yuanfang, I don't think there is a lack of interest from the DREAM community. The Idea challenge webinar had 100+ participants and more people have asked for access to the recording. I agree that it's often more convenient to have the data given to us and we just have to worry about solving a concrete task, but such a task comes with its imperfections. The Idea challenge is designed to overcome those imperfections and move the field forward. I believe, the participation in the webinar is a good indication of the DREAM community interest in this endeavour! kind of agree with Mi, and because i care so much about where dream is going and dream has brought me so much, i cannot help commenting:
no objective measurement then it is impossible to be convincing, and that's why no one is in. you and anna had great motivation. the problem is the people here: they believe in numbers... if you want to judge by 'innovation', that is fine, but it has to be judged by a piece of code. everyone knows what 'innovation' really means...
but in general, as an active participant in dream, i think i can represent the view of many other participants.
i do not like the following types of problem:
1. the ones that don't have enough training or test, which left most things to chance. e.g. RV, prostate cancer.
2. the ones that i have practiced too many times and i know i win before i enter..., e.g. drug synergy, TF-binding (i might skip this one this time). but i still have to do some of them to keep my list growing.
3. those lack a defined metric.
i would like to play with the following types of problems in the coming years (kind of as a summary of my previous experience, from most to least important :-):
1. that are completely new to me, so i can quickly broaden my expertise, and become top-tier expert in a new field. e.g. 2014 olfaction challenge, 2016 SMC-HET right now, would be good to have a signal-processing/fourier-transform based project, heart beat, ABR test. some biophysics projects would also be good, simulating microbial moving
2. those demand new engineering techniques, so i can quickly improve my programming, system techniques. e.g. requires adaptation to a new cloud platform, needs to learn a new programming language, needs a live demo.
3. problems that have mathematically optimal solutions, but not yet been found or reinvented in the bioinformatics field. e.g. one we will see the result soon (fingers crossed). these can be entertaining, because you can just sit back and watch the whole thing evolve.
4. problems that have money. e.g. 2014 network, 2015 ALS.
Hi Mi,
From my point of view, the power of the crowd here is to evaluate, discuss and better the ideas proposed by others. I agree this is challenging and our first attempt to do this in such a format, but this is what makes it exciting! The collaborative thinking phase comes after November 15, when the community discusses the models that have been proposed. In my mind the incentive goes as follows: the more discussion there is about a model, the more it will improve, and the more likely it is to win.
What you are suggesting is similar in spirit (but we are neither limiting the set of ML algorithms nor the biological question for the first phase) to the challenge we are proposing. I'm happy to see that others have been thinking along those lines, as I hope this means the time is right to propose such a challenge.
As you have already given some thought to this idea, are there any things you would like to see done differently in what we are proposing?