Hi,
Do you have any update on when the leaderboards will be published?
Also, do you need the description of the methods and source code for everyone, regardless of the scores?
Thanks,
Created by Adi Tarca bcbuprb As stated previously, you will only receive your score after Nov 22nd, so you can still make more submissions only the last will be scored.
tx Pablo ,
I cannot get our scores privately in my Email. I have received the below text.
"You prediction has been scored. Your scores will be announced by the DREAM conference on dec 8 after you have submitted a public writeup."
Best Thanks
Pablo You will get your scores privately on Nov 21st, this is to allow you to change your submission (if you want) until that date.
thanks
Pablo Hi, Pablo,
Can we still get our scores privately?
best No, the rank might be high, but will be disqualified or the metric will change in order to avoid prizing overfitted predictions. Just make sure I understand you correctly: if a high score is due to overfitting, a rank would be low? K. If you use all data without partitioning, you are clearly overfitting and will have a higher score BUT you will be disqualified because it does not make sense to do that... so you can choose!
P Pablo,
I second SJ's concern about overfitting. Our team also tired to limit overfitting in the predictive model but did not work out well. It seems to me that overfitting is encouraged in this challenge. Does this make sense?
I am confused by your response. What do you mean "by using all data you are overfitting?" Also, "don't worry about scores that are too high..."?
I appreciate if you clarify the overfitting issue. It is critical for us to build a predictive model in the final submission.
K. Thanks Hi Samad, seems to me that by using all data you are overfitting, so you should worry about doing your predictions as good as possible but following a correct strategy (i.e no overfiitting). Don't worry about scores that are too high, if their approach is incorrect, it will show!
thanks
P Hi Pablo,
I am confused with overfitting issue. We tried to limit overfiting of predictive model but our predictions will show Lower accuracy than those teams who did not pay attention much to overfitting. For example if we do 10 fold cross validation and submit our prediction, accuracy will be much lower than the same model but trained on all data and tested on all data. My question is which prediction should be submitted? Thanks! Samad Dear Adi
Regarding C. we will probably invite the 2 teams with highest rankings across sub-challenges.
thanks
Pablo Hi Adi,
A. We decided to reveal the results of the challenge during the conference, but we could reveal privately the rankings (I will consult). Thanks for the suggestion
B. Unfortunately we have no funding available ;(
C. Good point we have not decided yet, but will keep you informed.
thanks
Pablo Pablo, it would be great if you can clarify a few more things:
A.On nov 21 when submissions close, will you inform the best performers privately of their ranking? That would be helpful to make sure they can attend and eventually justify to their institutions the attendance to conference, and of course to prepare talks.
B. Are you able to fund the attendance of best performers to the conference?
C. The conference schedule has 2 slots for best performers. How do you pick those 2 if there are three distinct teams ranked at top in individual challenges.?
Thanks,
Adi
No more scores, private or public until december 8th when we reveal the best performers at the DREAM conference.
thanks Hi Pablo,
By "only your last submission will be scored.", did you mean only the last submission will be ranked like leaderboard?
Otherwise, can we still get our scores privately?
Best,
Yookyung Today! Hi Pablo,
Leaderboard is not updated. Can you please give us a date that we can expect seeing the results?
Thanks,
Ehsan Hi, you can send in as many as you want
BUT only your last submission will be scored.
P Hi Pablo,
Is it 5 times like round 1 or is it unlimited?
Thanks, Hi Adi
Should be published today or tomorrow.
Remember you can still submit one last time before Nov 21st (actually many times but only the last one will be scored), we will then need also a description of the methods.
thanks
Pablo