would be good to know to plan ahead. thank you.   also, it was said somewhere else there might be an extension of deadline. was that discussed/determined?   also, 3 submissions or 6 submissions?

Created by Yuanfang Guan ???? yuanfang.guan
> BTW, how do we check how many hours we have left? Please see the updated submission dashboard here: https://www.synapse.org/#!Synapse:syn4224222/wiki/409764
@yuanfang.guan === When someone makes a big statement, I just have to get to the bottom of it. :D I'm glad we are on the same page. Good luck with your submissions! Li
Hi, Li,   i am not sure what you mean by 'some people do not play fair'. I just said we might over-fit. Then you accused me for 'tricking the system'. Usually that type of statement require some evidence.   You ask for some examples, so i googled some. If you think they are inappropriate, then off they go.   I am just trying to contribute my very limited intelligence to this important question in cancer research. I believe better solution will soon be found by other experts, like yourself, in the field. I am eager to learn that solution as well.   There is no hidden tricks. This would defile one of the main purposes of DREAM, which is to evaluate methods in an unbiased and blind way. The aim of collaborative science can only be achieved if our dialogue and attitude is not confrontational. I believe that mutual respect, trust and the appreciation for everybody?s contributions are necessary to succeed as a community.   So now we should focus on why the server has been dead for a whole day now.   Yuanfang Guan
@yuanfang.guan === Thank you for recommending the Kaggle competition! But did you read through the article? The problem with that competition is that the test set is very small: only 198 cases. What's more, Kaggle allows participants to submit 3 submissions per day and provide immediate feedbacks. That's a lot of submissions in total during the entire competition. What's why somebody can guess the labels and get very nice scores. (The guy who got the perfect score was just lucky, he did it with only 14 submissions.) But the DM challenge is different. The test set is very large: there are hundreds of thousands of mammograms. What's more, the only feedback is one AUC score and one pAUC score after each round. No immediate feedback is provided for each submission. So you wouldn't be able to alter the predictions and observe the changes in your scores. With such a setup, I don't see how someone can use such a strategy to game the system. I would thank the organizers for coming up with a fair competition for everyone. Now, if you know some tricks to play the system, you should really point it out so that it is fair for all of us. Our purpose is to improve the technology for cancer detections. If some people do not play fair, it loses the purpose. As for the respiratory virus challenge, I don't see how you come up with the conclusion that no solution did better than a random number generator.
@thefaculty not a problem, here is an example: someone just perfected the score of another 1 million dollar competition that just began last week in 16 submissions (without any training). https://www.kaggle.com/c/data-science-bowl-2017/forums/t/27800/my-perfect-score   you can also take a look at the respiratory virus challenge. https://www.synapse.org/#!Synapse:syn5647810/wiki/399106, mark my words, no one in the >0.8-0.9 range on this leaderboard would get a significantly better than random score on blind dataset. We will soon see that result on Feb 1st.   I think 9 submissions on any problem would quickly lead to overfitting.
@yuanfang.guan === That's a very interesting statement. Do you have any research to support your claim that 9 submissions can cause 20-30% overfit?
based on my experience, 9 submissions are already sufficient to over-fit by 20%-30%. it is common to see the top-ranking one on leaderboard to drop by this much on final test.   so it is nice that in this challenge every round has money, so everyone can all take a small piece of the cake. otherwise, it is really annoying that in the end one winner takes all.   but of course, this is my first time hands-on deep learning, maybe my estimation is wrong and deep learning doesn't overfit that much
I do not beleive that 6 submissions per round could help anybody to overfit, especially without final test output file.
@thefaculty @ynikulin OK, I see your point. But more submissions will be more overfitting. Especially the leaderboard has cash incentive ( otherwise i am too lazy to even use leaderboard ) then the incentive will be given to an overfitting model. But I am fine with more submissions, as long as it is the same for every one, so we can all overfit the leaderboard, that is fair too.   I actually don't have many models to submit. I only had one model in round 1. and in round 2 I haven't thought about a good idea to try yet. But I am fine with multiple submissions, so maybe I can just change parameters.
Well, I think it will be more comfortable if I had more submissions. I had a couple of problems in the previous round (tensorflow and Docker issues). BTW, I do not see any link between 3 example networks and 3 submissions per round.
@yuanfang.guan === I failed to follow your logic. Why does the number of submissions have anything to do with the example networks? I may come up with many different strategies that perform similarly on my validation set but I'm not sure which one gives the best score on the test set. Then I'd like to submit all of them and pick the best.
@ynikulin   You can certainly express your wishes, but whether it will be changed will have to go through their weekly meeting which might not happen weekly. so no matter what there will be a significant time-delay.   but seriously do you really have more than 3 to submit? i thought the organizers only provided 3 example networks. i don't even know how to use up the 3 submissions...
If we can express our wishes, personally, I would like to have more than 3 submissions. 5 (like a submission per week) or 6 would be better. In any way it will not permit us to overfit the testing set, it's just will be more comfortable to work and not to wait the last week to submit something. The final of the first round was just overloaded by people using all the 3 submissions in the end.
Hi, For now it's 336h hours and 3 submissions / sub-challenge.
right now I just date before start and end, so I can manually add the hours together.
I'd love to see more training hours to be provided. BTW, how do we check how many hours we have left? Thx!

2nd phase 336 hours or 712 hours training hours? page is loading…