It would be nice to re-open the leatherboard for SC3 too. One reason could be related to misguidance in SC1 and SC2 submissions as the models people submitted to those challenges could be related to their SC3 models and if they were mislead in SC1-2 then this could have affected their SC3 submissions as well. Another reason is that not everybody have submitted 4 predictions and third, upon correcting their SC1-2 models competitors could now improve their SC3 models as well. It would be nice to make one extra submission for Time0 and Time24 of SC3. Thanks.
Created by Zafer Aydin zaferaydin Solly, I am so happy with database and the questions you guys designed. No complain though we are still confused with the subschallenge 2.
Cheers,
Samad Overfitting should be less of a concern for SC3 considering the fact that it is a regression problem and the top scoring method has a correlation around 0.5. I think there is still room for improvement in there and giving two more shots to everbody can be considered. On the other hand I understand the point raised by Samad about the fairness. I think this is particularly true for SC1 and SC2. I will accept the final decision made by the organizers. Thanks. Samad, the bug affected many submissions to some degree. Further, some or all of a given teams' submissions may have been affected. It is not clear to draw the line in terms of who should get more submissions and who shouldn't. Any attempt to do so will always leave some people feeling unhappy. Further, all scores are correct now, so each team will have the same number of scored submissions.
I'd also like to remind each of you that this stage is considered a "collaborative analysis". There is no advantage to outscoring other teams, nor to overfitting the leaderboard data. Quite the contrary, there is typically a disadvantage to overfitting the leaderboard, when models are evaluated on independent data.
Solly Solly, giving 2 more shots is OK but I think still giving 2 more shots to all groups is not fair. Some groups had 4 correct scores from leaderboard but a few got random scores! - S Yes the bugs did not affect SC3 directly but it can affect indirectly. If a submission is scored incorrectly in SC1 or SC2 and if this submission uses a particular feature set selection method that was also used in SC3 models then the SC3 submissions could be misguided too. Those SC3 models that use the same feature sets could have been eliminated and not submitted at all. They could have the potential to score highly. As the leaderboard bugs did not affect Subchallenge 3, we cannot offer further submissions at this time.