Dear @RA2DREAMChallengeParticipants,
The final round is now open.
Please be sure to review the announcement that was emailed out last week and is attached below for more information!
Best,
Robert on behalf of the RA2 DREAM Challenge Organizers
>Dear RA2 DREAM Challenge Participants,
>
>We would like to again thank you for your efforts with the RA2 DREAM Challenge, along with your patience in dealing with data issues in a very unusual time for all of us. We are coming to the close of the leaderboard round and we wanted to update everyone on how the final scoring round will proceed.
>
>After the leaderboard round closes on May 14 at 5pm PT, we will open the final scoring round with submissions starting to be scored on May 18, 2020 (the delay is due to some performance testing). Submission of your final model will be made in the exact same way as in the leaderboard round, but you will select RA2 Challenge Submission Final Round as the submission queue. Your final model will be used to score two sets of data: the leaderboard test set and the final scoring test set. We will first apply your final model to score the leaderboard round test set. There will be no leaderboard where this score is posted, but it will be returned via email to allow you to ensure that the final model is producing results that are within the expected scoring range. This will avoid the situation where there is a major problem with your final model and you have no way of knowing that. We will also apply your final model to the final scoring test set, but this score will NOT be returned to the submitting team.
>
>Each team has until June 11 at 5pm PT to make their final submission. Up until that date, we will allow up to 3 final model submissions and we will take the last submission as your final one. If your first model scores well on the leaderboard set, then feel free to keep that model as your final submission. Again, you will not receive feedback on the final scoring test set and you will be allowed up to 3 submissions, with the last submission being your final model.
>
>After the final scoring round closes on June 11, 2020, we will enter the validation phase where the organizing team will verify that submitted models are complete, they reproduce the final output and that there are no substantive issues with the code. We will contact teams that made final submissions on June 11, 2020 to ensure that they have submitted their required [writeup](https://www.synapse.org/#!Synapse:syn20545111/wiki/603038) describing their methodology, which will be due no later than June 25, 2020 (please use the [method writeup template](https://www.synapse.org/#!Synapse:syn20545111/wiki/603038) as a guide). We rely on your cooperation to get through the validation phase as quickly as possible, so please do make sure to make your method writeups complete and timely. Please note that method write-ups are needed to be eligible for [Challenge Incentives](https://www.synapse.org/#!Synapse:syn20545111/wiki/597241). As always, please post any questions about the final scoring round on the Discussion Forum.
>
>We will post the final scores on the final leaderboard on or before July 31, 2020. Best of luck to everyone on their final submission and we look forward to seeing some great submissions that will facilitate automated assessment of radiographic damage in rheumatoid arthritis!
>
>Kind Regards,
>
>RA2 DREAM Challenge Organizers
Created by Robert Allaway allawayr In other words, predictions can be continuous, even if the SvH scores are an integer-based system. It's interpreted as a 3.9. Your predictions are interpreted and scored as a double, not as an integer, so you can be as precise as you want (up to the double precision limit in R, so around 16 decimal places if i remember correctly).
One more question (a bit late in the game, but still): suppose I have a regression model and output 3.9 for a joint. Suppose the ground truth is 4. Is my output read as a 3 or a 4? Hi @lars.ericson , That is one way to use the final submissions (as long as you did not overtrain to the leaderboard set). In short, yes, you have 3 shots, the intention is to give you some feedback on your final submission without giving you the final score so that you cannot use your 3 submissions to optimize to the final scoring set. After each submission, you will get an email with your score on the leaderboard set. Nothing will be posted on leaderboards. If you are happy with the score, then you are done. If the score is wildly off from what you expect, go ahead and make another 1 or 2 more submissions (for a total of 3). We will take your last submission as your final submission. Good luck!!
Cheers,
jim Hi @james.costello , to recap, we have 3 shots, you will privately tell us the Leaderboard score on each shot, and you use the last shot. So effectively we have two shots and we use the third shot to reproduce the best of the first two? Dear @RA2DREAMChallengeParticipants ,
We would also like to note that we understand these are unusual times. We will stick to the timeline as posted above, but we also want give teams an opportunity to make a final submission, so if your team has extenuating circumstances that might prevent you from submitting by June 11, 2020, please contact @james.costello and/or @allawayr directly and we can discuss your situation.
Kind Regards,
On behalf of the Challenge Organizers.