Dear RA2 DREAM Participants,
We would like to update everyone on some of the changes that we have made to the challenge along with some updated timelines. Please read through the email for all important details.
**Scoring:** We have spent a great deal of time benchmarking different scoring approaches for the challenge and arrived at the average weighted absolute error (SC1), and average weighted RMSE (SC2/3). The weights have been assigned to different ranges of SvH, overall joint erosion, or overall joint narrowing scores in an attempt to balance the importance of scoring the low and high scores with equal importance. Immediately after the challenge closes, we will release the docker scoring container that contains the scoring code, including weights. The container is timestamped on Synapse and has a specific SHA digest that references that timestamped container. Participants will be able to inspect and run this code to reproduce the exact same score as was returned to participants in the Challenge. Please find updated text describing the scoring [here](https://www.synapse.org/#!Synapse:syn20545111/wiki/597242).
**Leaderboard:** The leaderboard (9614346) is now open. Teams can make submissions now and they will be scored and posted on the leaderboard. The leaderboard will be open through the end of the challenge. We have also provided teams with a fast lane submission queue (9614401). The purpose of the fast lane to perform a test run of your model on a small subset of the data to validate that it will run on the full dataset. If you submit to this queue, you will receive an email telling you the status (invalid or valid) of your submission. If your submission is valid, feel free to submit to the leaderboard queue. Otherwise, please run/debug your container locally to ensure it produces a correctly-formatted prediction file as described on the [Docker submission](https://www.synapse.org/#!Synapse:syn20545111/wiki/597249) wiki page. Note that a submission to the fast lane is not a submission to the leaderboard.
**Number of Submissions:** We recognize this challenge is complex and there have been several requests to update our submission limit of 3 total submissions. We have decided that we will allow 3 submissions from now until March 5, 2020, then 3 submissions per week for the following 7 weeks. Thus, teams will be allowed up to 24 submissions if you take advantage of the submission timeline. Teams will be limited to the number of submissions for the current period, thus, for example, if a team starts submitting in the 6th week, they will be allowed 3 submissions for that period and then 3 submissions for the 7th and 8th week, totaling 9 submissions. We hope this incentivizes teams to actively work on the challenge over the course of the leaderboard phase to take advantage of the maximum allowed number of submissions.
**Updated Timeline:** To match the updated submissions plan, we have updated the timeline for the challenge. The leaderboard round will extend until April 23, 2020. Details of the updated timeline can be found on the [RA2 DREAM Challenge landing page](https://www.synapse.org/#!Synapse:syn20545111/wiki/).
**Additional information on images:** The UAB team has put together a short walkthrough video demonstrating the process to apply the SvH scoring method. Details can be found [here](https://www.synapse.org/#!Synapse:syn20545111/discussion/threadId=6498&replyId=20959).
Finally, we want to thank Ralf Anzarouth from Computer Vision News for a [nice write-up of the RA2 DREAM Challenge](https://www.rsipvision.com/ComputerVisionNews-2020February/24/).
Best of luck with solving the RA2 DREAM Challenge. Please do post any questions here.
Kind Regards,
The RA2 DREAM Challenge Organizers.