The [Leaderboard](https://www.synapse.org/#!Synapse:syn20545111/wiki/597246) doesn't seem to be there yet. What's the schedule on that?
I'm just curious because I'd like to see how quickly people are progressing to solutions and what the scores are looking like for unlabelled joint localization.
I hope you will be breaking out localization and damage scores separately on the Leaderboard.
Created by Lars Ericson lars.ericson @allawayr , if you're not tracking localization, I could conceivably do something like focussing on damage score for a single joint and then report the average damage score for all joints in the training set where that joint has that score. I've seen something like that done in other challenges, i.e., don't actually look at the data, get a score just by returning the average in that situation. Not that there's anything wrong with that, if it scores well. However, a patient who has had all toes amputated but the one you inspect to key into the average score might object if you told him the rest looked fine, all other things being equal, if that one toe happened to be fine. Hi Lars,
Our target is late January to start scoring models.
We need to be totally sure that the test and leaderboard datasets are accurate before scoring, but we have also identified some technical issues that relate to running Docker containers on UAB's compute cluster as well as getting those Docker containers to reliably access the GPUs that we are making available to them. We are working with UAB to resolve these issues.
Once we are sure that the Docker/Singularity scoring process works reliably, I will release a "hello-world" docker container (as discussed in the webinar) that we have vetted as being able to run on UAB's infrastructure (in terms of accessing GPUs as well as being able to mount in the training and testing image directories).
SC1, SC2 and SC3 will all be scored on separate leaderboards, but we will not be scoring localization, just damage at a given location. The output file will be filling in this [template](https://www.synapse.org/#!Synapse:syn21072036); scores are entirely based on the values in that filled-in prediction file.