Given that we don't have any labelling on joints (which would allow us to reduce the problem from limb-level to joint-level analysis), it occurred to me to ask if there wasn't a there there already, i.e. some pretrained out of the box model. I found some work:
* [Convolutional Neural Networks for Automated Fracture Detection and Localization on Wrist Radiographs](https://pubs.rsna.org/doi/abs/10.1148/ryai.2019180001?journalCode=ai)
* [Feature Learning to Automatically Assess Radiographic Knee Osteoarthritis Severity](https://arxiv.org/abs/1908.08840)
* [Automatic Detection of Knee Joints and Quantification of Knee Osteoarthritis Severity using Convolutional Neural Networks](https://core.ac.uk/download/pdf/147610006.pdf)
* [A novel method for automatic localization of joint area on knee plain radiographs](https://arxiv.org/abs/1701.08991)
* [Automatic Knee Osteoarthritis Diagnosis from Plain Radiographs](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5789045/)
* [Automatic localization of anatomical regions in medical ultrasound images of rheumatoid arthritis using deep learning](https://journals.sagepub.com/doi/abs/10.1177/0954411919845747)
The topic of joint localization on it's own is not quite as crowded with results as I expected.
The metric of choice for this kind of work is AUC.
The only GitHub I could find stemming from the above work uses manually annotated location data to get started. That is, as a baseline, we can safely say that there is no magic unsupervised training bullet to get joint bounding polygons for free. Someone has to do the work of labeling each training image before you can train up a network to find the joints and hence reduce the problem to a jointwise problem:
* [Hourglass Networks for Knee Anatomical Landmark Localization](https://github.com/MIPT-Oulu/KNEEL)
Created by Lars Ericson lars.ericson @patbaa If you dont mind me asking , have you posed the 2 sub challenges as an object detection problem? What has been your experience so far(in terms of experiments)?
Feel free to ignore RMSE is the scoring function for all three challenges correct?
I'm not saying single shot would actually work. What I'm really saying is that there is a thin literature on radiograph joint localization. This means that, without adding joint scoring on top, localization on it's own is a publishable topic. So it's a pity, with multiple solvers providing independent localization solutions, not to weigh it properly, on its own, in a separate category in the challenge with its own suitable metrics. Yeah - and then you compare the ones that have found joints, versus the one that examined the whole hand?
If 3 people submit scores, they are different and they used different methods 2 find joints via different ways, one uses the whole hand -> try to make some assessment of why one way or another worked better to get the specified score. @stadlerm, not necessarily. A single-shot detector could look at the overall morphology of the hand, and, based on training images and learned cross-correlations involving other-than-joint-level features inferred in the hidden layers, impute a joint score for each each without actually examining each joint specifically. This would obviously be easy to spoof (for example submit hands with the right features but with missing digits), but might score well on average.
Here is an interesting discussion of spoofing: http://www.evolvingai.org/fooling But the quality of detected joints can be evaluated based on the resulting narrowing/erosion predictions
This wouldn't be as in-depth as a direct scoring as you propose, but still valuable under the current constraints @stadlerm it may be possible to do single-shot joint scoring without explicitly localizing each joint. The subchallenges report joint narrowing and damage scores, not joint localization scores.
The sponsors have not defined a metric to measure quality of joint localization. Without defining a metric and observing it, no information will be obtained to do cross-comparison of solutions on joint localization .
Various measures for localization are available:
* **Area under the receiver operating characteristic curve ** ([AUC)](https://pubs.rsna.org/doi/abs/10.1148/ryai.2019180001?journalCode=ai&)
* **Intersection over union mean average precision** ( [IOU mAP](https://challenge.xviewdataset.org/challenge-evaluation)) It's up to the organizers to see who and what they want to include in the write up. every team will submit a write up, so there is room for a comparison based on how well the detected joints did in the prediction
Additionally, one method might end up better for narrowing and the other for erosion. Though maybe unlikely, I wouldn't rule it out. Meanwhile, challenge 1 might be won without any segmentation - therefore I think there is value in trying to get different detection ideas in the pool @stadlerm, without breaking out joint localization as a separately scored item, sponsors won't be able to compare different localization implementations objectively. That information gets lost. Also I don't know that we solvers will necessarily get to see any detail about other solvers approaches. Only the highest scoring implementations will go into the final challenge article. We are more than on board with a timeline extension - in fact we expect one. It's all but confirmed that the timeline will be extended, which we think is more than fair due to the leaderboard issues and the initial dataset issues.
I think that should give more teams adequate time to explore solutions to find joints. From our limited experiments so far, even unprecise joints work better, than no form of segmentation at all. I would be very curious though to see some successful models that do not use any form of segmentation at all
Generally, I think it's not too late to try and do some sort of detection and I'd prefer it this part is not standardized, and we instead get a range of different solutions. I think this will improve the quality of the output as well, since we will be able to compare different "joint detection" approaches, with respect to how well they perform in the classification
So to summarize, RA2 has got a quorum of solvers, enough to explore the problem adequately. Solvers don't want a change in terms or joint labels, especially those that did the labelling work. You want a Leaderboard, a score and an award, sooner rather than later. Like everyone, I'm waiting for the organizers' response to the problems raised.
I just wanted to clarify - because it was assumed that I already work in this domain - that even though I did some work in machine learning before, until this competition, I did not work on any project that involved image processing. I just want to clarify that, while I can only speak for our team, I can assure you we did not have a solution in hand - at all. In fact we have spent the past weeks solely working on a solution for this issue, which is precisely why we think it'd be unfair to now remove that part of the challenge
I am fine with an extension to the deadline, due to the delays with the dataset, and the leaderboard
Lastly, while we are confident in our solution for the joint localization, it is impossible to tell if it works as intended before we aren't able to test it, and the resulting predictions, on the validation/test set
Generally, I agree with your points though - I just think that adding labels retroactively now, after explicitly saying they were not gonna be provided, is unfair to teams that have invested the time in trying to get a label solution working
```
No joint labels were provided, but two solvers have indicated that their solutions require labels for training and that they have separately both performed labelling and implemented localization as part of their solution.
```
Actually that 3 teams at least. We've also annotated the joints and trained object detection on them. @decentmakeover, challenges of this sort have become popular in the last few years. There are many of them. There is no standard for challenges and no guarantee that a challenge put on by organization A will be at all similar to a challenge put on by organization B. I am fine with challenges being re-worked when the original design turns out to be impractical.
In the case of RA2:
* No joint labels were provided, but two solvers have indicated that their solutions require labels for training and that they have separately both performed labelling and implemented localization as part of their solution.
* No detailed information on scoring was provided, because organizers have a belief, based on a prior challenge, that exposing the scoring algorithm will lead to overtrained solutions that don't achieve the scientific goals. In addition, a small limit on submissions was declared on the Leaderboard, which doesn't actually exist yet. The net of the two decisions is to make the contingent payment that much more of a gamble. The two solvers who have localization algorithms already work directly in this domain, and one has already submitted a solution. I infer that they came to the challenge with a solution in hand, thus making the preparation of a submission a quick, low-risk exercise.
My view is that
* Scoring should be transparent
* Submissions should be unlimited during the challenge period
* Training labels for joint location should be provided by the sponsors
* Joint localization as a task should be broken out with an appropriate AUC score
* We need a Leaderboard that works for a significant amount of challenge time
If sponsors agreed with these views, then they would need to change the challenge rules and extend the timeline. I would prefer this, so I'm disagreeing with you. I don't speak for them. We should hear from them soon because @dongmeisun said they were going to have a management meeting yesterday.
Sorry for being late to this party, extending deadlines and changing the scope of the challenge either by introducing a new task or removing an existing task is not a sign of a well organised challenge.Besides, it's been more than a month and any changes including extending the deadline should not entertained lightly,i hope the organisers work on getting up the leader-board up as soon as possible instead of extending the deadline of the challenge.
When taking part in a challenge the participants usually have a time-line they adhere to, and any changes severely disrupt the workflow of the participant, as it prove advantageous to some participants while serving as a disadvantage to others. IT's for sure we will extend our deadline for the challenge since we delayed leaderboard opening so far. We will let you all know asap. How about this: Let's assume sponsors actually have annotations for joint location. Since we now know there are two independent solutions for localization, maybe localization on its own with some AUC metric should be broken out so we can accurately compare solutions. Also the Challenge timeline should be extended 3-4 months because they don't even have a functioning Leaderboard yet or a single scored entry, and the game is almost over while never having really begun. @arielis While I think that's very generous of you, I think the organizers have made it clear that they intend this problem to be part of the challenge (by saying no bounding boxes will be provided, and stressing that that is why there are detailed joint specific sub challenges)
As a direct response to that, our team has now invested considerable time and effort into building a solution to this.
Especially since the timeframe for the challenge is becoming quite narrow, I don't think it's fair to teams that already have solutions, to retroactively remove this part of the challenge @arielis I support your generous offer. If organizers take you up on that, my opinion as a fellow solver is it would be fair to guarantee you an author credit on the paper they are writing and some chunk of the cash they are intending to award, because joint localization / ground truth annotation is a significant subproblem, which I feel is unavoidable to really solve the main challenge. Most of the work cited above is just about joint localization, so localization in and of itself is worth an article on its own. I've solved this part of the problem.
If the organizers wish to take away this part of this problem as a subchallenge so that other users can use already annotated bounding boxes, I would be happy to share my work. Thanks @stadlerm, that's what I was looking for, the idea of using someone else's trained network as a magic wand. This sort of transfer learning is also used in other contexts where people start with a frozen ResNet and add some trainable layers on top. However even in this paper there's no getting around the need to manually annotate the training set: "the COCO dataset, was utilized and finetuned with 240 **manually annotated** radiographs". You might enjoy this one as well:
https://github.com/razorx89/rsna-boneage-ossification-roi-detection
Unfortunately hands only, and we have yet to test it to see how well it performs on this dataset but a starting point
The issue of needing boxes remains of course