I am very curious to know how are the other participants approaching the problem? For sub-task 1 and 2, are they treating it as a classification problem or a regression problem?
Since it can be tackled using both approaches
Thoughts?
Created by decentmakeover Sorry no, we are using tensorflow :( @stadlerm I am still in the docker submission phase, are you by any chance using a pytorch docker image? Now that we have the first models in and the challenge has really begun I wanted to come back to this - though i can't reveal too much, i wanted to at least let you know that we've decided to train our models with classification loss after all, as opposed to a regression
This just seemed to work better for our current setup - this is largely due to how we decided to deal with the issues of class imbalance, though I'm sure there are also ways of doing similar things in a regression setting @stadlerm,
That's very well explained.
Although I do not agree with you that from a clinician standpoint there is no inbetween.
I think that there is a continuum between the different degrees of narrowing and erosion, and that it is only in order to keep the rules simple to use by a clinician, and ensure less subjectivity, that the scoring system uses integer scoring.
For instance, in narrowing the difference is between less than 50% (2) and more than 50% narrowing (3). The 50% is arbitrary, but is easier to spot visually.
An objective and accurate measurement, by an algorithm can get to a higher resolution, and get a continuum between 2 and 3 according to the exact percentage of narrowing,.
To my opinion the measurement by an algorithm would be even more accurate on some points, than a human evaluation, and would be better correlated to clinical outcomes.
thanks for the inputs
I think in practice you'd probably want to do a multi-label classification. I believe that is the more accurate score, since the values are only semi continous - ultimately the clinician assigns some values and then creates a score that leads to one of a few categories
e.g.: if your joint space narrowing is either 2 or 3, then there is no inbetween in terms of the way the score works
That being said, I think for the competition you should definitely go with a regression, as already discussed by arielis.
Reasons for this are simple:
- less outputs per image/joint, since you would quite a few outputs to one-hot encode the different possible classes (assuming you are doing some sort of softmax neural network output)
- connected to that first issue is also that we have relatively few samples, and many samples have relatively low scores. This means you would have a quite unbalanced dataset from a classification point of view
- lastly, the scoring metric is rooted in RMSE, which definitely favours regression. If you misclassify, you will always incur an error of at least 1. Imagine you were undecided between the score of 2 and 3, if you classify 3 instead of 2, you get an error of 1. Regression might instead output maybe 2.5, which incurs a lower error in terms of RMSE. If you are convinced you can get many correct classifications (which I doubt, see points 1 and 2), then this could maybe offset these errors (as in those cases you would incur a 0 error, whereas with regression you will probably always get some small error)
So overall, regression seems the way to go No problem,
Good luck! Thank you for your inputs. That would not be enough, we need to rate the degree of narrowing or erosion, not only tell if it is present.
This is a graded outcome, so it makes more sense for me to use regression, What is your opinion of using classification based approach? Hi,
I used a regression approach.