Hi @PitVis2023Participants. The full dataset is released (https://www.synapse.org/#!Synapse:syn52031744). Please read the "read-me.pdf" document, it contains important information regarding the data. Details regarding the evaluation metrics will be released shortly.
Created by Adrito Das dreets @PitVis2023Participants.
Evaluation scripts are now available in data/metadata/evaluation_*.py.
The evaluation metrics are: (i) (macro-F1 + edit-score) / 2 (ii) macro-F1 (iii) (steps-macro-F1 + steps-edit-score)/4 + instrument-macro-F1/2.
- Macro-F1 is the sum of F1 per class divided by the total number of classes.
- The formal definition of edit-score can be found here: https://en.wikipedia.org/wiki/Levenshtein_distance.
For clarity, I have changed the "-" instrument annotation description to "no_secondary_instrument", and changed the integer mapping from "-1" to "-2". There is no functional change here. Please check the challenge task model clarifications H (https://www.synapse.org/#!Synapse:syn51232283/wiki/621586). "Models will not be judged on the performance of these steps." During testing, any frames labelled as steps 11 or 13 will not be included in the final evaluation metrics. You can choose how you whether you wish to train with these frames or not. I decided not to remove the frames entirely as they may contain important temporal information, you can remove them from your own training if you wish. Hello everyone,
I hope you're all doing well. I'm currently working on the steps task and I've come across a bit of confusion regarding the classes that should be considered for training and validation.
In the dataset description page, it's mentioned that classes 11 and 13 were removed due to insufficient volume. However, I've noticed that the last data annotations provided still include these classes.
Could someone kindly clarify whether we should consider classes 11 and 13 for our training and validation processes? Specifically, I'm wondering if we should:
- Include these classes even though they were removed from the dataset description page?
- Discard frames that contain these classes during training and validation?
- Label classes 11 and 13 differently, considering the discrepancy between the dataset description and the last data annotations?
Any guidance on how to handle this situation would be greatly appreciated. I want to make sure I'm following the correct approach while working on the task. Thank you in advance for your help! @PitVis2023Participants In-case you have not been following this discussion.
A new folder, "annotations", contained the combined step and instrument annotations as one csv (frame-by-frame). There has also been a minor corrections to both step and instrument annotations (they were out by 1 second). The instrument annotations were out by 1 second - this has been corrected. Many thanks to the participant who brought this to our attention! A new folder, "annotations", contained the combined step and instrument annotations as one csv (frame-by-frame). After quality control, we have decided to include the addition of one more step (synthetic_graft_placement, 9). Please see the updated step annotations and use this for the challenge. I received emails saying the dataset could not be downloaded - this should now be fixed. Please add it to your downloads basket before downloading.