Dear Organizers,
I am not sure, but it could be that there is still an error in the value coverage scoring. I will present here a small test, where I think the results are not correct. If you think they are correct, then I might have glossed over something in the scoring algorithm description.
This is related to the APOLLO-2, column #38.
Please create the following three files to reproduce the error.
**Annotated-APOLLO-2-leaderboard.json - extract from the goldstandard:**
```
{
"columns": [
{
"columnNumber": 1,
"headerValue": "causeofdeath_apollo2",
"results": [
{
"resultNumber": 1,
"result": {
"dataElement": {
"id": 6002302,
"name": "Cause of Death Reason"
},
"dataElementConcept": {
"id": 2012739,
"name": "Death Primary Cause",
"conceptCodes": [
"ncit:C28554",
"ncit:C25251",
"ncit:C25638"
]
},
"valueDomain": [
{
"observedValue": "Disease progression",
"permissibleValue": {
"value": "Progressive Disease",
"conceptCode": "ncit:C35571"
}
},
{
"observedValue": "Both disease progression and cancer treatment",
"permissibleValue": {
"value": "NOMATCH",
"conceptCode": null
}
},
{
"observedValue": "Unknown",
"permissibleValue": {
"value": "NOMATCH",
"conceptCode": null
}
}
]
}
}
]
}
]
}
```
**APOLLO-2-leaderboard-Submission.json - bogous submission:**
```
{
"columns": [
{
"columnNumber": 1,
"headerValue": "causeofdeath_apollo2",
"results": [
{
"resultNumber": 1,
"result": {
"dataElement": {
"id": 1,
"name": "abc"
},
"dataElementConcept": {
"id": 2,
"name": "def",
"conceptCodes": []
},
"valueDomain": [
{
"observedValue": "Disease progression",
"permissibleValue": {
"value": "XYZ",
"conceptCode": ""
}
},
{
"observedValue": "Both disease progression and cancer treatment",
"permissibleValue": {
"value": "ABC",
"conceptCode": ""
}
},
{
"observedValue": "Unknown",
"permissibleValue": {
"value": "Unknown",
"conceptCode": "ncit:CABC"
}
}
]
}
}
]
}
]
}
```
**score.sh - wrapper to launch the scoring**
```
#!/bin/bash
FILE_SUB=`pwd`/APOLLO-2-leaderboard-Submission.json
FILE_GOLD=`pwd`/Annotated-APOLLO-2-leaderboard.json
docker run --rm\
-v ${FILE_SUB}:/submission.json:ro \
-v ${FILE_GOLD}:/goldstandard.json:ro \
metadata-scoring score-submission /submission.json /goldstandard.json
```
The partial scores seem to be:
**0, 0, 0.333333333333333, 0**
As you can see, the value coverage gives a 0.33, although in my views that should be zero as well.
Let us know what you think about this.
On a different note: Thank you for organizing the challenge, and keeping it running in these not so ordinary times!
Created by Attila L. Egyedi attila.egyedi Hi @attila.egyedi ,
Thank you for your incredible patience as we attempt to sort through all of the scoring bugs! With respect to this one -- because the observed values are enumerated, we would be more interested in comparing the `conceptCode`s rather than the `value`s. That being said, 0.333 is still incorrect, as there are two mismatches between the bogus submission and goldstandard, not just one; the expected score should be 0.167.
Thank you again for the good catch! Please continue to let us know of any other cases we may have missed.
Best,
Verena
EDIT: formatting Thank you, @attila.egyedi, for pointing this out and for providing steps on reproducing the error!
We will look into this further and hopefully have a(nother) fix out soon.
Best,
Verena
Drop files to upload
Possible bug is scoring - find_mismatch_rows or score_value_coverage page is loading…