Hi everyone.
We've posted round 1 preliminary results.
By dataset correlation leaderboards are [here.](https://www.synapse.org/#!Synapse:syn15589870/wiki/595546).
We've also posted dataset-celltype correlations in tables [here.](https://www.synapse.org/#!Synapse:syn15589870/tables/).
The dataset-celltype values are in tables as there are too many fields to include in a leaderboard. We hope the options for faceting the tables are useful.
Thanks to all the participants for their contributions!
-Andrew
Created by Andrew Lamb andrewelamb Thank you so much, Dr. White.
This information is really helpful.
Best regards,
Wennan Chang
Hi @maja , @rsyu , @chang91 , and all,
Please disregard the dataset correlations. We will soon take them down. For the reasons you describe, these are misleading:
(1) they are not consistent with the median / mean / summary statistic-based scoring approach sketched in the Wiki
(2) they require that the constant of proportionality across cell types be the same and, hence, favor regression-based methods.
I apologize for the confusion and any wasted effort. We are interested in doing this type of comparison, but will do so as part of the post-competition collaborative phase for those methods where it is applicable.
Best,
Brian I strongly agree with @rsyu. The whole data set score is not rational for the relative proportion.
In addition, this data set score is much friendly to the linear regression-based method than the semi-supervised method. Hi Andrew,
Thank you very much for the explanation.
It seems that this method should work fine in most cases. However, it may have problems when dealing with a predictor that produces outputs at very different scales for different cell types, e.g., those based on averaging marker gene expression like MCP counter. For example, let's say it's predictions for cell type A that are significantly larger, say 100x, compared to predictions for other cell types. In that case, would the dataset correlation score be dominated by the correlation of cell type A after combination? In that case, would it be better that we perform some sort of normalization before we group prediction results from different cell types together? Hi,
Sorry for being slow, but in the scoring description it says that "the dataset aggregate score will be the median (or mean) cell-type-specific score, calculated across all cell types", but in the dataset scores here thats not the case, right? This is a bit hard to explain, but I'll try with an example of three cell types A, B, and C:
I interpreted from the scoring description that the score for dataset DS1 would be median(correlation_A, correlation_B, correlation_C), but in these leaderboard results the dataset score is defined as correlation_c(A,B,C), where _c(A,B,C) indicates that that the three cell types are combined into one vector. Did I misunderstand something? If not, which metric is going to be used in the final validation?
Thank you for your patience.
BR,
Maria Dear Andrew,
Thank you so much for the explanation.
It is much more clear now for me as well.
Kind regards,
Yongsoo Thanks Andrew! yes it does, in fact I thought that that was the possible explanation to get those kinds of results.
In this case you are indirectly scoring the accuaracy of the proportionality constant, which in my case I did't even try to get, because it was not clear to me that this kind of scoring was going to be in the challenge. Anyway, is good to know and now is clear to me, shouldnt be that hard to correct (i hope hehe).
Thank you for taking the time to explain it trough!
Best,
Martin Hi @martinguerrero89 @yokim,
I think there is some misunderstanding on what the dataset correlation leaderboards are showing, and it's probably my fault for not being clear.
The dataset-celltype scores I think are clearly understood.
If you broke down a submission into its N datasets, and then broke those down further into M cell-types, and did a correlation with the measured values you would have N x M correlation scores (minus celltypes for which a dataset had no measurements)
DS CT P M
----------------
1 A .1 .2
1 A .3 .4
-------
1 B .7 .9
1 B .8 .6
-------
2 A .4 .5
2 A .3 .6
-------
2 B .5 .6
2 B .6 .5
In the above example you'd have a correlation for DS1_CTA, DS1_CTB, DS2_CTA, DS2_CTB
The dataset scores are exactly the same, except we didn't break the datasets down by cell-type. In other words for each dataset we correlated all predictions vs measurements together.
DS CT P M
----------------
1 A .1 .2
1 A .3 .4
1 B .7 .9
1 B .8 .6
-------
2 A .4 .5
2 A .3 .6
2 B .5 .6
2 B .6 .5
Using the same example we would have two correlations: DS1 and DS2. You can see while the cell-type correlations may be poor, but when you group them just by dataset, the correlation could potentially improve.
Does this make sense? I apologize for the confusion.
-Andrew
I am also confused by the scores.
How is the per-data set score measured?
I thought this is the median across the per cell type correlation.
However, we get correlation > 0.9 for a data set while we did not score higher than 0.9 correlation for any of the cell types in the dataset.
Thank you Andrew for your response! I get your point, but how come that people get >0.95 in all celltypes in a given dataset and then get <0.75 as a datasets score?
or get 0.9 and -0.55 in two different submissions that performed very similar according the cell type prediction (and are supposed to had just a minor tweak) ?
I'm trying to understand the dataset metrics and the math just does not fit according to (what i understand from) the explanation given in the challenge Hi @martinguerrero89
I don't think we sent out the dataset correlations before so these shouldn't match anything you've seen. Thanks for uploading the results!
I am a little bit puzzled by the dataset correlation leaderboards [(this)](https://www.synapse.org/#!Synapse:syn15589870/wiki/595546). They don't match with my results at all, while the dataset-celltype does.
Am I missing something?
Thanks!