In a previous thread Pablo wrote:
> At this time we are not revealing the details of the scoring but it will be based on the position as calculated by the MCC [...] but we will also add weights depending which genes you use and maybe other metrics.
This sounds like different genes would be assigned different weights. What is the rationale for that? Are the weights related to the amount of effort needed to generate the in-situ data, or related to how unique the in-situ pattern is? Would be nice to get at least a hint, since there might be cases where, during gene selection, we think two genes are equally useful, but you might assign different weights during evaluation.
Created by Christoph Hafemeister Christoph.H Something like that, but we have a quantitative way of evaluating the genes Thanks, I see, so it is not that you are looking for specific genes that might appear in the list. Instead, you would be looking for a good justification for how the list of genes has been produced. And, your ranking would be based on the method by which the genes were chosen by the different groups, right? Yes, of course there will be a rationale behind the difference in scoring which will be related to how you chose the genes, also there will be a leaderboard for one submission well we will rank submissions.... Pablo, if I understand this correctly, it means that two gene lists that produce the same level of success in determining cell locations, would be ranked differently in the final score? And the way you would perform this ranking is hidden?
Thanks
Hi christoph,
indeed the positional information contained by the gene will already be factored into the prediction, if the prediction is well performed.
This is what we want to evaluate. We are definitely not penalizing as you suggest in 2.
Pablo Thanks Pablo, but I still don't understand. The positional information contained by the gene will already be factored into the prediction.
At this point, I can think of two weighting strategies
1) Reward the use of genes with a lot of positional information (this seems strange to me, since these genes should already improve the predicted locations - why give extra credit for using them?)
2) Penalize the use of genes with a lot of positional information (this would make sense with respect to the ultimate goal of not using any in-situ data at all)
So, while it is true that two genes with identical positional information would give the same score, I wonder what your motivation is. Hi Christoph,
because the "gold standard" is public, we prefer to keep the details of scoring private in order to keep some unknowns.
However, we can share that the weights are chosen in function of the positional information contained by the genes you selected, so if two genes have the same information there should be no difference in the scoring.
thanks
Pablo
Drop files to upload
What is the rationale for gene weights during evaluation? page is loading…