Hi dear organizers,
Before submitting to the leaderboard we are wondering what evaluation function will be used for RF and triplet distance?
The file submission page has given examples of R version of RF distance and python version of triplet score. However, when I was trying to implement the python version of normalized RF, I was unable to replicate the results from the NJ baseline provided. Then I noticed the RF provided in the subchallenge1 was unrooted, rather than rooted tree we are supposed to construct. The scoring code in the file submission page is:
RF = RF.dist(ground_truth,reconstruction,normalize=T)
score = 1-RF
didn't define whether the tree is rooted, therefore is set to default as unrooted. So as the RF in the train_setDREAM2019.txt table in subchallenge 1.
And I wonder if we can get an example of the standard evaluation source code if possible?
Thank you!
Created by Hanrui Zhang rayezh @Jasper06,
That is correct; the smaller the number, the better (for both RF and Triplet).
Best,
Verena Hi @v.chung , thanks for your reply. Just to confirm that in the current leaderboard, 0 is the best score we can get for both RF and Triples right? Dear @Jasper06,
Thank you for participating! The leaderboard is currently displaying the RF distance, not 1 - RF.
Best,
Verena Hi @v.chung, when I'm using the evaluation code you posted, I found the score is calculated as RF instead of 1-RF, which means 0 is the best. I just want to make sure that the current leaderboard still uses 1-RF right? Indeed, you can read more about this here: https://academic.oup.com/sysbio/article/64/2/205/1630737 Yes I get it. it changes a bit numerically but doesn't change the order of similarity right? @jeriscience Hi @rayezh just to clarify adding a root changes a bit the RF score, but it not matter where you add the root you get the same score ;) Hi Pablo and Verena,
Thank you very much! That clears my doubts.
FYI: the roots actually changed a bit RF score, since with root the tree will have one more internal edge. But the TMC function has it well defined.
Best,
Hanrui Dear @rayezh ,
Thank you for participating! You can also find the scoring functionalities for the Challenge [here](https://github.com/Sage-Bionetworks/Allen-DREAM-Challenge/blob/master/Docker/score.py).
Best,
Verena Hi, we implemented the RF and triplets functions from the TMC algorithm as described [here](https://www.synapse.org/#!Synapse:syn20692755/wiki/597065) in the "Example of reconstruction for Subchallenge 3".
The rooting does not change the RF scores, it only matters for the triplets.
We also detected a small difference between the R and python RF scores.
Best
Pablo