It is now over 7 months since the end of the challenge. Please update us on what is happening with the evaluation. Thanks!
Yudi
Created by Yudi Pawitan yudpaw We're still working on the evaluation. Generating matched long read data to validate the calls in cell lines has been much slower than anticipated. But we are still actively working on the benchmark.
If you have new methods to submit for the community phase, please submit them. We'll add them to the evaluation queue and make sure it is part of the final leaderboard.
I also wanted to make sure we are clear, the purpose of the community phase is to make sure the leaderboard is fully representative of full range of possibilities, including meta callers. So we will be sure to run all of them for the paper. But they won't be considered 'official' challengers, because at this point anyone can see all of the entries ( at https://github.com/smc-rna-challenge ). Thanks Kyle for the update. Do you know approximately about the timeline for getting the long-read data? We are very eager to submit our new algorithm for rna-fusion for the community-phase but we do not know if the evaluation committee is still active?
I know these evaluation tasks are difficult and time consuming, so thank you very much for your organizing and evaluating efforts!
Son Part of our experimental design was to design several novel fusion spike ins, and test the methods using the short read sequencing. We know the detection rates for methods, but we must also consider the pre-existing fusions that were already part of the cell lines. If challenge methods detected these, but we called them 'false positives' because they aren't part of the spike ins we added, that would be bad. For that we are doing long read sequencing to detect the pre-existing fusions in the cell lines. We've had multiple issues on that front. But we've changed sequencing labs and are getting the data generated. Once this data is ready, we'll be able to start posting the final leaderboards.