Dear organizers,
I've two questions regarding leaderboard rounds:
* In the datasets used for leaderboard round 1, it seems that no fibroblasts nor endothelial cells were measured experimentally? At least the e-mail with the scores didn't contain any correlation information for these cell types.
* I know that these cell types are not commonly measured experimentally, but will there be some datasets including these at least in the round 2? Because it will be very hard to improve our model between the rounds if it not the same cell types that are tested in the leaderboard and validation phases.
* Will the scores from the different teams be posted on the leaderboard page? Because without such a comparison, it's difficult to know if our results are good or bad (can better scores be achieved or is it just the data that contains biases, etc.).
Thank you. Best wishes,
Julien
Created by Julien Racle jracle Ok, thank you @brian.white for the answers.
Best,
Julien Hi @jracle
Yes, we will have a dataset with fibroblasts and endothelial cells in leaderboard round 2. Unfortunately, it will have only fibroblasts and endothelial cells. We are going to introduce a third leaderboard round. I will try to include a richer dataset there (with fibroblasts, endothelial, and immune cells).
We will post pearson and spearman correlation broken down by team, dataset, and cell type at the conclusion of the round. This was an intentional decision to avoid introducing bias--for example, if we posted results while the round was active, those who submitted later might be glean additional information (such as what cell types are missing) that participants submitting early would not be privy to.
Best,
Brian
@jracle
Brian will be getting back to you shortly.
@tomsnir
"To clarify, per queue means per sub-challenge, in total, correct? not two submissions in queue at a given time, suggesting we can submit a larger number as long as no more than two submissions are processing at a time. Apologies if this is a given, we want to be certain."
For each leaderboard phase, you will be allowed two coarse-grained submissions, and two fine-grained submissions.
Each queue has a limit of two which includes all previous valid submissions, and submissions that are currently being processed.
Does that answer your question?
"Unless we get an email back with scores, that submission is not considered valid?"
You should get an e-mail back either way, but yes, you will get one back with scores if you make a valid submission.
"The higher score of the two submissions will be the one posted to the leaderboards?"
Both will be posted, but the higher of the two will count as your official score.
"When can we expect to see everyone's scores so that we can make adjustments? Will the scores be posted at the end of the leaderboard phase only?"
This is correct. I'd like to add a relevant question regarding the max number of possible submissions. @andrewelamb wrote that we are "allowed a maximum of two valid submissions per queue."
- To clarify, per queue means per sub-challenge, in total, correct? not two submissions in queue at a given time, suggesting we can submit a larger number as long as no more than two submissions are processing at a time. Apologies if this is a given, we want to be certain.
- Unless we get an email back with scores, that submission is not considered valid?
- The higher score of the two submissions will be the one posted to the leaderboards?
- When can we expect to see everyone's scores so that we can make adjustments? Will the scores be posted at the end of the leaderboard phase only?
Thank you,
TS.