Could the organizers please confirm that iAUC will be used as the primary metric in final ranking, rather than a mixture of iAUC and BAC? Please also clarify how close two iAUC scores have to be for the organizers to use BAC for ranking.
Created by Wei Dong wdong bayes factor of 5 corresponds to about 0.03-0.05 on leaderboard, and 0.015-0.02 for final. rough estimation. the real deviation on leaderboard is about 0.05-0.1, as one of my model has been the same across all rounds so i can estimate. but as bf is calculated within the same population, it tends to beoverestimation. e.g. if you have only 2 example, all bf will be infinity.
let us take round 2 for example. for sub1, smith will likely be sole winner. for sub 2, it will be likely shared between sugo and pirrmoro, because primary is so close, second will decide. two iAUC will be compared with a bayes factor cutoff of 3-5, by sampling the examples 80% or bootstrap. In case iAUC tie, the second metric will be used. In case both tie, the winning position maybe shared.
For your information, challenges of this sample size has resulted in 0 to 7 co-winners per sub-challenge, with almost even distribution. it was claimed that 0 had occured before when no one statstically beat baseline according to primary metrics, which hasn't occur in recent years so i can't tell. Hi Wei,
We will be using the iAUC as the primary metric with BAC as the tie-breaking metrics (please [section 3.7](https://www.synapse.org/#!Synapse:syn6187098/wiki/449444) of the wiki for details).