Hi,
for round 3 is the data just a 3rd bootstrap of the same set aside dataset that was used for rounds 1 and 2?
thanks
DA
Created by exquirentibus veritatem exquirentibus That would be one way to go but we are not doing that for this round since the round and the rewards were an afterthought. The leaderboard rounds are not large enough to support that approach. As noted in the wiki we are only using the primary metric here and *not* employing the tie-breaking metric after computing the Bayes factor. I expect many teams would be statistically tied based on the Bayes factor since there are fewer samples. But given that this third round contains rewards associated shouldn't it be more stable than the others by perhaps generating multiple bootstraps and averaging?
It seems to me that a single bootstrap will have a lot of variance.
DA Exactly!
I am actually working generating it now.