I recall from previous threads that building different models for different validation sets is allowed. This, however, might create loopholes in light of the first round leader board results.   For example, it is interesting to note from Ch2 leader board that many teams' iAUC fared pretty well on DFCI, Hose, and GSE15965, but most had flipped iAUC (e.g, 0.2) for M2Gen. So one way to improve is then to simply put a negative sign to the original score for M2Gen (so that iAUC becomes 0.8 now), but keep others intact. This does not seem to break the rule, as it amounts to using two models, one being the reverse of the other. However, these 2 models are entirely contradictory to each other, and certainly not usable in any clinical sense!!   Therefore, it seems more reasonable to only allow a single model for all validation datasets, instead of multiple models which might be tuned to each dataset's own idiosyncrasy. From a translational point of view, a single, consistent model might be also preferred.