Hi. I'd like some clarifications about the lifetime of /modelState:
- If I cancel a running training container, is modelState still saved?
- If I run a second training container after the first one finishes, will it have access to the previous modelState? If not will it replace the previous modelState when it finishes, or will both be saved?
Best regards,
Created by Matthieu Nottale mnottale > If I cancel a running training container, is modelState still saved?
Yes. We save your model state if either (1) it completes normally, (2) you cancel it or (3) you exceed your server time quota and we cancel it. We do not save your model (1) if it terminates with an error condition or (2) if, after zipping, it exceeds 5GB.
> If I run a second training container after the first one finishes, will it have access to the previous modelState?
The second submitted training container will not have access to the saved state of the first one.
> If not will it replace the previous modelState when it finishes, or will both be saved?
Both will be saved. This is central to the design of the challenge. You may use your server time quota as you see fit to train models and we save the states of those models. We hope that by examining the logs we return to you, you can pick your top three models in each five week round. You then submit for scoring a 'scoring container' along with the Synapse ID of the chosen model state, up to three times per round. We assemble your submitted container, the reconstituted model state, and the scoring data, to generate your predictions, which we then score and publish to the leaderboard.