Hi,
On behalf of my teammate, we killed a Leaderboard training job (submission ID = 8276078) yesterday. The Model State File file under the tracking dashboard is EMPTY. However, I confirmed from the log that the model should be successfully saved to the model state. I further confirmed that Model State Quota Used (GB) is 0.0444 GB which is about the size of our saved model.
I am wondering if there is a way that I can recover my model state files. It's very important to us. Thanks for taking time looking into this issue!
Thanks,
A.S.
Created by Yiqiu Shen ashen Thanks @brucehoff. That solves our problem. Thank you very much! > The Model State File file under the tracking dashboard is EMPTY.
Correct. The file doesn't not show up until you 'redeem' part of your data download quota to unlock/retrieve the model state file. We verified that the model state was indeed archived to Synapse.
> I further confirmed that Model State Quota Used (GB) is 0.0444 GB which is about the size of our saved model.
The size of your model state file is not decremented from your quota until you select to unlock/retrieve it . The quota used may simply be due to your log files (about 17MB for this submission, the rest due to other submissions.)
> I am wondering if there is a way that I can recover my model state files. It's very important to us.
Yes. When you canceled your submission you should have received an email confirmation containing a redemption token along with instructions for how to redeem it. Once you redeem it (by forwarding the email to an alias provided in the message) the model will be unlocked, the amount decremented from your quota, and a link to the model state returned to you.
If for any reason you have not received the expected email message(s) please check your 'spam' folder.