Hi,
My submission (submission ID 8398083) rerun the preprocessing stage, although it was completed in the past.
Can you please help?
Created by Eli Meirom bloodymeli > my training model started running the preprocessing again.
That would make sense if you halted your first submission before preprocessing had completed. Other reasons preprocessing might start again include the need to remove a server where your preprocessed data was stored from the cluster (though this has not happened in a long time) or if you had not submitted in months (and so your 10TB slot was freed up for active participants). The bottom line is that we seek to cache the latest preprocessing output for your team, but we cannot guarantee that we do it 100% of the time.
Also the caching logic treats the preprocessing output as atomic and exclusive: There is no caching of part of your preprocessing output or combining the result of one preprocessing run with another.
> Is there any chance that the data that I used during preprocessing is still there, or is there a problem because I halted it before it completed.
Our records show no cached preprocessing results for QUT_EECS_imaging team [https://www.synapse.org/#!Team:3343461]. If you terminate a job before preprocessing is complete then the 10TB cache is freed up for someone else to use. I have also had problems with my preprocessing being run more than once.
My preprocessing submission (submission ID 8376165) was run, though I halted it before it completed as I wanted to save more time for training my model. I tried to submit a training model (submission ID 8436194) using the scans that I did preprocess, which was roughly 250k of them, my training model started running the preprocessing again.
@thomas.yu @brucehoff is there any chance that the data that I used during preprocessing is still there, or is there a problem because I halted it before it completed.
My model will still perform well with training on just a few thousand scans, so if it is gone I could make a submission just preprocessing and training on fewer scans to have a valid submission before the challenge ends. I am hoping though that the preprocessed data that I have already done is still available, as my preprocessing is quite comprehensive and has already used a significant amount of my computational quota.
Thanks for all your help :)
Ethan Dear Eli,
My apologies for the delay. I have marked your submission as invalid.
Best,
Tom @thomas.yu
@brucehoff
Can you please assist? It has been 3 days now... I cancelled the job - @thomas.yu , please mark it as invalid.
Please let me know when I can resubmit as a team. @thomas.yu If he cancels his job you can mark it 'invalid' which should allow him to submit on behalf of his team (that is, if he hasn't submitted any other jobs during this round solo).
Maybe @thomas.yu can help?
Any administrator around? Hmm, anyone? @@brucehoff
My guess is that maybe the preprocessing had restarted as I activated the script from my account rather than my team's account. In this case, it is poorly documented.
If this is indeed the case, and as my team was dissembled, may I ask that you abort this task, returning the corresponding quota, and either :
A) Copy / duplicate my team's preprocessing folder to my account so I could rerun only the training process.
B) Delete this training submission (it was the first one) and allow me to perform all following submissions in this round using my team's account.
Thanks!