Hello,
My submission with ID 8096198 stopped updating the log file 4 hours ago, but according to the Dashboard it is still running. Could you please take a look at it? I need the log file updated, because if something went wrong during training I would better stop the container and save my training time.
Thanks!
Created by Yaroslav Nikulin (Therapixel) ynikulin Dear Daniel,
The column that says 1/31/2017: 9:37:23AM is actually the last updated date. Whenever the submission status gets updated it will update that date. That is not the time that your model is done processing. Does the log file look like what your model outputs?
It does appear that you finished your training in 11 minutes, which is why it only counted 11 minutes towards your submission quota.
Best,
Thomas
My log files also got updated, but it seems like they're coming from the wrong job! It looks like I'm having the same issue again as I had before, where the job had a wrong log file associated with it!
https://www.synapse.org/#!Synapse:syn4224222/discussion/threadId=1510&replyId=9789
Please check for yourself. The Submission Tracking Dashboard says that job 8103447 ran from 01/31/2017 01:18:22AM until 01/31/2017 09:37:23AM. However, open the log file for the job and you can see it says it started at "06:20:39 UTC" and ended at "06:43:43 UTC", so after a mere 23 minutes, rather than after more than eight hours! The times don't match either.
Here's the problematic submission in the dashboard. It only counts 11 minutes towards the quota.
${imageLink?synapseId=syn8104955&align=None&responsive=true}
I've seemingly seen this issue twice now where the log files and jobs do not match. It's very easy to not realize that you're looking at the wrong log.
Does this bug make sense to the DREAM team at all? Thanks.
I've suddenly got an updated log. Seems like the server was overwhelmed ? Same here with submission ID 8103447, which is still running but the logs have stopped updating. Thanks for your assistance, team!