I submitted a job (Synapse ID: syn7383553) a few days ago but the log file has not been updated since 10/18/2016 2am. So far, I haven't had any success for a job that costs more than 10 hours. Has anyone had any success with long jobs?

Created by Li Shen thefaculty
@mobileroaming: At the moment if your preprocessing log file hits 1MB then log file capture is turned off, both for the remainder of preprocessing as well as for training. This was not our intent. We will change the system so that you can get two log files, one for each phase, each having its own size limit.
It would appear that I am having this same problem. I received an email notification about submission ID 7373553 having completed its training phase. The log of this submission seems to have met the max size of 1MB at version 42, however (https://www.synapse.org/#!Synapse:syn7419065), and so only preprocessing outputs were captured. The log contains no feedback about the subsequent training performed.
Yes, that's is job I was referring to. So it eventually updated but there was a long delay... We can do a simple calculation to see how much space we need for a log file. Let's say I'm processing all the image files and would like to emit a string for each file to indicate the status. Assume each string is 80 bytes. Multiple that by 100,000 images equals 8MB. But we often process an image multiple times. Let's say five times maximum. Then we'll need 8*5=40MB. The zip can usually achieve 4-5 times compression so we'll need 8-10MB for a log file to be safe. @brucehoff
Can you please give the ID of the submission you are asking about? (The 10/21 software update added submission IDs to the notifications we send you, to help address any questions that arise about them.) At this time there is one submission from you, ID 7421927. It is up and running but the log file reached 1MB as you can see here: https://www.synapse.org/#!Synapse:syn7421946 So it will not update further. > BTW, don't you think 1MB size limit for log files is a bit too restricted? We are open to a discussion about the size limit restriction but the log file cannot be unrestricted in size.
Hi @brucehoff, Thank you for addressing the **row id** problem! However, I'm not sure if the log file not updating problem is related to that. I resubmitted the job but the log file hasn't updated since 9:14 am EST. It is 10:07 pm EST now. I also don't think it is related with the size limit because it is only 4KB now: ``` Version 34 dmchallengeit 2016-10-21 4.02 KB ``` BTW, don't you think 1MB size limit for log files is a bit too restricted? Especially given the situation that this is the only way we can get feedback on our model training...
Hi: Your question is regarding submission ID 7383545. It encountered the problem "Row id: has been changed" which we have now addressed. If you would like the submission restarted, please let us know. We would be happy to do so.   On another note, looking at the file history of your logs: https://www.synapse.org/#!Synapse:syn7364963 ("click on File History" to see the history), the job originally started on 10/14 and within the day reached the 1MB log size limit. Once the limit is reached the logs will stop being accumulated though your job will continue to run. To get continuous feedback for a long running job you would have to collect less verbose log information.   Hope this feedback is useful.

Log file has not been updated for a long time page is loading…