Our submission 9697495 failed for fast_lane but the train and infer logs seems about right.
```
STDERR: 2020-01-10T00:16:16.605837475Z WARNING:toil.leader:o/O/jobt_BaV9 Welcome, ehrdreamservice!
STDERR: 2020-01-10T00:16:16.605913205Z WARNING:toil.leader:o/O/jobt_BaV9
STDERR: 2020-01-10T00:16:16.605979185Z WARNING:toil.leader:o/O/jobt_BaV9 Traceback (most recent call last):
STDERR: 2020-01-10T00:16:16.605988945Z WARNING:toil.leader:o/O/jobt_BaV9 File "annotationSubmission.py", line 75, in
STDERR: 2020-01-10T00:16:16.606068365Z WARNING:toil.leader:o/O/jobt_BaV9 _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10)
STDERR: 2020-01-10T00:16:16.606138215Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 105, in _with_retry
STDERR: 2020-01-10T00:16:16.606181546Z WARNING:toil.leader:o/O/jobt_BaV9 six.reraise(exc_info[0], exc_info[1], exc_info[2])
STDERR: 2020-01-10T00:16:16.606249885Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 53, in _with_retry
STDERR: 2020-01-10T00:16:16.606366575Z WARNING:toil.leader:o/O/jobt_BaV9 response = function()
STDERR: 2020-01-10T00:16:16.606378505Z WARNING:toil.leader:o/O/jobt_BaV9 File "annotationSubmission.py", line 75, in
STDERR: 2020-01-10T00:16:16.606435835Z WARNING:toil.leader:o/O/jobt_BaV9 _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10)
STDERR: 2020-01-10T00:16:16.606468946Z WARNING:toil.leader:o/O/jobt_BaV9 File "annotationSubmission.py", line 61, in annotate_submission
STDERR: 2020-01-10T00:16:16.606547006Z WARNING:toil.leader:o/O/jobt_BaV9 status = syn.store(status)
STDERR: 2020-01-10T00:16:16.606575386Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 912, in store
STDERR: 2020-01-10T00:16:16.606722586Z WARNING:toil.leader:o/O/jobt_BaV9 return type(obj)(**self.restPUT(obj.putURI(), obj.json()))
STDERR: 2020-01-10T00:16:16.606736916Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 3368, in restPUT
STDERR: 2020-01-10T00:16:16.606814417Z WARNING:toil.leader:o/O/jobt_BaV9 exceptions._raise_for_status(response, verbose=self.debug)
STDERR: 2020-01-10T00:16:16.606855746Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/synapseclient/exceptions.py", line 153, in _raise_for_status
STDERR: 2020-01-10T00:16:16.606928526Z WARNING:toil.leader:o/O/jobt_BaV9 raise SynapseHTTPError(message, response=response)
STDERR: 2020-01-10T00:16:16.606961127Z WARNING:toil.leader:o/O/jobt_BaV9 synapseclient.exceptions.SynapseHTTPError: 412 Client Error:
STDERR: 2020-01-10T00:16:16.607044526Z WARNING:toil.leader:o/O/jobt_BaV9 Submission Status: 9697495 was updated since you last fetched it, retrieve it again and reapply the update
STDERR: 2020-01-10T00:16:16.607072307Z WARNING:toil.leader:o/O/jobt_BaV9 [job annotate_submission.cwl] Max memory used: 47MiB
STDERR: 2020-01-10T00:16:16.607147007Z WARNING:toil.leader:o/O/jobt_BaV9 INFO:cwltool:[job annotate_submission.cwl] Max memory used: 47MiB
STDERR: 2020-01-10T00:16:16.607167857Z WARNING:toil.leader:o/O/jobt_BaV9 [job annotate_submission.cwl] completed permanentFail
STDERR: 2020-01-10T00:16:16.607273867Z WARNING:toil.leader:o/O/jobt_BaV9 WARNING:cwltool:[job annotate_submission.cwl] completed permanentFail
STDERR: 2020-01-10T00:16:16.607319377Z WARNING:toil.leader:o/O/jobt_BaV9 Traceback (most recent call last):
STDERR: 2020-01-10T00:16:16.607384397Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 331, in workerScript
STDERR: 2020-01-10T00:16:16.607394548Z WARNING:toil.leader:o/O/jobt_BaV9 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
STDERR: 2020-01-10T00:16:16.607473677Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1378, in _runner
STDERR: 2020-01-10T00:16:16.607520087Z WARNING:toil.leader:o/O/jobt_BaV9 returnValues = self._run(jobGraph, fileStore)
STDERR: 2020-01-10T00:16:16.607552717Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1323, in _run
STDERR: 2020-01-10T00:16:16.607617818Z WARNING:toil.leader:o/O/jobt_BaV9 return self.run(fileStore)
STDERR: 2020-01-10T00:16:16.607697368Z WARNING:toil.leader:o/O/jobt_BaV9 File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 606, in run
STDERR: 2020-01-10T00:16:16.607731737Z WARNING:toil.leader:o/O/jobt_BaV9 raise cwltool.errors.WorkflowException(status)
STDERR: 2020-01-10T00:16:16.607811358Z WARNING:toil.leader:o/O/jobt_BaV9 WorkflowException: permanentFail
STDERR: 2020-01-10T00:16:16.607821488Z WARNING:toil.leader:o/O/jobt_BaV9 ERROR:toil.worker:Exiting the worker because of a failed job on host 8377945867b5
...
...
STDERR: 2020-01-10T00:52:14.664387829Z INFO:toil.leader:Finished toil run with 4 failed jobs.
STDERR: 2020-01-10T00:52:14.664596650Z INFO:toil.leader:Failed jobs at end of the run: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python o/O/jobt_BaV9 'CWLWorkflow' C/h/job3MXbms 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python 6/0/jobEAqGyI 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python m/S/job60Vf7v
STDERR: 2020-01-10T00:52:14.744867517Z Traceback (most recent call last):
STDERR: 2020-01-10T00:52:14.744895347Z File "/usr/local/bin/toil-cwl-runner", line 8, in
STDERR: 2020-01-10T00:52:14.744900197Z sys.exit(main())
STDERR: 2020-01-10T00:52:14.744903877Z File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1276, in main
STDERR: 2020-01-10T00:52:14.745107499Z outobj = toil.start(wf1)
STDERR: 2020-01-10T00:52:14.745140639Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 781, in start
STDERR: 2020-01-10T00:52:14.745282159Z return self._runMainLoop(rootJobGraph)
STDERR: 2020-01-10T00:52:14.745289889Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 1054, in _runMainLoop
STDERR: 2020-01-10T00:52:14.745572472Z jobCache=self._jobCache).run()
STDERR: 2020-01-10T00:52:14.745579222Z File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 246, in run
STDERR: 2020-01-10T00:52:14.745585242Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
STDERR: 2020-01-10T00:52:14.745595562Z toil.leader.FailedJobsException
```
any clue of what can we do? We resubmitted right before the deadline, but what if it fails again?
Created by Carlos Vega carlosvega Thanks! Thanks Tim! I've changed the status of 9694795 to ACCEPTED and it has been queued to run on the UW server.
Thanks,
Tim 
It produced ERROR but VALIDATED
But when we resubmitted it, it produced only ERROR
${badge?isUser=false&id=3381295} If you could send it to main challenge and the UW that would be great. The submission 9697495 Worked fine. Both train and infer logs seem alright. Please do, the submission was marked as failed. Hi @carlosvega,
That looks like an occasional error we see that won't affect your submission. And if it does, we can fix it pretty quick.
Thanks,
Tim