Hi, I am a bit at a loss here. The workflow works well locally but somehow fails when submitted to fast lane. I paste the log below for details. Any idea what might be happening?
STDERR: 2019-10-13T19:04:37.917678098Z INFO:cwltool:Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/EHR-challenge-develop/synthetic_docker_agent_workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/EHR-challenge-develop/synthetic_docker_agent_workflow.cwl'
STDERR: 2019-10-13T19:04:40.129806482Z EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-13T19:04:40.129842591Z WARNING:salad:EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-13T19:04:40.131858898Z EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-13T19:04:40.131878429Z WARNING:salad:EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-13T19:04:44.753143732Z EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-13T19:04:44.753192783Z WARNING:salad:EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-13T19:04:45.021033997Z EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-13T19:04:45.021054977Z WARNING:salad:EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-13T19:04:46.244568681Z WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (32).
STDERR: 2019-10-13T19:04:46.244602722Z WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (268108005376).
STDERR: 2019-10-13T19:04:46.245782166Z WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (316986105856).
STDERR: 2019-10-13T19:04:46.500657315Z INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-13T19:04:47.012569743Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:47.013704767Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmpTnTH73/worker_log.txt
STDERR: 2019-10-13T19:04:47.885566919Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python F/b/jobVujKiT with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-13T19:04:47.892110922Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python z/t/job8AjC_O with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-13T19:04:47.893700617Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python T/b/jobIAY7cJ with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-13T19:04:47.894685420Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python t/a/jobytkgQj with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-13T19:04:48.430614352Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:48.431922817Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmprU2GfL/worker_log.txt
STDERR: 2019-10-13T19:04:48.438248089Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:48.439452253Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmpmpa1pu/worker_log.txt
STDERR: 2019-10-13T19:04:48.447167549Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:48.448432584Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmpkwVKhQ/worker_log.txt
STDERR: 2019-10-13T19:04:48.476415601Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:48.477446104Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmp3OAc5z/worker_log.txt
STDERR: 2019-10-13T19:04:50.561745490Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python z/t/job8AjC_O
STDERR: 2019-10-13T19:04:51.377990533Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python T/b/jobIAY7cJ
STDERR: 2019-10-13T19:04:51.592873337Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python F/b/jobVujKiT
STDERR: 2019-10-13T19:04:51.593137118Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python t/a/jobytkgQj
STDERR: 2019-10-13T19:04:51.594048342Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python E/z/jobD3jbef with job batch system ID: 5 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-13T19:04:52.068700234Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:52.069690787Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmpCms2QW/worker_log.txt
STDERR: 2019-10-13T19:04:55.220385796Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python E/z/jobD3jbef
STDERR: 2019-10-13T19:04:55.221578861Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/EHR-challenge-develop/run_synthetic_training_docker.cwl' python q/2/jobTfpoAa with job batch system ID: 6 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-13T19:04:55.228512005Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python X/0/job3mgrZC with job batch system ID: 7 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-13T19:04:55.719282985Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:55.720444210Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmpce1xlx/worker_log.txt
STDERR: 2019-10-13T19:04:55.726165310Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/tmpL2STC3'.
STDERR: 2019-10-13T19:04:55.727424084Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/8bba551a-5a4b-40f6-810c-0432323d4c3a/toil-2de22dc6-92f9-4e46-9c53-f802ead5edf8-e89ed93837262f1d6b20cd9e97e84170/tmpXcMWZ8/worker_log.txt
STDERR: 2019-10-13T19:04:58.696651469Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python X/0/job3mgrZC
STDERR: 2019-10-13T20:04:47.181103090Z INFO:toil.leader:Reissued any over long jobs
Created by tgaudelet Hi Tim, thanks for the swift reply!
Yes, it definitely takes more than an hour (reading the data pretty much takes the hour...).
However, it fails after an hour as well when submitted to the Challenge directly. The time quota for the Challenge should be 6 hrs, right? Pasted the error log when submitted to the challenge (not fast lane).
Cheers,
Thomas
STDERR: 2019-10-14T19:34:57.972547304Z INFO:cwltool:Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/EHR-challenge-develop/main_docker_agent_workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/EHR-challenge-develop/main_docker_agent_workflow.cwl'
STDERR: 2019-10-14T19:35:00.013205723Z EHR-challenge-develop/main_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-14T19:35:00.013221243Z WARNING:salad:EHR-challenge-develop/main_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-14T19:35:00.014655760Z EHR-challenge-develop/main_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-14T19:35:00.014669090Z WARNING:salad:EHR-challenge-develop/main_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-14T19:35:04.014420697Z EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-14T19:35:04.014446088Z WARNING:salad:EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-14T19:35:04.250904682Z EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-14T19:35:04.250930832Z WARNING:salad:EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-14T19:35:05.432039658Z WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (32).
STDERR: 2019-10-14T19:35:05.432061559Z WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (268108005376).
STDERR: 2019-10-14T19:35:05.432066059Z WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (314849574912).
STDERR: 2019-10-14T19:35:05.648392807Z INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-14T19:35:06.078360521Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:06.079405906Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmpGd7Z80/worker_log.txt
STDERR: 2019-10-14T19:35:06.877706911Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python y/7/job_3N5Xw with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-14T19:35:06.885719900Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python t/W/joba3cyIM with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-14T19:35:06.886418053Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python k/T/jobKFR64y with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-14T19:35:06.886967116Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python T/7/jobLOwk4b with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-14T19:35:07.323627772Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:07.324530196Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:07.324544646Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmp6VnD_6/worker_log.txt
STDERR: 2019-10-14T19:35:07.325856822Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmpMHzAhC/worker_log.txt
STDERR: 2019-10-14T19:35:07.348066607Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:07.348804991Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmpX_dUa2/worker_log.txt
STDERR: 2019-10-14T19:35:07.352220127Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:07.353196592Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmptbvSe_/worker_log.txt
STDERR: 2019-10-14T19:35:09.053059192Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python k/T/jobKFR64y
STDERR: 2019-10-14T19:35:09.769766367Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python y/7/job_3N5Xw
STDERR: 2019-10-14T19:35:10.034538624Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python T/7/jobLOwk4b
STDERR: 2019-10-14T19:35:10.066124845Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python t/W/joba3cyIM
STDERR: 2019-10-14T19:35:10.066931809Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python u/g/jobR93Wmd with job batch system ID: 5 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-14T19:35:10.505532213Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:10.506373156Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmp2gQF2b/worker_log.txt
STDERR: 2019-10-14T19:35:13.284965726Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python u/g/jobR93Wmd
STDERR: 2019-10-14T19:35:13.286286453Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python Y/c/jobcQPEg9 with job batch system ID: 6 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-14T19:35:13.294363011Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/EHR-challenge-develop/run_synthetic_training_docker.cwl' python A/F/job1JVYAz with job batch system ID: 7 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-14T19:35:13.720186953Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:13.721108827Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmp0miaqj/worker_log.txt
STDERR: 2019-10-14T19:35:13.729941469Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/tmpo7Zo7l'.
STDERR: 2019-10-14T19:35:13.730937924Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0d142f37-a53e-4d52-90ea-c3d03b3977c9/toil-66486f3d-063f-4f13-8f2b-e01d4f97b358-e89ed93837262f1d6b20cd9e97e84170/tmps3XuhW/worker_log.txt
STDERR: 2019-10-14T19:35:16.311553767Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python Y/c/jobcQPEg9
STDERR: 2019-10-14T20:35:06.730160663Z INFO:toil.leader:Reissued any over long jobs Hi @tgaudelet,
From our logs, it looks like your models ran over the allotted time quota. How long is it taking your models to run on your local environment? The time quota on the fast lane is currently 1 hour.
Thanks,
Tim
Drop files to upload
Workflow repeatedly fails when submitted but runs fine locally page is loading…