I have a docker image that I am able to run locally following the instructions in the tutorial. However, when I submit it using the process shown in the webinar, I get an error log I can't make any sense of. Any advice would be greatly appreciated:
STDERR: 2019-10-03T06:31:44.372894512Z INFO:cwltool:Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/EHR-challenge-develop/synthetic_docker_agent_workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/EHR-challenge-develop/synthetic_docker_agent_workflow.cwl'
STDERR: 2019-10-03T06:31:46.518210141Z EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-03T06:31:46.518225501Z WARNING:salad:EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-03T06:31:46.519693296Z EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-03T06:31:46.519707827Z WARNING:salad:EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-03T06:31:50.609203349Z EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-03T06:31:50.609232199Z WARNING:salad:EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-03T06:31:50.855339782Z EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-03T06:31:50.855355982Z WARNING:salad:EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-03T06:31:51.896111825Z WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (32).
STDERR: 2019-10-03T06:31:51.896128936Z WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (268107968512).
STDERR: 2019-10-03T06:31:51.896325566Z WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (359171788800).
STDERR: 2019-10-03T06:31:52.148559461Z INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-03T06:31:52.572297680Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN'.
STDERR: 2019-10-03T06:31:52.573510254Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/toil-830d0062-b533-41fd-8bc7-cc2c1a142fae-e89ed93837262f1d6b20cd9e97e84170/tmpjYA6QW/worker_log.txt
STDERR: 2019-10-03T06:31:53.380268706Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python l/j/jobCXOviK with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-03T06:31:53.387817272Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/G/jobTxtDR7 with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-03T06:31:53.388389934Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python L/4/jobicfjX2 with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-03T06:31:53.389115786Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python W/k/job_UU0Nq with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-03T06:31:53.823578822Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN'.
STDERR: 2019-10-03T06:31:53.824881717Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/toil-830d0062-b533-41fd-8bc7-cc2c1a142fae-e89ed93837262f1d6b20cd9e97e84170/tmpOxCaNQ/worker_log.txt
STDERR: 2019-10-03T06:31:53.844107681Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN'.
STDERR: 2019-10-03T06:31:53.845019834Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/toil-830d0062-b533-41fd-8bc7-cc2c1a142fae-e89ed93837262f1d6b20cd9e97e84170/tmpr_UuYQ/worker_log.txt
STDERR: 2019-10-03T06:31:53.858849810Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN'.
STDERR: 2019-10-03T06:31:53.859885104Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/toil-830d0062-b533-41fd-8bc7-cc2c1a142fae-e89ed93837262f1d6b20cd9e97e84170/tmpZBbEbG/worker_log.txt
STDERR: 2019-10-03T06:31:53.865542733Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN'.
STDERR: 2019-10-03T06:31:53.866435276Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/toil-830d0062-b533-41fd-8bc7-cc2c1a142fae-e89ed93837262f1d6b20cd9e97e84170/tmpdKfkDk/worker_log.txt
STDERR: 2019-10-03T06:31:55.656274849Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python W/k/job_UU0Nq
STDERR: 2019-10-03T06:31:56.372660183Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python L/4/jobicfjX2
STDERR: 2019-10-03T06:31:56.587734815Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/G/jobTxtDR7
STDERR: 2019-10-03T06:31:56.588149917Z WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/G/jobTxtDR7
STDERR: 2019-10-03T06:31:56.588215567Z WARNING:toil.leader:R/G/jobTxtDR7 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2019-10-03T06:31:56.588292578Z WARNING:toil.leader:R/G/jobTxtDR7 INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-03T06:31:56.588355738Z WARNING:toil.leader:R/G/jobTxtDR7 [job get_submission_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN/b/8/out_tmpdirCh_EU6$ python \
STDERR: 2019-10-03T06:31:56.588465938Z WARNING:toil.leader:R/G/jobTxtDR7 get_submission_docker.py \
STDERR: 2019-10-03T06:31:56.588474659Z WARNING:toil.leader:R/G/jobTxtDR7 -s \
STDERR: 2019-10-03T06:31:56.588594599Z WARNING:toil.leader:R/G/jobTxtDR7 9693558 \
STDERR: 2019-10-03T06:31:56.588605279Z WARNING:toil.leader:R/G/jobTxtDR7 -r \
STDERR: 2019-10-03T06:31:56.588701759Z WARNING:toil.leader:R/G/jobTxtDR7 results.json \
STDERR: 2019-10-03T06:31:56.588708689Z WARNING:toil.leader:R/G/jobTxtDR7 -c \
STDERR: 2019-10-03T06:31:56.588842690Z WARNING:toil.leader:R/G/jobTxtDR7 /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpzw78W9/stge2b6abb3-ecf5-4af6-b97b-ea34631bc4da/.synapseConfig
STDERR: 2019-10-03T06:31:56.588853800Z WARNING:toil.leader:R/G/jobTxtDR7 INFO:cwltool:[job get_submission_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN/b/8/out_tmpdirCh_EU6$ python \
STDERR: 2019-10-03T06:31:56.588951490Z WARNING:toil.leader:R/G/jobTxtDR7 get_submission_docker.py \
STDERR: 2019-10-03T06:31:56.588957780Z WARNING:toil.leader:R/G/jobTxtDR7 -s \
STDERR: 2019-10-03T06:31:56.589064261Z WARNING:toil.leader:R/G/jobTxtDR7 9693558 \
STDERR: 2019-10-03T06:31:56.589074221Z WARNING:toil.leader:R/G/jobTxtDR7 -r \
STDERR: 2019-10-03T06:31:56.589168530Z WARNING:toil.leader:R/G/jobTxtDR7 results.json \
STDERR: 2019-10-03T06:31:56.589174811Z WARNING:toil.leader:R/G/jobTxtDR7 -c \
STDERR: 2019-10-03T06:31:56.589287951Z WARNING:toil.leader:R/G/jobTxtDR7 /var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpzw78W9/stge2b6abb3-ecf5-4af6-b97b-ea34631bc4da/.synapseConfig
STDERR: 2019-10-03T06:31:56.589298241Z WARNING:toil.leader:R/G/jobTxtDR7 Welcome, ehrdreamservice!
STDERR: 2019-10-03T06:31:56.589389891Z WARNING:toil.leader:R/G/jobTxtDR7
STDERR: 2019-10-03T06:31:56.589396282Z WARNING:toil.leader:R/G/jobTxtDR7 Traceback (most recent call last):
STDERR: 2019-10-03T06:31:56.589510142Z WARNING:toil.leader:R/G/jobTxtDR7 File "get_submission_docker.py", line 15, in
STDERR: 2019-10-03T06:31:56.589520962Z WARNING:toil.leader:R/G/jobTxtDR7 raise Exception('Expected DockerRepository type but found '+sub.entity.concreteType)
STDERR: 2019-10-03T06:31:56.589638162Z WARNING:toil.leader:R/G/jobTxtDR7 Exception: Expected DockerRepository type but found org.sagebionetworks.repo.model.Project
STDERR: 2019-10-03T06:31:56.589644723Z WARNING:toil.leader:R/G/jobTxtDR7 [job get_submission_docker.cwl] Max memory used: 50MiB
STDERR: 2019-10-03T06:31:56.589740102Z WARNING:toil.leader:R/G/jobTxtDR7 INFO:cwltool:[job get_submission_docker.cwl] Max memory used: 50MiB
STDERR: 2019-10-03T06:31:56.589750862Z WARNING:toil.leader:R/G/jobTxtDR7 [job get_submission_docker.cwl] Job error:
STDERR: 2019-10-03T06:31:56.589845023Z WARNING:toil.leader:R/G/jobTxtDR7 Error collecting output for parameter 'docker_repository':
STDERR: 2019-10-03T06:31:56.589851383Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: Expression evaluation error:
STDERR: 2019-10-03T06:31:56.589963643Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: No JSON object could be decoded
STDERR: 2019-10-03T06:31:56.589973913Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: script was:
STDERR: 2019-10-03T06:31:56.590070733Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 01 "use strict";
STDERR: 2019-10-03T06:31:56.590077384Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 02 var inputs = {
STDERR: 2019-10-03T06:31:56.590182064Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 03 "synapse_config": {
STDERR: 2019-10-03T06:31:56.590192424Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 04 "basename": ".synapseConfig",
STDERR: 2019-10-03T06:31:56.590287425Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 05 "nameroot": ".synapseConfig",
STDERR: 2019-10-03T06:31:56.590294515Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 06 "nameext": "",
STDERR: 2019-10-03T06:31:56.590400414Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 07 "location": "toilfs:F/S/tmpi7Ak6H-x-.synapseConfig",
STDERR: 2019-10-03T06:31:56.590413605Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 08 "path": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpzw78W9/stge2b6abb3-ecf5-4af6-b97b-ea34631bc4da/.synapseConfig",
STDERR: 2019-10-03T06:31:56.590524145Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 09 "dirname": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpzw78W9/stge2b6abb3-ecf5-4af6-b97b-ea34631bc4da",
STDERR: 2019-10-03T06:31:56.590531135Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 10 "class": "File",
STDERR: 2019-10-03T06:31:56.590638846Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 11 "size": 64
STDERR: 2019-10-03T06:31:56.590648806Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 12 },
STDERR: 2019-10-03T06:31:56.590742066Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 13 "submissionid": 9693558
STDERR: 2019-10-03T06:31:56.590748636Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 14 };
STDERR: 2019-10-03T06:31:56.590860826Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 15 var self = [];
STDERR: 2019-10-03T06:31:56.590871416Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 16 var runtime = {
STDERR: 2019-10-03T06:31:56.590980396Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 17 "outdirSize": 1024,
STDERR: 2019-10-03T06:31:56.590987627Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 18 "ram": 100,
STDERR: 2019-10-03T06:31:56.591095617Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 19 "cores": 1,
STDERR: 2019-10-03T06:31:56.591105737Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 20 "tmpdirSize": 1024,
STDERR: 2019-10-03T06:31:56.591210728Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 21 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmp",
STDERR: 2019-10-03T06:31:56.591218318Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 22 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN/b/8/out_tmpdirCh_EU6"
STDERR: 2019-10-03T06:31:56.591313428Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 23 };
STDERR: 2019-10-03T06:31:56.591343028Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 24 (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-03T06:31:56.591432158Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: stdout was: ''
STDERR: 2019-10-03T06:31:56.591462448Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: stderr was: 'evalmachine.:24
STDERR: 2019-10-03T06:31:56.591581448Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-03T06:31:56.591592278Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: ^
STDERR: 2019-10-03T06:31:56.591682749Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1:
STDERR: 2019-10-03T06:31:56.591689360Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: TypeError: Cannot read property 'contents' of undefined
STDERR: 2019-10-03T06:31:56.591772199Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at evalmachine.:24:41
STDERR: 2019-10-03T06:31:56.591818730Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at evalmachine.:24:76
STDERR: 2019-10-03T06:31:56.591907420Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Script.runInContext (vm.js:107:20)
STDERR: 2019-10-03T06:31:56.591917420Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Script.runInNewContext (vm.js:113:17)
STDERR: 2019-10-03T06:31:56.592003230Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Object.runInNewContext (vm.js:296:38)
STDERR: 2019-10-03T06:31:56.592009561Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Socket. ([eval]:11:57)
STDERR: 2019-10-03T06:31:56.592118141Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Socket.emit (events.js:189:13)
STDERR: 2019-10-03T06:31:56.592128001Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at addChunk (_stream_readable.js:284:12)
STDERR: 2019-10-03T06:31:56.592210231Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at readableAddChunk (_stream_readable.js:261:13)
STDERR: 2019-10-03T06:31:56.592226641Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Socket.Readable.push (_stream_readable.js:220:10)'
STDERR: 2019-10-03T06:31:56.592329842Z WARNING:toil.leader:R/G/jobTxtDR7 ERROR:cwltool:[job get_submission_docker.cwl] Job error:
STDERR: 2019-10-03T06:31:56.592336091Z WARNING:toil.leader:R/G/jobTxtDR7 Error collecting output for parameter 'docker_repository':
STDERR: 2019-10-03T06:31:56.592420702Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: Expression evaluation error:
STDERR: 2019-10-03T06:31:56.592467202Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: No JSON object could be decoded
STDERR: 2019-10-03T06:31:56.592551002Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: script was:
STDERR: 2019-10-03T06:31:56.592556772Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 01 "use strict";
STDERR: 2019-10-03T06:31:56.592640033Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 02 var inputs = {
STDERR: 2019-10-03T06:31:56.592686682Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 03 "synapse_config": {
STDERR: 2019-10-03T06:31:56.592775793Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 04 "basename": ".synapseConfig",
STDERR: 2019-10-03T06:31:56.592786633Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 05 "nameroot": ".synapseConfig",
STDERR: 2019-10-03T06:31:56.592878473Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 06 "nameext": "",
STDERR: 2019-10-03T06:31:56.592885353Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 07 "location": "toilfs:F/S/tmpi7Ak6H-x-.synapseConfig",
STDERR: 2019-10-03T06:31:56.592995363Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 08 "path": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpzw78W9/stge2b6abb3-ecf5-4af6-b97b-ea34631bc4da/.synapseConfig",
STDERR: 2019-10-03T06:31:56.593006793Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 09 "dirname": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpzw78W9/stge2b6abb3-ecf5-4af6-b97b-ea34631bc4da",
STDERR: 2019-10-03T06:31:56.593101694Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 10 "class": "File",
STDERR: 2019-10-03T06:31:56.593108394Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 11 "size": 64
STDERR: 2019-10-03T06:31:56.593207104Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 12 },
STDERR: 2019-10-03T06:31:56.593217654Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 13 "submissionid": 9693558
STDERR: 2019-10-03T06:31:56.593311225Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 14 };
STDERR: 2019-10-03T06:31:56.593317505Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 15 var self = [];
STDERR: 2019-10-03T06:31:56.593422005Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 16 var runtime = {
STDERR: 2019-10-03T06:31:56.593431875Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 17 "outdirSize": 1024,
STDERR: 2019-10-03T06:31:56.593524405Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 18 "ram": 100,
STDERR: 2019-10-03T06:31:56.593530866Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 19 "cores": 1,
STDERR: 2019-10-03T06:31:56.593623685Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 20 "tmpdirSize": 1024,
STDERR: 2019-10-03T06:31:56.593634725Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 21 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmp",
STDERR: 2019-10-03T06:31:56.593742396Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 22 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/94082fd9-f0a7-40e2-9e48-cf3ccdfe2c64/tmpRQWSZN/b/8/out_tmpdirCh_EU6"
STDERR: 2019-10-03T06:31:56.593749616Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 23 };
STDERR: 2019-10-03T06:31:56.593856126Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: 24 (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-03T06:31:56.593866336Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: stdout was: ''
STDERR: 2019-10-03T06:31:56.593958857Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: stderr was: 'evalmachine.:24
STDERR: 2019-10-03T06:31:56.593965496Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-03T06:31:56.594069567Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: ^
STDERR: 2019-10-03T06:31:56.594086807Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1:
STDERR: 2019-10-03T06:31:56.594171588Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: TypeError: Cannot read property 'contents' of undefined
STDERR: 2019-10-03T06:31:56.594178597Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at evalmachine.:24:41
STDERR: 2019-10-03T06:31:56.594286067Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at evalmachine.:24:76
STDERR: 2019-10-03T06:31:56.594296928Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Script.runInContext (vm.js:107:20)
STDERR: 2019-10-03T06:31:56.594388238Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Script.runInNewContext (vm.js:113:17)
STDERR: 2019-10-03T06:31:56.594394678Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Object.runInNewContext (vm.js:296:38)
STDERR: 2019-10-03T06:31:56.594497548Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Socket. ([eval]:11:57)
STDERR: 2019-10-03T06:31:56.594508108Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Socket.emit (events.js:189:13)
STDERR: 2019-10-03T06:31:56.594601239Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at addChunk (_stream_readable.js:284:12)
STDERR: 2019-10-03T06:31:56.594607889Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at readableAddChunk (_stream_readable.js:261:13)
STDERR: 2019-10-03T06:31:56.594712069Z WARNING:toil.leader:R/G/jobTxtDR7 :1:1: at Socket.Readable.push (_stream_readable.js:220:10)'
STDERR: 2019-10-03T06:31:56.594722730Z WARNING:toil.leader:R/G/jobTxtDR7 [job get_submission_docker.cwl] completed permanentFail
STDERR: 2019-10-03T06:31:56.594814240Z WARNING:toil.leader:R/G/jobTxtDR7 WARNING:cwltool:[job get_submission_docker.cwl] completed permanentFail
STDERR: 2019-10-03T06:31:56.594820610Z WARNING:toil.leader:R/G/jobTxtDR7 Traceback (most recent call last):
STDERR: 2019-10-03T06:31:56.594926600Z WARNING:toil.leader:R/G/jobTxtDR7 File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 331, in workerScript
STDERR: 2019-10-03T06:31:56.594937280Z WARNING:toil.leader:R/G/jobTxtDR7 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
STDERR: 2019-10-03T06:31:56.595033441Z WARNING:toil.leader:R/G/jobTxtDR7 File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1378, in _runner
STDERR: 2019-10-03T06:31:56.595040100Z WARNING:toil.leader:R/G/jobTxtDR7 returnValues = self._run(jobGraph, fileStore)
STDERR: 2019-10-03T06:31:56.595144681Z WARNING:toil.leader:R/G/jobTxtDR7 File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1323, in _run
STDERR: 2019-10-03T06:31:56.595154770Z WARNING:toil.leader:R/G/jobTxtDR7 return self.run(fileStore)
STDERR: 2019-10-03T06:31:56.595250811Z WARNING:toil.leader:R/G/jobTxtDR7 File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 606, in run
STDERR: 2019-10-03T06:31:56.595257711Z WARNING:toil.leader:R/G/jobTxtDR7 raise cwltool.errors.WorkflowException(status)
STDERR: 2019-10-03T06:31:56.595364262Z WARNING:toil.leader:R/G/jobTxtDR7 WorkflowException: permanentFail
STDERR: 2019-10-03T06:31:56.595374012Z WARNING:toil.leader:R/G/jobTxtDR7 ERROR:toil.worker:Exiting the worker because of a failed job on host 353d597a895f
STDERR: 2019-10-03T06:31:56.595467441Z WARNING:toil.leader:R/G/jobTxtDR7 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/G/jobTxtDR7 with ID R/G/jobTxtDR7 to 0
STDERR: 2019-10-03T06:31:56.597586449Z WARNING:toil.leader:Job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/G/jobTxtDR7 with ID R/G/jobTxtDR7 is completely failed
STDERR: 2019-10-03T06:31:57.012796083Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python l/j/jobCXOviK
STDERR: 2019-10-03T06:32:02.209118523Z INFO:toil.leader:Finished toil run with 4 failed jobs.
STDERR: 2019-10-03T06:32:02.209144663Z INFO:toil.leader:Failed jobs at end of the run: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/G/jobTxtDR7 'CWLWorkflow' Z/5/jobLSjIfM 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python W/k/job_UU0Nq 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python L/4/jobicfjX2
STDERR: 2019-10-03T06:32:02.281230427Z Traceback (most recent call last):
STDERR: 2019-10-03T06:32:02.281257718Z File "/usr/local/bin/toil-cwl-runner", line 10, in
STDERR: 2019-10-03T06:32:02.281264918Z sys.exit(main())
STDERR: 2019-10-03T06:32:02.281270507Z File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1276, in main
STDERR: 2019-10-03T06:32:02.281434728Z outobj = toil.start(wf1)
STDERR: 2019-10-03T06:32:02.281448158Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 781, in start
STDERR: 2019-10-03T06:32:02.281652258Z return self._runMainLoop(rootJobGraph)
STDERR: 2019-10-03T06:32:02.281663509Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 1054, in _runMainLoop
STDERR: 2019-10-03T06:32:02.281922639Z jobCache=self._jobCache).run()
STDERR: 2019-10-03T06:32:02.281930240Z File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 246, in run
STDERR: 2019-10-03T06:32:02.281991490Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
STDERR: 2019-10-03T06:32:02.282005550Z toil.leader.FailedJobsException
Created by Ivan Brugere ivanbrugere mine also runs to completion too, @ajg. I still don't understand how projects were being submitted rather than docker images, but the "Submit Docker Repository to Challenge" from project docker pages seems the correct path now. Just for those of you that might be interested (e.g. @ivanbrugere):
The submission now suddenly works (without I have changed anything). When submitting to the challenge I do not get the error message anymore. However, I don't know yet whether the container will run through without errors (fingers crossed)! :-) HI! Yes, I also don't get it running...
I believe I have followed the video in all details and used the GitHub code. I managed to run the code locally and push it onto my synapse account.
However, when I press on the Submit to the challenge button under point "7. Submit your Docker Image" I can only see my projects. Submitting the project results in the error message we have posted above.
Searching for the Docker synapse id I was able to load it and submit also the Docker, but it resulted in a "Docker Digest cannot be null" message when pressing "submit".
Going to the project docker page and then pressing "Submit to challenge" results in the previously described "It is currently outside of the time range allowed for submissions." message.
So I ran out of ideas now (assuming that the code provided on GitHub is ok to submit).
Many thanks for any suggestions!
Best wishes
that is my suspicion as well, from the line:
"STDERR: 2019-10-03T06:31:56.589638162Z WARNING:toil.leader:R/G/jobTxtDR7 Exception: Expected DockerRepository type but found org.sagebionetworks.repo.model.Project"
but I followed the instructions as shown on the webinar to submit the repository and did this exactly. I'm not sure what I'm missing. I guess we have also submitted the project instead of the container... However, I cannot test if this is true as I get currently the error message "It is currently outside of the time range allowed for submissions." when trying to submit the docker container. Same here: I have uploaded the Github example, which is nicely running locally, but gives me the following error message, when submitting it:
STDERR: 2019-10-02T13:11:59.867726448Z INFO:cwltool:Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/EHR-challenge-develop/synthetic_docker_agent_workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/EHR-challenge-develop/synthetic_docker_agent_workflow.cwl'
STDERR: 2019-10-02T13:12:01.991200982Z EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-02T13:12:01.991217312Z WARNING:salad:EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-02T13:12:01.992880599Z EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-02T13:12:01.992895370Z WARNING:salad:EHR-challenge-develop/synthetic_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/synthetic_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-02T13:12:06.118327680Z EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-02T13:12:06.118360330Z WARNING:salad:EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-02T13:12:06.356717267Z EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-02T13:12:06.356751927Z WARNING:salad:EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-02T13:12:07.370542626Z WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (32).
STDERR: 2019-10-02T13:12:07.370576056Z WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (268107968512).
STDERR: 2019-10-02T13:12:07.370684826Z WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (359380717568).
STDERR: 2019-10-02T13:12:07.587573915Z INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-02T13:12:08.023944016Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg'.
STDERR: 2019-10-02T13:12:08.025091612Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/toil-95843318-3a76-4136-b382-4f6d17829bef-e89ed93837262f1d6b20cd9e97e84170/tmpoo17ZF/worker_log.txt
STDERR: 2019-10-02T13:12:08.819578459Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python a/t/job6e4RHs with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-02T13:12:08.827910837Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python b/D/jobK4V8wq with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-02T13:12:08.829214633Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python f/V/jobw2R1RB with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-02T13:12:08.829793835Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python A/G/job0OlGFq with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-02T13:12:09.290411466Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg'.
STDERR: 2019-10-02T13:12:09.291306029Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/toil-95843318-3a76-4136-b382-4f6d17829bef-e89ed93837262f1d6b20cd9e97e84170/tmpx1GXic/worker_log.txt
STDERR: 2019-10-02T13:12:09.295486308Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg'.
STDERR: 2019-10-02T13:12:09.296337712Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/toil-95843318-3a76-4136-b382-4f6d17829bef-e89ed93837262f1d6b20cd9e97e84170/tmp_7lHHl/worker_log.txt
STDERR: 2019-10-02T13:12:09.312281394Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg'.
STDERR: 2019-10-02T13:12:09.313143128Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/toil-95843318-3a76-4136-b382-4f6d17829bef-e89ed93837262f1d6b20cd9e97e84170/tmpCzGPB1/worker_log.txt
STDERR: 2019-10-02T13:12:09.315240837Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg'.
STDERR: 2019-10-02T13:12:09.316254831Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/toil-95843318-3a76-4136-b382-4f6d17829bef-e89ed93837262f1d6b20cd9e97e84170/tmplRfcJ2/worker_log.txt
STDERR: 2019-10-02T13:12:11.046276002Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python a/t/job6e4RHs
STDERR: 2019-10-02T13:12:11.763060747Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python b/D/jobK4V8wq
STDERR: 2019-10-02T13:12:11.763301398Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python f/V/jobw2R1RB
STDERR: 2019-10-02T13:12:11.978543740Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python A/G/job0OlGFq
STDERR: 2019-10-02T13:12:11.979141393Z WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python A/G/job0OlGFq
STDERR: 2019-10-02T13:12:11.979311343Z WARNING:toil.leader:A/G/job0OlGFq INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2019-10-02T13:12:11.979346783Z WARNING:toil.leader:A/G/job0OlGFq INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-02T13:12:11.979482674Z WARNING:toil.leader:A/G/job0OlGFq [job get_submission_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg/f/2/out_tmpdirDwYfym$ python \
STDERR: 2019-10-02T13:12:11.979525984Z WARNING:toil.leader:A/G/job0OlGFq get_submission_docker.py \
STDERR: 2019-10-02T13:12:11.979653065Z WARNING:toil.leader:A/G/job0OlGFq -s \
STDERR: 2019-10-02T13:12:11.979721595Z WARNING:toil.leader:A/G/job0OlGFq 9693551 \
STDERR: 2019-10-02T13:12:11.979841256Z WARNING:toil.leader:A/G/job0OlGFq -r \
STDERR: 2019-10-02T13:12:11.979896196Z WARNING:toil.leader:A/G/job0OlGFq results.json \
STDERR: 2019-10-02T13:12:11.979993837Z WARNING:toil.leader:A/G/job0OlGFq -c \
STDERR: 2019-10-02T13:12:11.980082857Z WARNING:toil.leader:A/G/job0OlGFq /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmplpHNSu/stgf3b3307c-b276-4a14-b88d-1f94c7df1bcb/.synapseConfig
STDERR: 2019-10-02T13:12:11.980183487Z WARNING:toil.leader:A/G/job0OlGFq INFO:cwltool:[job get_submission_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg/f/2/out_tmpdirDwYfym$ python \
STDERR: 2019-10-02T13:12:11.980256148Z WARNING:toil.leader:A/G/job0OlGFq get_submission_docker.py \
STDERR: 2019-10-02T13:12:11.980336958Z WARNING:toil.leader:A/G/job0OlGFq -s \
STDERR: 2019-10-02T13:12:11.980417698Z WARNING:toil.leader:A/G/job0OlGFq 9693551 \
STDERR: 2019-10-02T13:12:11.980496789Z WARNING:toil.leader:A/G/job0OlGFq -r \
STDERR: 2019-10-02T13:12:11.980585809Z WARNING:toil.leader:A/G/job0OlGFq results.json \
STDERR: 2019-10-02T13:12:11.980653799Z WARNING:toil.leader:A/G/job0OlGFq -c \
STDERR: 2019-10-02T13:12:11.980761970Z WARNING:toil.leader:A/G/job0OlGFq /var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmplpHNSu/stgf3b3307c-b276-4a14-b88d-1f94c7df1bcb/.synapseConfig
STDERR: 2019-10-02T13:12:11.980829320Z WARNING:toil.leader:A/G/job0OlGFq Welcome, ehrdreamservice!
STDERR: 2019-10-02T13:12:11.980910810Z WARNING:toil.leader:A/G/job0OlGFq
STDERR: 2019-10-02T13:12:11.980989150Z WARNING:toil.leader:A/G/job0OlGFq Traceback (most recent call last):
STDERR: 2019-10-02T13:12:11.981077642Z WARNING:toil.leader:A/G/job0OlGFq File "get_submission_docker.py", line 15, in
STDERR: 2019-10-02T13:12:11.981156352Z WARNING:toil.leader:A/G/job0OlGFq raise Exception('Expected DockerRepository type but found '+sub.entity.concreteType)
STDERR: 2019-10-02T13:12:11.981245592Z WARNING:toil.leader:A/G/job0OlGFq Exception: Expected DockerRepository type but found org.sagebionetworks.repo.model.Project
STDERR: 2019-10-02T13:12:11.981324472Z WARNING:toil.leader:A/G/job0OlGFq Could not collect memory usage, job ended before monitoring began.
STDERR: 2019-10-02T13:12:11.981405902Z WARNING:toil.leader:A/G/job0OlGFq INFO:cwltool:Could not collect memory usage, job ended before monitoring began.
STDERR: 2019-10-02T13:12:11.981503303Z WARNING:toil.leader:A/G/job0OlGFq [job get_submission_docker.cwl] Job error:
STDERR: 2019-10-02T13:12:11.981585743Z WARNING:toil.leader:A/G/job0OlGFq Error collecting output for parameter 'docker_repository':
STDERR: 2019-10-02T13:12:11.981667743Z WARNING:toil.leader:A/G/job0OlGFq :1:1: Expression evaluation error:
STDERR: 2019-10-02T13:12:11.981764985Z WARNING:toil.leader:A/G/job0OlGFq :1:1: No JSON object could be decoded
STDERR: 2019-10-02T13:12:11.981834675Z WARNING:toil.leader:A/G/job0OlGFq :1:1: script was:
STDERR: 2019-10-02T13:12:11.981918045Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 01 "use strict";
STDERR: 2019-10-02T13:12:11.981996915Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 02 var inputs = {
STDERR: 2019-10-02T13:12:11.982077956Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 03 "synapse_config": {
STDERR: 2019-10-02T13:12:11.982161835Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 04 "basename": ".synapseConfig",
STDERR: 2019-10-02T13:12:11.982240577Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 05 "nameroot": ".synapseConfig",
STDERR: 2019-10-02T13:12:11.982318717Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 06 "nameext": "",
STDERR: 2019-10-02T13:12:11.982429667Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 07 "location": "toilfs:C/J/tmpGQRaYT-x-.synapseConfig",
STDERR: 2019-10-02T13:12:11.982501838Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 08 "path": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmplpHNSu/stgf3b3307c-b276-4a14-b88d-1f94c7df1bcb/.synapseConfig",
STDERR: 2019-10-02T13:12:11.982589488Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 09 "dirname": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmplpHNSu/stgf3b3307c-b276-4a14-b88d-1f94c7df1bcb",
STDERR: 2019-10-02T13:12:11.982670578Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 10 "class": "File",
STDERR: 2019-10-02T13:12:11.982748339Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 11 "size": 64
STDERR: 2019-10-02T13:12:11.982829799Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 12 },
STDERR: 2019-10-02T13:12:11.982910090Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 13 "submissionid": 9693551
STDERR: 2019-10-02T13:12:11.982986690Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 14 };
STDERR: 2019-10-02T13:12:11.983072130Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 15 var self = [];
STDERR: 2019-10-02T13:12:11.983145400Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 16 var runtime = {
STDERR: 2019-10-02T13:12:11.983243601Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 17 "outdirSize": 1024,
STDERR: 2019-10-02T13:12:11.983306522Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 18 "ram": 100,
STDERR: 2019-10-02T13:12:11.983390621Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 19 "cores": 1,
STDERR: 2019-10-02T13:12:11.983474982Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 20 "tmpdirSize": 1024,
STDERR: 2019-10-02T13:12:11.983599973Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 21 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmp",
STDERR: 2019-10-02T13:12:11.983684213Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 22 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg/f/2/out_tmpdirDwYfym"
STDERR: 2019-10-02T13:12:11.983760403Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 23 };
STDERR: 2019-10-02T13:12:11.983836343Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 24 (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-02T13:12:11.983921664Z WARNING:toil.leader:A/G/job0OlGFq :1:1: stdout was: ''
STDERR: 2019-10-02T13:12:11.983998474Z WARNING:toil.leader:A/G/job0OlGFq :1:1: stderr was: 'evalmachine.:24
STDERR: 2019-10-02T13:12:11.984079214Z WARNING:toil.leader:A/G/job0OlGFq :1:1: (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-02T13:12:11.984157415Z WARNING:toil.leader:A/G/job0OlGFq :1:1: ^
STDERR: 2019-10-02T13:12:11.984236616Z WARNING:toil.leader:A/G/job0OlGFq :1:1:
STDERR: 2019-10-02T13:12:11.984319286Z WARNING:toil.leader:A/G/job0OlGFq :1:1: TypeError: Cannot read property 'contents' of undefined
STDERR: 2019-10-02T13:12:11.984398726Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at evalmachine.:24:41
STDERR: 2019-10-02T13:12:11.984494766Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at evalmachine.:24:76
STDERR: 2019-10-02T13:12:11.984554716Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Script.runInContext (vm.js:107:20)
STDERR: 2019-10-02T13:12:11.984643188Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Script.runInNewContext (vm.js:113:17)
STDERR: 2019-10-02T13:12:11.984708117Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Object.runInNewContext (vm.js:296:38)
STDERR: 2019-10-02T13:12:11.984801948Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Socket. ([eval]:11:57)
STDERR: 2019-10-02T13:12:11.984861848Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Socket.emit (events.js:189:13)
STDERR: 2019-10-02T13:12:11.984957958Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at addChunk (_stream_readable.js:284:12)
STDERR: 2019-10-02T13:12:11.985019328Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at readableAddChunk (_stream_readable.js:261:13)
STDERR: 2019-10-02T13:12:11.985118530Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Socket.Readable.push (_stream_readable.js:220:10)'
STDERR: 2019-10-02T13:12:11.985181639Z WARNING:toil.leader:A/G/job0OlGFq ERROR:cwltool:[job get_submission_docker.cwl] Job error:
STDERR: 2019-10-02T13:12:11.985276680Z WARNING:toil.leader:A/G/job0OlGFq Error collecting output for parameter 'docker_repository':
STDERR: 2019-10-02T13:12:11.985337770Z WARNING:toil.leader:A/G/job0OlGFq :1:1: Expression evaluation error:
STDERR: 2019-10-02T13:12:11.985435370Z WARNING:toil.leader:A/G/job0OlGFq :1:1: No JSON object could be decoded
STDERR: 2019-10-02T13:12:11.985495120Z WARNING:toil.leader:A/G/job0OlGFq :1:1: script was:
STDERR: 2019-10-02T13:12:11.985591371Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 01 "use strict";
STDERR: 2019-10-02T13:12:11.985653502Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 02 var inputs = {
STDERR: 2019-10-02T13:12:11.985749982Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 03 "synapse_config": {
STDERR: 2019-10-02T13:12:11.985809572Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 04 "basename": ".synapseConfig",
STDERR: 2019-10-02T13:12:11.985905563Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 05 "nameroot": ".synapseConfig",
STDERR: 2019-10-02T13:12:11.985965243Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 06 "nameext": "",
STDERR: 2019-10-02T13:12:11.986063793Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 07 "location": "toilfs:C/J/tmpGQRaYT-x-.synapseConfig",
STDERR: 2019-10-02T13:12:11.986129814Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 08 "path": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmplpHNSu/stgf3b3307c-b276-4a14-b88d-1f94c7df1bcb/.synapseConfig",
STDERR: 2019-10-02T13:12:11.986211324Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 09 "dirname": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmplpHNSu/stgf3b3307c-b276-4a14-b88d-1f94c7df1bcb",
STDERR: 2019-10-02T13:12:11.986288774Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 10 "class": "File",
STDERR: 2019-10-02T13:12:11.986383465Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 11 "size": 64
STDERR: 2019-10-02T13:12:11.986444465Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 12 },
STDERR: 2019-10-02T13:12:11.986540395Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 13 "submissionid": 9693551
STDERR: 2019-10-02T13:12:11.986600425Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 14 };
STDERR: 2019-10-02T13:12:11.986696406Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 15 var self = [];
STDERR: 2019-10-02T13:12:11.986754186Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 16 var runtime = {
STDERR: 2019-10-02T13:12:11.986849687Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 17 "outdirSize": 1024,
STDERR: 2019-10-02T13:12:11.986909317Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 18 "ram": 100,
STDERR: 2019-10-02T13:12:11.987004577Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 19 "cores": 1,
STDERR: 2019-10-02T13:12:11.987064218Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 20 "tmpdirSize": 1024,
STDERR: 2019-10-02T13:12:11.987166828Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 21 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmp",
STDERR: 2019-10-02T13:12:11.987230799Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 22 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/0b1cbb3b-2a4c-47c6-a4ac-39c4cb9d3010/tmpK9HEYg/f/2/out_tmpdirDwYfym"
STDERR: 2019-10-02T13:12:11.987318449Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 23 };
STDERR: 2019-10-02T13:12:11.987392479Z WARNING:toil.leader:A/G/job0OlGFq :1:1: 24 (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-02T13:12:11.987486250Z WARNING:toil.leader:A/G/job0OlGFq :1:1: stdout was: ''
STDERR: 2019-10-02T13:12:11.987577290Z WARNING:toil.leader:A/G/job0OlGFq :1:1: stderr was: 'evalmachine.:24
STDERR: 2019-10-02T13:12:11.987661751Z WARNING:toil.leader:A/G/job0OlGFq :1:1: (function(){return ((JSON.parse(self[0].contents)['docker_repository']));})()
STDERR: 2019-10-02T13:12:11.987755991Z WARNING:toil.leader:A/G/job0OlGFq :1:1: ^
STDERR: 2019-10-02T13:12:11.987831301Z WARNING:toil.leader:A/G/job0OlGFq :1:1:
STDERR: 2019-10-02T13:12:11.987925432Z WARNING:toil.leader:A/G/job0OlGFq :1:1: TypeError: Cannot read property 'contents' of undefined
STDERR: 2019-10-02T13:12:11.987995972Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at evalmachine.:24:41
STDERR: 2019-10-02T13:12:11.988075593Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at evalmachine.:24:76
STDERR: 2019-10-02T13:12:11.988155323Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Script.runInContext (vm.js:107:20)
STDERR: 2019-10-02T13:12:11.988246413Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Script.runInNewContext (vm.js:113:17)
STDERR: 2019-10-02T13:12:11.988312343Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Object.runInNewContext (vm.js:296:38)
STDERR: 2019-10-02T13:12:11.988399804Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Socket. ([eval]:11:57)
STDERR: 2019-10-02T13:12:11.988468324Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Socket.emit (events.js:189:13)
STDERR: 2019-10-02T13:12:11.988554745Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at addChunk (_stream_readable.js:284:12)
STDERR: 2019-10-02T13:12:11.988622865Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at readableAddChunk (_stream_readable.js:261:13)
STDERR: 2019-10-02T13:12:11.988716626Z WARNING:toil.leader:A/G/job0OlGFq :1:1: at Socket.Readable.push (_stream_readable.js:220:10)'
STDERR: 2019-10-02T13:12:11.988777116Z WARNING:toil.leader:A/G/job0OlGFq [job get_submission_docker.cwl] completed permanentFail
STDERR: 2019-10-02T13:12:11.988875086Z WARNING:toil.leader:A/G/job0OlGFq WARNING:cwltool:[job get_submission_docker.cwl] completed permanentFail
STDERR: 2019-10-02T13:12:11.988935886Z WARNING:toil.leader:A/G/job0OlGFq Traceback (most recent call last):
STDERR: 2019-10-02T13:12:11.989032366Z WARNING:toil.leader:A/G/job0OlGFq File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 331, in workerScript
STDERR: 2019-10-02T13:12:11.989095487Z WARNING:toil.leader:A/G/job0OlGFq job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
STDERR: 2019-10-02T13:12:11.989193128Z WARNING:toil.leader:A/G/job0OlGFq File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1378, in _runner
STDERR: 2019-10-02T13:12:11.989253398Z WARNING:toil.leader:A/G/job0OlGFq returnValues = self._run(jobGraph, fileStore)
STDERR: 2019-10-02T13:12:11.989349118Z WARNING:toil.leader:A/G/job0OlGFq File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1323, in _run
STDERR: 2019-10-02T13:12:11.989411339Z WARNING:toil.leader:A/G/job0OlGFq return self.run(fileStore)
STDERR: 2019-10-02T13:12:11.989508480Z WARNING:toil.leader:A/G/job0OlGFq File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 606, in run
STDERR: 2019-10-02T13:12:11.989566970Z WARNING:toil.leader:A/G/job0OlGFq raise cwltool.errors.WorkflowException(status)
STDERR: 2019-10-02T13:12:11.989664420Z WARNING:toil.leader:A/G/job0OlGFq WorkflowException: permanentFail
STDERR: 2019-10-02T13:12:11.989723700Z WARNING:toil.leader:A/G/job0OlGFq ERROR:toil.worker:Exiting the worker because of a failed job on host a1006dff0902
STDERR: 2019-10-02T13:12:11.989835030Z WARNING:toil.leader:A/G/job0OlGFq WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python A/G/job0OlGFq with ID A/G/job0OlGFq to 0
STDERR: 2019-10-02T13:12:11.992187321Z WARNING:toil.leader:Job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python A/G/job0OlGFq with ID A/G/job0OlGFq is completely failed
STDERR: 2019-10-02T13:12:19.658932064Z INFO:toil.leader:Finished toil run with 4 failed jobs.
STDERR: 2019-10-02T13:12:19.659048474Z INFO:toil.leader:Failed jobs at end of the run: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python f/V/jobw2R1RB 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python A/G/job0OlGFq 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python a/t/job6e4RHs 'CWLWorkflow' Y/9/job7HC1VE
STDERR: 2019-10-02T13:12:19.733980353Z Traceback (most recent call last):
STDERR: 2019-10-02T13:12:19.734012242Z File "/usr/local/bin/toil-cwl-runner", line 10, in
STDERR: 2019-10-02T13:12:19.734018003Z sys.exit(main())
STDERR: 2019-10-02T13:12:19.734022612Z File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1276, in main
STDERR: 2019-10-02T13:12:19.734166813Z outobj = toil.start(wf1)
STDERR: 2019-10-02T13:12:19.734181733Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 781, in start
STDERR: 2019-10-02T13:12:19.734336403Z return self._runMainLoop(rootJobGraph)
STDERR: 2019-10-02T13:12:19.734345864Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 1054, in _runMainLoop
STDERR: 2019-10-02T13:12:19.734565945Z jobCache=self._jobCache).run()
STDERR: 2019-10-02T13:12:19.734575795Z File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 246, in run
STDERR: 2019-10-02T13:12:19.734580805Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
STDERR: 2019-10-02T13:12:19.734632645Z toil.leader.FailedJobsException