I submitted my image which completes under Fast Lane. In the challenge submission it fails with the following log. Any suggestion greatly appreciated.
I think the point of interest is:
STDERR: 2019-10-11T09:02:16.847606193Z WARNING:toil.leader:O/e/jobgAbzCO raise SynapseHTTPError(message, response=response)
STDERR: 2019-10-11T09:02:16.847712163Z WARNING:toil.leader:O/e/jobgAbzCO synapseclient.exceptions.SynapseHTTPError: 412 Client Error:
FULL:
STDERR: 2019-10-11T08:56:49.589826204Z INFO:cwltool:Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/main_docker_agent_workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/main_docker_agent_workflow.cwl'
STDERR: 2019-10-11T08:56:52.800387270Z EHR-challenge-develop/main_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-11T08:56:52.800440230Z WARNING:salad:EHR-challenge-develop/main_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_infer/status` previously defined
STDERR: 2019-10-11T08:56:52.802392148Z EHR-challenge-develop/main_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-11T08:56:52.802411678Z WARNING:salad:EHR-challenge-develop/main_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_train/status` previously defined
STDERR: 2019-10-11T08:56:59.621498847Z EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-11T08:56:59.621561207Z WARNING:salad:EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined
STDERR: 2019-10-11T08:57:00.041943295Z EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-11T08:57:00.042004175Z WARNING:salad:EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined
STDERR: 2019-10-11T08:57:02.255005598Z WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (32).
STDERR: 2019-10-11T08:57:02.255192520Z WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (268107968512).
STDERR: 2019-10-11T08:57:02.255621361Z WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (327929716736).
STDERR: 2019-10-11T08:57:02.851613973Z INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-11T08:57:03.728129747Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:03.729754794Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpHiy6fw/worker_log.txt
STDERR: 2019-10-11T08:57:05.280798494Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T08:57:05.280965925Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python m/p/jobcphbSj with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T08:57:05.304734338Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/t/jobUZXriU with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T08:57:05.307992350Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python O/X/jobIXelLz with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T08:57:06.100644318Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:06.105372645Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpCtQdF5/worker_log.txt
STDERR: 2019-10-11T08:57:06.139646619Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:06.143353544Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpyaTeoM/worker_log.txt
STDERR: 2019-10-11T08:57:06.163606213Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:06.165244169Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmp55a8YL/worker_log.txt
STDERR: 2019-10-11T08:57:06.218745927Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:06.220020242Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpsXChqm/worker_log.txt
STDERR: 2019-10-11T08:57:09.035478393Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw
STDERR: 2019-10-11T08:57:09.978791094Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python O/X/jobIXelLz
STDERR: 2019-10-11T08:57:10.465191488Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python m/p/jobcphbSj
STDERR: 2019-10-11T08:57:10.794114027Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/t/jobUZXriU
STDERR: 2019-10-11T08:57:10.794893300Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python C/h/job3b7SVR with job batch system ID: 5 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T08:57:11.647407708Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:11.648778954Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmppiG6Z4/worker_log.txt
STDERR: 2019-10-11T08:57:16.251194536Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python C/h/job3b7SVR
STDERR: 2019-10-11T08:57:16.252474871Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python u/e/job8rtC9_ with job batch system ID: 6 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T08:57:16.263066802Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_training_docker.cwl' python C/y/job4RYdld with job batch system ID: 7 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T08:57:16.938142648Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:16.939199872Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpf8PP8t/worker_log.txt
STDERR: 2019-10-11T08:57:16.955715166Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T08:57:16.957930995Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpLHEfdm/worker_log.txt
STDERR: 2019-10-11T08:57:20.905365029Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python u/e/job8rtC9_
STDERR: 2019-10-11T09:00:10.348889645Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_training_docker.cwl' python C/y/job4RYdld
STDERR: 2019-10-11T09:00:10.349860799Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_infer_docker.cwl' python E/S/jobt6lgEA with job batch system ID: 8 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:00:10.921143657Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:00:10.922415522Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpqkJB5C/worker_log.txt
STDERR: 2019-10-11T09:02:05.428061251Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_infer_docker.cwl' python E/S/jobt6lgEA
STDERR: 2019-10-11T09:02:05.429394346Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 with job batch system ID: 9 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:05.445570156Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/upload_to_synapse.cwl' python j/I/jobrBWwTu with job batch system ID: 10 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:06.270719811Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:06.271744145Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpvVBxV8/worker_log.txt
STDERR: 2019-10-11T09:02:06.290250183Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:06.293056924Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmphpsbhB/worker_log.txt
STDERR: 2019-10-11T09:02:11.447692443Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4
STDERR: 2019-10-11T09:02:11.477289143Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM with job batch system ID: 11 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:11.478363996Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_email.cwl' python f/S/jobW0HJQ7 with job batch system ID: 12 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:11.507584144Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/upload_to_synapse.cwl' python j/I/jobrBWwTu
STDERR: 2019-10-11T09:02:11.509113570Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with job batch system ID: 13 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:12.337178115Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:12.344123990Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpLQF_O_/worker_log.txt
STDERR: 2019-10-11T09:02:12.348998809Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:12.350455384Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmptDNiEy/worker_log.txt
STDERR: 2019-10-11T09:02:12.358912475Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:12.360240640Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpq3MQCx/worker_log.txt
STDERR: 2019-10-11T09:02:16.828447582Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM
STDERR: 2019-10-11T09:02:16.829365606Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/submit_to_challenge.cwl' python D/t/jobBT4ymB with job batch system ID: 14 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:16.829549317Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO
STDERR: 2019-10-11T09:02:16.830026348Z WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO
STDERR: 2019-10-11T09:02:16.830163739Z WARNING:toil.leader:O/e/jobgAbzCO INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2019-10-11T09:02:16.830207919Z WARNING:toil.leader:O/e/jobgAbzCO INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
STDERR: 2019-10-11T09:02:16.830301040Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh/6/d/out_tmpdiryWBX6M$ python \
STDERR: 2019-10-11T09:02:16.830344569Z WARNING:toil.leader:O/e/jobgAbzCO annotationSubmission.py \
STDERR: 2019-10-11T09:02:16.830435920Z WARNING:toil.leader:O/e/jobgAbzCO -s \
STDERR: 2019-10-11T09:02:16.830476440Z WARNING:toil.leader:O/e/jobgAbzCO 9693830 \
STDERR: 2019-10-11T09:02:16.830547170Z WARNING:toil.leader:O/e/jobgAbzCO -v \
STDERR: 2019-10-11T09:02:16.830617760Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stge96a5b5a-ccd6-4874-a04c-596d3cc904b8/results.json \
STDERR: 2019-10-11T09:02:16.830673861Z WARNING:toil.leader:O/e/jobgAbzCO -p \
STDERR: 2019-10-11T09:02:16.830745362Z WARNING:toil.leader:O/e/jobgAbzCO true \
STDERR: 2019-10-11T09:02:16.830786461Z WARNING:toil.leader:O/e/jobgAbzCO -f \
STDERR: 2019-10-11T09:02:16.830879381Z WARNING:toil.leader:O/e/jobgAbzCO true \
STDERR: 2019-10-11T09:02:16.830920302Z WARNING:toil.leader:O/e/jobgAbzCO -c \
STDERR: 2019-10-11T09:02:16.831010312Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stgde9a4a84-33ab-4fcf-bb37-30329cbf99ce/.synapseConfig
STDERR: 2019-10-11T09:02:16.831100572Z WARNING:toil.leader:O/e/jobgAbzCO INFO:cwltool:[job annotate_submission.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh/6/d/out_tmpdiryWBX6M$ python \
STDERR: 2019-10-11T09:02:16.831129622Z WARNING:toil.leader:O/e/jobgAbzCO annotationSubmission.py \
STDERR: 2019-10-11T09:02:16.831175503Z WARNING:toil.leader:O/e/jobgAbzCO -s \
STDERR: 2019-10-11T09:02:16.831216123Z WARNING:toil.leader:O/e/jobgAbzCO 9693830 \
STDERR: 2019-10-11T09:02:16.831317864Z WARNING:toil.leader:O/e/jobgAbzCO -v \
STDERR: 2019-10-11T09:02:16.831435234Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stge96a5b5a-ccd6-4874-a04c-596d3cc904b8/results.json \
STDERR: 2019-10-11T09:02:16.831614864Z WARNING:toil.leader:O/e/jobgAbzCO -p \
STDERR: 2019-10-11T09:02:16.844984853Z WARNING:toil.leader:O/e/jobgAbzCO true \
STDERR: 2019-10-11T09:02:16.845105693Z WARNING:toil.leader:O/e/jobgAbzCO -f \
STDERR: 2019-10-11T09:02:16.845118504Z WARNING:toil.leader:O/e/jobgAbzCO true \
STDERR: 2019-10-11T09:02:16.845124484Z WARNING:toil.leader:O/e/jobgAbzCO -c \
STDERR: 2019-10-11T09:02:16.845275505Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stgde9a4a84-33ab-4fcf-bb37-30329cbf99ce/.synapseConfig
STDERR: 2019-10-11T09:02:16.845394325Z WARNING:toil.leader:O/e/jobgAbzCO Welcome, ehrdreamservice!
STDERR: 2019-10-11T09:02:16.845421235Z WARNING:toil.leader:O/e/jobgAbzCO
STDERR: 2019-10-11T09:02:16.845575766Z WARNING:toil.leader:O/e/jobgAbzCO Traceback (most recent call last):
STDERR: 2019-10-11T09:02:16.845639045Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 75, in
STDERR: 2019-10-11T09:02:16.845808286Z WARNING:toil.leader:O/e/jobgAbzCO _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10)
STDERR: 2019-10-11T09:02:16.845894567Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 105, in _with_retry
STDERR: 2019-10-11T09:02:16.845998278Z WARNING:toil.leader:O/e/jobgAbzCO six.reraise(exc_info[0], exc_info[1], exc_info[2])
STDERR: 2019-10-11T09:02:16.846190547Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 53, in _with_retry
STDERR: 2019-10-11T09:02:16.846350139Z WARNING:toil.leader:O/e/jobgAbzCO response = function()
STDERR: 2019-10-11T09:02:16.846442638Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 75, in
STDERR: 2019-10-11T09:02:16.846616679Z WARNING:toil.leader:O/e/jobgAbzCO _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10)
STDERR: 2019-10-11T09:02:16.846798920Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 61, in annotate_submission
STDERR: 2019-10-11T09:02:16.846976460Z WARNING:toil.leader:O/e/jobgAbzCO status = syn.store(status)
STDERR: 2019-10-11T09:02:16.847180662Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 912, in store
STDERR: 2019-10-11T09:02:16.847238912Z WARNING:toil.leader:O/e/jobgAbzCO return type(obj)(**self.restPUT(obj.putURI(), obj.json()))
STDERR: 2019-10-11T09:02:16.847326752Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 3368, in restPUT
STDERR: 2019-10-11T09:02:16.847408072Z WARNING:toil.leader:O/e/jobgAbzCO exceptions._raise_for_status(response, verbose=self.debug)
STDERR: 2019-10-11T09:02:16.847547082Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/exceptions.py", line 153, in _raise_for_status
STDERR: 2019-10-11T09:02:16.847606193Z WARNING:toil.leader:O/e/jobgAbzCO raise SynapseHTTPError(message, response=response)
STDERR: 2019-10-11T09:02:16.847712163Z WARNING:toil.leader:O/e/jobgAbzCO synapseclient.exceptions.SynapseHTTPError: 412 Client Error:
STDERR: 2019-10-11T09:02:16.847793554Z WARNING:toil.leader:O/e/jobgAbzCO Submission Status: 9693830 was updated since you last fetched it, retrieve it again and reapply the update
STDERR: 2019-10-11T09:02:16.847895994Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] Max memory used: 45MiB
STDERR: 2019-10-11T09:02:16.847978974Z WARNING:toil.leader:O/e/jobgAbzCO INFO:cwltool:[job annotate_submission.cwl] Max memory used: 45MiB
STDERR: 2019-10-11T09:02:16.848032944Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] completed permanentFail
STDERR: 2019-10-11T09:02:16.848116075Z WARNING:toil.leader:O/e/jobgAbzCO WARNING:cwltool:[job annotate_submission.cwl] completed permanentFail
STDERR: 2019-10-11T09:02:16.848181275Z WARNING:toil.leader:O/e/jobgAbzCO Traceback (most recent call last):
STDERR: 2019-10-11T09:02:16.848264375Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 331, in workerScript
STDERR: 2019-10-11T09:02:16.848323155Z WARNING:toil.leader:O/e/jobgAbzCO job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
STDERR: 2019-10-11T09:02:16.848424796Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1378, in _runner
STDERR: 2019-10-11T09:02:16.848521136Z WARNING:toil.leader:O/e/jobgAbzCO returnValues = self._run(jobGraph, fileStore)
STDERR: 2019-10-11T09:02:16.848617087Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1323, in _run
STDERR: 2019-10-11T09:02:16.848681536Z WARNING:toil.leader:O/e/jobgAbzCO return self.run(fileStore)
STDERR: 2019-10-11T09:02:16.848783537Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 606, in run
STDERR: 2019-10-11T09:02:16.848835347Z WARNING:toil.leader:O/e/jobgAbzCO raise cwltool.errors.WorkflowException(status)
STDERR: 2019-10-11T09:02:16.848921158Z WARNING:toil.leader:O/e/jobgAbzCO WorkflowException: permanentFail
STDERR: 2019-10-11T09:02:16.848977148Z WARNING:toil.leader:O/e/jobgAbzCO ERROR:toil.worker:Exiting the worker because of a failed job on host fb0a4e18f2b6
STDERR: 2019-10-11T09:02:16.849137638Z WARNING:toil.leader:O/e/jobgAbzCO WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with ID O/e/jobgAbzCO to 0
STDERR: 2019-10-11T09:02:16.849974372Z WARNING:toil.leader:Job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with ID O/e/jobgAbzCO is completely failed
STDERR: 2019-10-11T09:02:17.536308186Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_email.cwl' python f/S/jobW0HJQ7
STDERR: 2019-10-11T09:02:17.649865633Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:17.651042818Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpsUaR_v/worker_log.txt
STDERR: 2019-10-11T09:02:21.260331349Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/submit_to_challenge.cwl' python D/t/jobBT4ymB
STDERR: 2019-10-11T09:02:21.260517391Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM with job batch system ID: 15 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:21.947581207Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:21.948908681Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpg46MYY/worker_log.txt
STDERR: 2019-10-11T09:02:22.039889007Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM
STDERR: 2019-10-11T09:02:22.042349566Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 with job batch system ID: 16 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:22.666455150Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:22.667485404Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpHfx4e1/worker_log.txt
STDERR: 2019-10-11T09:02:22.770922494Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4
STDERR: 2019-10-11T09:02:22.773802695Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw with job batch system ID: 17 and cores: 1, disk: 2.0 G, and memory: 100.0 M
STDERR: 2019-10-11T09:02:23.389965881Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'.
STDERR: 2019-10-11T09:02:23.391183415Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmp9Ei3aR/worker_log.txt
STDERR: 2019-10-11T09:02:23.499023062Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw
STDERR: 2019-10-11T09:02:33.595430779Z INFO:toil.leader:Finished toil run with 8 failed jobs.
STDERR: 2019-10-11T09:02:33.596009471Z INFO:toil.leader:Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_training_docker.cwl' python C/y/job4RYdld 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/t/jobUZXriU 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python m/p/jobcphbSj 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/upload_to_synapse.cwl' python j/I/jobrBWwTu 'CWLWorkflow' 8/U/jobniYcvf 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_infer_docker.cwl' python E/S/jobt6lgEA 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python C/h/job3b7SVR 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO
STDERR: 2019-10-11T09:02:33.683849084Z Traceback (most recent call last):
STDERR: 2019-10-11T09:02:33.683877244Z File "/usr/local/bin/toil-cwl-runner", line 10, in
STDERR: 2019-10-11T09:02:33.683882544Z sys.exit(main())
STDERR: 2019-10-11T09:02:33.683886614Z File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1276, in main
STDERR: 2019-10-11T09:02:33.684134945Z outobj = toil.start(wf1)
STDERR: 2019-10-11T09:02:33.684176775Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 781, in start
STDERR: 2019-10-11T09:02:33.684298716Z return self._runMainLoop(rootJobGraph)
STDERR: 2019-10-11T09:02:33.684308816Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 1054, in _runMainLoop
STDERR: 2019-10-11T09:02:33.684594787Z jobCache=self._jobCache).run()
STDERR: 2019-10-11T09:02:33.684607187Z File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 246, in run
STDERR: 2019-10-11T09:02:33.684611617Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
STDERR: 2019-10-11T09:02:33.684632247Z toil.leader.FailedJobsException
Created by Ivan Brugere ivanbrugere Hi @trberg , I noticed that this model was re-launched this morning (not by me). I suspected you or someone else were debugging this issue.
I re-ran on my own to confirm. Now this runs to completion on the "Challenge Submission" channel. Was there a change on your end?
Thanks,
Ivan Hi @ivanbrugere,
The Error on our end was:
Traceback (most recent call last):
File "/app/train.py", line 12, in
data_train, labels_train, labels_train_expand = model_includes.read_ehrdc_data(config["train path"])
File "/app/includes/models.py", line 35, in read_ehrdc_data
data[k] = pd.read_csv(f, usecols=useful_keys[k])
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f
return _read(filepath_or_buffer, kwds)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 457, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in __init__
self._make_engine(self.engine)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1135, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1959, in __init__
_validate_usecols_names(usecols, self.orig_names)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1255, in _validate_usecols_names
"columns expected but not found: {missing}".format(missing=missing)
ValueError: Usecols do not match columns, columns expected but not found: ['time_of_birth']
This looks like an oversight on my part. I'll make the adjustment to the synthetic data to remove the 'time_of_birth' column. In the meantime, you can go ahead and ignore that column as it's not in our data.
Thank you,
Tim
Drop files to upload
"EHR DREAM Challenge Submission" channel failed page is loading…