Hi,
I confirmed our docker image ran successfully and I initially got prediction probability depending on participants.
I submitted the model and received confirmation and submission accepted email. However, I also received workflow failed email which shows errors in urllib3 (scoket.timeout: timed out) - which I don't understand since inference.py doesn't make any outside request?
Error message is :
```
DERR: 2023-02-13T22:40:53.324138006Z Traceback (most recent call last):
STDERR: 2023-02-13T22:40:53.324141006Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
STDERR: 2023-02-13T22:40:53.324144389Z six.raise_from(e, None)
STDERR: 2023-02-13T22:40:53.324147329Z File "", line 3, in raise_from
STDERR: 2023-02-13T22:40:53.324150518Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
STDERR: 2023-02-13T22:40:53.324153616Z httplib_response = conn.getresponse()
STDERR: 2023-02-13T22:40:53.324156674Z File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
STDERR: 2023-02-13T22:40:53.324159847Z response.begin()
STDERR: 2023-02-13T22:40:53.324162698Z File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
STDERR: 2023-02-13T22:40:53.324165850Z version, status, reason = self._read_status()
STDERR: 2023-02-13T22:40:53.324168657Z File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
STDERR: 2023-02-13T22:40:53.324171886Z line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
STDERR: 2023-02-13T22:40:53.324175087Z File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
STDERR: 2023-02-13T22:40:53.324178158Z return self._sock.recv_into(b)
STDERR: 2023-02-13T22:40:53.324180937Z socket.timeout: timed out
STDERR: 2023-02-13T22:40:53.324183722Z
```
Could you please check this error?
Thank you in advance.
Created by Jouhyun Jeon Clare @Clare @Sakb Your submissions are successful and accepted now. Please continue with your writeup submission. @vjbytes102 I ran the docker image on the randomly selected 36 audio files (from a training set) and there was no problem. Also, running the image using the full dataset worked perfectly. Plus, current model that I submitted today is little bit different from the previous model that I submitted during the leaderboard so I think there's no specific reason to receive an error message on my end...
I wondering if you can help me to diagnose the error message? It would be great if you can run the container in non-detached mode so that we can get better output on what the error is. Since I am not importing urllib, I am wondering the reason to receive urllib-related timeout error message.
@Clare we did not detect any error on our end, to make sure i re-run your code with correct memory configurations but still got the failure. It appears to me that the issue is on your end with the docker image Thank you for your help @vjbytes102
However, I received a same error. I received both submission accepted email and workflow fail email two times.
It is weird. I found out that max memory is different between 1st and 2nd log files.
my error message (2nd attempt) is:
```
STDERR: 2023-02-13T23:27:33.593140332Z 915a75a9bc13 2023-02-13 23:27:33,592 MainThread INFO cwltool: Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/workflow.cwl'
STDERR: 2023-02-13T23:27:45.647881726Z 915a75a9bc13 2023-02-13 23:27:45,647 MainThread WARNING cwltool: Workflow checker warning:
STDERR: 2023-02-13T23:27:45.647937467Z TB-DREAM-Challenge-Infra-main/workflow.cwl:72:9: Source 'docker_repository' of type ["null",
STDERR: 2023-02-13T23:27:45.647944522Z "string"] may be incompatible
STDERR: 2023-02-13T23:27:45.647948276Z TB-DREAM-Challenge-Infra-main/workflow.cwl:172:9: with sink 'docker_repository' of type "string"
STDERR: 2023-02-13T23:27:45.647951530Z TB-DREAM-Challenge-Infra-main/workflow.cwl:73:9: Source 'docker_digest' of type ["null", "string"]
STDERR: 2023-02-13T23:27:45.647955152Z may be incompatible
STDERR: 2023-02-13T23:27:45.647958277Z TB-DREAM-Challenge-Infra-main/workflow.cwl:174:9: with sink 'docker_digest' of type "string"
STDERR: 2023-02-13T23:27:48.000667087Z 915a75a9bc13 2023-02-13 23:27:48,000 MainThread INFO toil: Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2023-02-13T23:27:50.104790570Z 915a75a9bc13 2023-02-13 23:27:50,104 MainThread INFO toil.leader: 1 jobs are running, 0 jobs are issued and waiting to run
STDERR: 2023-02-13T23:27:51.505723627Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp2ppqbogh/worker_log.txt
STDERR: 2023-02-13T23:27:53.264660239Z 915a75a9bc13 2023-02-13 23:27:53,264 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-3ysg8m4a with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:27:53.264861293Z 915a75a9bc13 2023-02-13 23:27:53,264 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-fvph_4q0 with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:27:53.265049855Z 915a75a9bc13 2023-02-13 23:27:53,264 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9 with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:27:53.265234946Z 915a75a9bc13 2023-02-13 23:27:53,265 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn with job batch system ID: 4 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:27:53.265418815Z 915a75a9bc13 2023-02-13 23:27:53,265 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:27:58.484758700Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp1ud8wy2h/worker_log.txt
STDERR: 2023-02-13T23:27:58.910826648Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpt7mfzwcb/worker_log.txt
STDERR: 2023-02-13T23:28:06.969228691Z 915a75a9bc13 2023-02-13 23:28:06,968 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-fvph_4q0
STDERR: 2023-02-13T23:28:07.239510004Z 915a75a9bc13 2023-02-13 23:28:07,239 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-3ysg8m4a
STDERR: 2023-02-13T23:28:12.689017618Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpyr31ewre/worker_log.txt
STDERR: 2023-02-13T23:28:12.944952148Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp7c5gyese/worker_log.txt
STDERR: 2023-02-13T23:28:21.435802250Z 915a75a9bc13 2023-02-13 23:28:21,432 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l
STDERR: 2023-02-13T23:28:24.220466974Z 915a75a9bc13 2023-02-13 23:28:24,220 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9
STDERR: 2023-02-13T23:28:24.229296796Z 915a75a9bc13 2023-02-13 23:28:24,229 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:28:29.879042652Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpddcd1hq7/worker_log.txt
STDERR: 2023-02-13T23:28:31.784624194Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp7fky1eda/worker_log.txt
STDERR: 2023-02-13T23:28:33.413694162Z 915a75a9bc13 2023-02-13 23:28:33,413 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em
STDERR: 2023-02-13T23:28:33.414736228Z 915a75a9bc13 2023-02-13 23:28:33,414 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:28:36.803842509Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp7bdid1ab/worker_log.txt
STDERR: 2023-02-13T23:28:39.736862657Z 915a75a9bc13 2023-02-13 23:28:39,736 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr
STDERR: 2023-02-13T23:28:39.738108664Z 915a75a9bc13 2023-02-13 23:28:39,737 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881 with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:28:41.243442548Z 915a75a9bc13 2023-02-13 23:28:41,230 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn
STDERR: 2023-02-13T23:28:41.243481043Z 915a75a9bc13 2023-02-13 23:28:41,231 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with job batch system ID: 9 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:28:41.243486335Z 915a75a9bc13 2023-02-13 23:28:41,231 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8 with job batch system ID: 10 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:28:41.243501846Z 915a75a9bc13 2023-02-13 23:28:41,231 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929 with job batch system ID: 11 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:28:47.746881392Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp92labq61/worker_log.txt
STDERR: 2023-02-13T23:28:49.027313044Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/worker_log.txt
STDERR: 2023-02-13T23:28:55.224779277Z 915a75a9bc13 2023-02-13 23:28:55,224 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881
STDERR: 2023-02-13T23:28:59.754063219Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpmhsryigz/worker_log.txt
STDERR: 2023-02-13T23:29:10.390312437Z 915a75a9bc13 2023-02-13 23:29:10,389 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929
STDERR: 2023-02-13T23:29:13.217146098Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpgjhszgr5/worker_log.txt
STDERR: 2023-02-13T23:29:21.892246475Z 915a75a9bc13 2023-02-13 23:29:21,891 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8
STDERR: 2023-02-13T23:29:21.893103228Z 915a75a9bc13 2023-02-13 23:29:21,892 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-u9ppmdb8 with job batch system ID: 12 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:29:26.275481164Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpf2e38hok/worker_log.txt
STDERR: 2023-02-13T23:29:27.778853667Z 915a75a9bc13 2023-02-13 23:29:27,778 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-u9ppmdb8
STDERR: 2023-02-13T23:29:27.779638816Z 915a75a9bc13 2023-02-13 23:29:27,779 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8 with job batch system ID: 13 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-02-13T23:29:30.495458436Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpver2jaik/worker_log.txt
STDERR: 2023-02-13T23:29:31.111741723Z 915a75a9bc13 2023-02-13 23:29:31,111 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8
STDERR: 2023-02-13T23:40:46.470141401Z 915a75a9bc13 2023-02-13 23:40:46,468 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys
STDERR: 2023-02-13T23:40:46.471211077Z 915a75a9bc13 2023-02-13 23:40:46,470 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys
STDERR: 2023-02-13T23:40:46.471219851Z 915a75a9bc13 2023-02-13 23:40:46,470 MainThread WARNING toil.leader: Log from job kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys follows:
STDERR: 2023-02-13T23:40:46.471237092Z =========>
STDERR: 2023-02-13T23:40:46.471240740Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2023-02-13T23:40:46.471244182Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2023-02-13T23:40:46.471247156Z [job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \
STDERR: 2023-02-13T23:40:46.471250595Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \
STDERR: 2023-02-13T23:40:46.471254024Z -s \
STDERR: 2023-02-13T23:40:46.471256894Z 9732155 \
STDERR: 2023-02-13T23:40:46.471259732Z -p \
STDERR: 2023-02-13T23:40:46.471262460Z docker.synapse.org/syn45594136/docker-final-submission \
STDERR: 2023-02-13T23:40:46.471274654Z -d \
STDERR: 2023-02-13T23:40:46.471277584Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \
STDERR: 2023-02-13T23:40:46.471280489Z --store \
STDERR: 2023-02-13T23:40:46.471283221Z --status \
STDERR: 2023-02-13T23:40:46.471286701Z VALIDATED \
STDERR: 2023-02-13T23:40:46.471289350Z --parentid \
STDERR: 2023-02-13T23:40:46.471291997Z syn51089588 \
STDERR: 2023-02-13T23:40:46.471294508Z -c \
STDERR: 2023-02-13T23:40:46.471297375Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \
STDERR: 2023-02-13T23:40:46.471313161Z -t \
STDERR: 2023-02-13T23:40:46.471315819Z 2
STDERR: 2023-02-13T23:40:46.471319447Z INFO:cwltool:[job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \
STDERR: 2023-02-13T23:40:46.471322983Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \
STDERR: 2023-02-13T23:40:46.471326341Z -s \
STDERR: 2023-02-13T23:40:46.471329067Z 9732155 \
STDERR: 2023-02-13T23:40:46.471332045Z -p \
STDERR: 2023-02-13T23:40:46.471334656Z docker.synapse.org/syn45594136/docker-final-submission \
STDERR: 2023-02-13T23:40:46.471337670Z -d \
STDERR: 2023-02-13T23:40:46.471340246Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \
STDERR: 2023-02-13T23:40:46.471343121Z --store \
STDERR: 2023-02-13T23:40:46.471345667Z --status \
STDERR: 2023-02-13T23:40:46.471348217Z VALIDATED \
STDERR: 2023-02-13T23:40:46.471350624Z --parentid \
STDERR: 2023-02-13T23:40:46.471353617Z syn51089588 \
STDERR: 2023-02-13T23:40:46.471356373Z -c \
STDERR: 2023-02-13T23:40:46.471358777Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \
STDERR: 2023-02-13T23:40:46.471361972Z -t \
STDERR: 2023-02-13T23:40:46.471364417Z 2
STDERR: 2023-02-13T23:40:46.471366852Z Welcome, Vijay Yadav!
STDERR: 2023-02-13T23:40:46.471370057Z
STDERR: 2023-02-13T23:40:46.471372613Z
STDERR: 2023-02-13T23:40:46.471387291Z UPGRADE AVAILABLE
STDERR: 2023-02-13T23:40:46.471390211Z
STDERR: 2023-02-13T23:40:46.471392965Z A more recent version of the Synapse Client (2.7.0) is available. Your version (2.5.1) can be upgraded by typing:
STDERR: 2023-02-13T23:40:46.471399956Z pip install --upgrade synapseclient
STDERR: 2023-02-13T23:40:46.471402825Z
STDERR: 2023-02-13T23:40:46.471405559Z Python Synapse Client version 2.7.0 release notes
STDERR: 2023-02-13T23:40:46.471408017Z
STDERR: 2023-02-13T23:40:46.471410622Z https://python-docs.synapse.org/build/html/news.html
STDERR: 2023-02-13T23:40:46.471413377Z
STDERR: 2023-02-13T23:40:46.471416044Z root
STDERR: 2023-02-13T23:40:46.471418744Z mounting volumes
STDERR: 2023-02-13T23:40:46.471421507Z checking for containers
STDERR: 2023-02-13T23:40:46.471424128Z running container
STDERR: 2023-02-13T23:40:46.471426659Z creating logfile
STDERR: 2023-02-13T23:40:46.471429197Z Traceback (most recent call last):
STDERR: 2023-02-13T23:40:46.471431783Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
STDERR: 2023-02-13T23:40:46.471434623Z six.raise_from(e, None)
STDERR: 2023-02-13T23:40:46.471437233Z File "", line 3, in raise_from
STDERR: 2023-02-13T23:40:46.471440165Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
STDERR: 2023-02-13T23:40:46.471443265Z httplib_response = conn.getresponse()
STDERR: 2023-02-13T23:40:46.471446225Z File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
STDERR: 2023-02-13T23:40:46.471449481Z response.begin()
STDERR: 2023-02-13T23:40:46.471464566Z File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
STDERR: 2023-02-13T23:40:46.471467637Z version, status, reason = self._read_status()
STDERR: 2023-02-13T23:40:46.471470365Z File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
STDERR: 2023-02-13T23:40:46.471475809Z line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
STDERR: 2023-02-13T23:40:46.471478618Z File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
STDERR: 2023-02-13T23:40:46.471481987Z return self._sock.recv_into(b)
STDERR: 2023-02-13T23:40:46.471484753Z socket.timeout: timed out
STDERR: 2023-02-13T23:40:46.471487343Z
STDERR: 2023-02-13T23:40:46.471490243Z During handling of the above exception, another exception occurred:
STDERR: 2023-02-13T23:40:46.471493321Z
STDERR: 2023-02-13T23:40:46.471495813Z Traceback (most recent call last):
STDERR: 2023-02-13T23:40:46.471499110Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
STDERR: 2023-02-13T23:40:46.471502181Z resp = conn.urlopen(
STDERR: 2023-02-13T23:40:46.471505195Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
STDERR: 2023-02-13T23:40:46.471507959Z retries = retries.increment(
STDERR: 2023-02-13T23:40:46.471510895Z File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
STDERR: 2023-02-13T23:40:46.471517884Z raise six.reraise(type(error), error, _stacktrace)
STDERR: 2023-02-13T23:40:46.471520724Z File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
STDERR: 2023-02-13T23:40:46.471523784Z raise value
STDERR: 2023-02-13T23:40:46.471526609Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
STDERR: 2023-02-13T23:40:46.471541769Z httplib_response = self._make_request(
STDERR: 2023-02-13T23:40:46.471544933Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 451, in _make_request
STDERR: 2023-02-13T23:40:46.471547757Z self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
STDERR: 2023-02-13T23:40:46.471550660Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
STDERR: 2023-02-13T23:40:46.471553541Z raise ReadTimeoutError(
STDERR: 2023-02-13T23:40:46.471556124Z urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
STDERR: 2023-02-13T23:40:46.471559003Z
STDERR: 2023-02-13T23:40:46.471561781Z During handling of the above exception, another exception occurred:
STDERR: 2023-02-13T23:40:46.471564713Z
STDERR: 2023-02-13T23:40:46.471567250Z Traceback (most recent call last):
STDERR: 2023-02-13T23:40:46.471569981Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 252, in
STDERR: 2023-02-13T23:40:46.471573485Z main(syn, args)
STDERR: 2023-02-13T23:40:46.471576361Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 193, in main
STDERR: 2023-02-13T23:40:46.471579921Z while container in client.containers.list(ignore_removed=True):
STDERR: 2023-02-13T23:40:46.471583991Z File "/usr/local/lib/python3.8/site-packages/docker/models/containers.py", line 952, in list
STDERR: 2023-02-13T23:40:46.471587070Z resp = self.client.api.containers(all=all, before=before,
STDERR: 2023-02-13T23:40:46.471590261Z File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 209, in containers
STDERR: 2023-02-13T23:40:46.471593185Z res = self._result(self._get(u, params=params), True)
STDERR: 2023-02-13T23:40:46.471596302Z File "/usr/local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
STDERR: 2023-02-13T23:40:46.471599837Z return f(self, *args, **kwargs)
STDERR: 2023-02-13T23:40:46.471602522Z File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
STDERR: 2023-02-13T23:40:46.471620317Z return self.get(url, **self._set_request_timeout(kwargs))
STDERR: 2023-02-13T23:40:46.471623558Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in get
STDERR: 2023-02-13T23:40:46.471626564Z return self.request('GET', url, **kwargs)
STDERR: 2023-02-13T23:40:46.471629117Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
STDERR: 2023-02-13T23:40:46.471632230Z resp = self.send(prep, **send_kwargs)
STDERR: 2023-02-13T23:40:46.471635193Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
STDERR: 2023-02-13T23:40:46.471637978Z r = adapter.send(request, **kwargs)
STDERR: 2023-02-13T23:40:46.471640801Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
STDERR: 2023-02-13T23:40:46.471643793Z raise ReadTimeout(e, request=request)
STDERR: 2023-02-13T23:40:46.471646557Z requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
STDERR: 2023-02-13T23:40:46.471649668Z [job run_docker.cwl] Max memory used: 19MiB
STDERR: 2023-02-13T23:40:46.471652147Z INFO:cwltool:[job run_docker.cwl] Max memory used: 19MiB
STDERR: 2023-02-13T23:40:46.471654805Z [job run_docker.cwl] Job error:
STDERR: 2023-02-13T23:40:46.471657533Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {})
STDERR: 2023-02-13T23:40:46.471660590Z ERROR:cwltool:[job run_docker.cwl] Job error:
STDERR: 2023-02-13T23:40:46.471663346Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {})
STDERR: 2023-02-13T23:40:46.471667044Z [job run_docker.cwl] completed permanentFail
STDERR: 2023-02-13T23:40:46.471669769Z WARNING:cwltool:[job run_docker.cwl] completed permanentFail
STDERR: 2023-02-13T23:40:46.471672443Z Traceback (most recent call last):
STDERR: 2023-02-13T23:40:46.471675409Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2023-02-13T23:40:46.471679339Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2023-02-13T23:40:46.471682542Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2023-02-13T23:40:46.471698377Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2023-02-13T23:40:46.471701584Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run
STDERR: 2023-02-13T23:40:46.471704700Z return self.run(fileStore)
STDERR: 2023-02-13T23:40:46.471707653Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2023-02-13T23:40:46.471710440Z raise cwltool.errors.WorkflowException(status)
STDERR: 2023-02-13T23:40:46.471713277Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2023-02-13T23:40:46.471719190Z ERROR:toil.worker:Exiting the worker because of a failed job on host 915a75a9bc13
STDERR: 2023-02-13T23:40:46.471722452Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys to 0
STDERR: 2023-02-13T23:40:46.471726407Z <=========
STDERR: 2023-02-13T23:40:46.489520979Z 915a75a9bc13 2023-02-13 23:40:46,489 MainThread WARNING toil.leader: Job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys is completely failed
STDERR: 2023-02-13T23:40:59.984032393Z 915a75a9bc13 2023-02-13 23:40:59,983 MainThread INFO toil.leader: Finished toil run with 9 failed jobs.
STDERR: 2023-02-13T23:40:59.984348979Z 915a75a9bc13 2023-02-13 23:40:59,984 MainThread INFO toil.leader: Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l 'CWLWorkflow' kind-CWLWorkflow/instance-fd9vo75k 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn
STDERR: 2023-02-13T23:41:00.040493963Z Traceback (most recent call last):
STDERR: 2023-02-13T23:41:00.040523148Z File "/usr/local/bin/toil-cwl-runner", line 8, in
STDERR: 2023-02-13T23:41:00.040527842Z sys.exit(main())
STDERR: 2023-02-13T23:41:00.040531118Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 1691, in main
STDERR: 2023-02-13T23:41:00.040534543Z outobj = toil.start(wf1)
STDERR: 2023-02-13T23:41:00.040538513Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 829, in start
STDERR: 2023-02-13T23:41:00.040542046Z return self._runMainLoop(rootJobGraph)
STDERR: 2023-02-13T23:41:00.040545161Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 1115, in _runMainLoop
STDERR: 2023-02-13T23:41:00.040548646Z return Leader(config=self.config,
STDERR: 2023-02-13T23:41:00.040551955Z File "/usr/local/lib/python3.8/site-packages/toil/leader.py", line 269, in run
STDERR: 2023-02-13T23:41:00.040555242Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
STDERR: 2023-02-13T23:41:00.040755837Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmp9z6ejoye' contains 9 failed jobs: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l, 'CWLWorkflow' kind-CWLWorkflow/instance-fd9vo75k, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn
STDERR: 2023-02-13T23:41:00.040782150Z Log from job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys follows:
STDERR: 2023-02-13T23:41:00.040787110Z =========>
STDERR: 2023-02-13T23:41:00.040790540Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2023-02-13T23:41:00.040793579Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2023-02-13T23:41:00.040796734Z [job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \
STDERR: 2023-02-13T23:41:00.040800412Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \
STDERR: 2023-02-13T23:41:00.040804743Z -s \
STDERR: 2023-02-13T23:41:00.040807648Z 9732155 \
STDERR: 2023-02-13T23:41:00.040811047Z -p \
STDERR: 2023-02-13T23:41:00.040814054Z docker.synapse.org/syn45594136/docker-final-submission \
STDERR: 2023-02-13T23:41:00.040817177Z -d \
STDERR: 2023-02-13T23:41:00.040820027Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \
STDERR: 2023-02-13T23:41:00.040823076Z --store \
STDERR: 2023-02-13T23:41:00.040825861Z --status \
STDERR: 2023-02-13T23:41:00.040828500Z VALIDATED \
STDERR: 2023-02-13T23:41:00.040831431Z --parentid \
STDERR: 2023-02-13T23:41:00.040834271Z syn51089588 \
STDERR: 2023-02-13T23:41:00.040837244Z -c \
STDERR: 2023-02-13T23:41:00.040840503Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \
STDERR: 2023-02-13T23:41:00.040844139Z -t \
STDERR: 2023-02-13T23:41:00.040847063Z 2
STDERR: 2023-02-13T23:41:00.040904755Z INFO:cwltool:[job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \
STDERR: 2023-02-13T23:41:00.040910145Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \
STDERR: 2023-02-13T23:41:00.040914005Z -s \
STDERR: 2023-02-13T23:41:00.040922531Z 9732155 \
STDERR: 2023-02-13T23:41:00.040925870Z -p \
STDERR: 2023-02-13T23:41:00.040938033Z docker.synapse.org/syn45594136/docker-final-submission \
STDERR: 2023-02-13T23:41:00.040941120Z -d \
STDERR: 2023-02-13T23:41:00.040944164Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \
STDERR: 2023-02-13T23:41:00.040947533Z --store \
STDERR: 2023-02-13T23:41:00.040950281Z --status \
STDERR: 2023-02-13T23:41:00.040952948Z VALIDATED \
STDERR: 2023-02-13T23:41:00.040955880Z --parentid \
STDERR: 2023-02-13T23:41:00.040958965Z syn51089588 \
STDERR: 2023-02-13T23:41:00.040961607Z -c \
STDERR: 2023-02-13T23:41:00.040964715Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \
STDERR: 2023-02-13T23:41:00.040968033Z -t \
STDERR: 2023-02-13T23:41:00.040970868Z 2
STDERR: 2023-02-13T23:41:00.040973619Z Welcome, Vijay Yadav!
STDERR: 2023-02-13T23:41:00.040976564Z
STDERR: 2023-02-13T23:41:00.040979352Z
STDERR: 2023-02-13T23:41:00.040982128Z UPGRADE AVAILABLE
STDERR: 2023-02-13T23:41:00.040985022Z
STDERR: 2023-02-13T23:41:00.040988062Z A more recent version of the Synapse Client (2.7.0) is available. Your version (2.5.1) can be upgraded by typing:
STDERR: 2023-02-13T23:41:00.040991300Z pip install --upgrade synapseclient
STDERR: 2023-02-13T23:41:00.040994415Z
STDERR: 2023-02-13T23:41:00.040997310Z Python Synapse Client version 2.7.0 release notes
STDERR: 2023-02-13T23:41:00.041000824Z
STDERR: 2023-02-13T23:41:00.041003796Z https://python-docs.synapse.org/build/html/news.html
STDERR: 2023-02-13T23:41:00.041006951Z
STDERR: 2023-02-13T23:41:00.041009854Z root
STDERR: 2023-02-13T23:41:00.041012850Z mounting volumes
STDERR: 2023-02-13T23:41:00.041016266Z checking for containers
STDERR: 2023-02-13T23:41:00.041019579Z running container
STDERR: 2023-02-13T23:41:00.041022947Z creating logfile
STDERR: 2023-02-13T23:41:00.041026030Z Traceback (most recent call last):
STDERR: 2023-02-13T23:41:00.041029170Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
STDERR: 2023-02-13T23:41:00.041032249Z six.raise_from(e, None)
STDERR: 2023-02-13T23:41:00.041035223Z File "", line 3, in raise_from
STDERR: 2023-02-13T23:41:00.041038705Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
STDERR: 2023-02-13T23:41:00.041041913Z httplib_response = conn.getresponse()
STDERR: 2023-02-13T23:41:00.041045147Z File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
STDERR: 2023-02-13T23:41:00.041051633Z response.begin()
STDERR: 2023-02-13T23:41:00.041054763Z File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
STDERR: 2023-02-13T23:41:00.041057897Z version, status, reason = self._read_status()
STDERR: 2023-02-13T23:41:00.041060866Z File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
STDERR: 2023-02-13T23:41:00.041064894Z line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
STDERR: 2023-02-13T23:41:00.041068178Z File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
STDERR: 2023-02-13T23:41:00.041071421Z return self._sock.recv_into(b)
STDERR: 2023-02-13T23:41:00.041074442Z socket.timeout: timed out
STDERR: 2023-02-13T23:41:00.041077411Z
STDERR: 2023-02-13T23:41:00.041080339Z During handling of the above exception, another exception occurred:
STDERR: 2023-02-13T23:41:00.041083531Z
STDERR: 2023-02-13T23:41:00.041086463Z Traceback (most recent call last):
STDERR: 2023-02-13T23:41:00.041089471Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
STDERR: 2023-02-13T23:41:00.041092716Z resp = conn.urlopen(
STDERR: 2023-02-13T23:41:00.041095721Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
STDERR: 2023-02-13T23:41:00.041099095Z retries = retries.increment(
STDERR: 2023-02-13T23:41:00.041102216Z File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
STDERR: 2023-02-13T23:41:00.041105722Z raise six.reraise(type(error), error, _stacktrace)
STDERR: 2023-02-13T23:41:00.041108843Z File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
STDERR: 2023-02-13T23:41:00.041112007Z raise value
STDERR: 2023-02-13T23:41:00.041114795Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
STDERR: 2023-02-13T23:41:00.041131187Z httplib_response = self._make_request(
STDERR: 2023-02-13T23:41:00.041134167Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 451, in _make_request
STDERR: 2023-02-13T23:41:00.041137866Z self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
STDERR: 2023-02-13T23:41:00.041141051Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
STDERR: 2023-02-13T23:41:00.041144552Z raise ReadTimeoutError(
STDERR: 2023-02-13T23:41:00.041147647Z urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
STDERR: 2023-02-13T23:41:00.041151235Z
STDERR: 2023-02-13T23:41:00.041154353Z During handling of the above exception, another exception occurred:
STDERR: 2023-02-13T23:41:00.041157762Z
STDERR: 2023-02-13T23:41:00.041163817Z Traceback (most recent call last):
STDERR: 2023-02-13T23:41:00.041167087Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 252, in
STDERR: 2023-02-13T23:41:00.041171095Z main(syn, args)
STDERR: 2023-02-13T23:41:00.041213788Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 193, in main
STDERR: 2023-02-13T23:41:00.041219232Z while container in client.containers.list(ignore_removed=True):
STDERR: 2023-02-13T23:41:00.041222350Z File "/usr/local/lib/python3.8/site-packages/docker/models/containers.py", line 952, in list
STDERR: 2023-02-13T23:41:00.041225766Z resp = self.client.api.containers(all=all, before=before,
STDERR: 2023-02-13T23:41:00.041229049Z File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 209, in containers
STDERR: 2023-02-13T23:41:00.041232476Z res = self._result(self._get(u, params=params), True)
STDERR: 2023-02-13T23:41:00.041235648Z File "/usr/local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
STDERR: 2023-02-13T23:41:00.041238906Z return f(self, *args, **kwargs)
STDERR: 2023-02-13T23:41:00.041242056Z File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
STDERR: 2023-02-13T23:41:00.041245574Z return self.get(url, **self._set_request_timeout(kwargs))
STDERR: 2023-02-13T23:41:00.041248917Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in get
STDERR: 2023-02-13T23:41:00.041261718Z return self.request('GET', url, **kwargs)
STDERR: 2023-02-13T23:41:00.041264992Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
STDERR: 2023-02-13T23:41:00.041268120Z resp = self.send(prep, **send_kwargs)
STDERR: 2023-02-13T23:41:00.041271034Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
STDERR: 2023-02-13T23:41:00.041278261Z r = adapter.send(request, **kwargs)
STDERR: 2023-02-13T23:41:00.041281349Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
STDERR: 2023-02-13T23:41:00.041284548Z raise ReadTimeout(e, request=request)
STDERR: 2023-02-13T23:41:00.041287407Z requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
STDERR: 2023-02-13T23:41:00.041290537Z [job run_docker.cwl] Max memory used: 19MiB
STDERR: 2023-02-13T23:41:00.041293596Z INFO:cwltool:[job run_docker.cwl] Max memory used: 19MiB
STDERR: 2023-02-13T23:41:00.041296671Z [job run_docker.cwl] Job error:
STDERR: 2023-02-13T23:41:00.041299833Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {})
STDERR: 2023-02-13T23:41:00.041307870Z ERROR:cwltool:[job run_docker.cwl] Job error:
STDERR: 2023-02-13T23:41:00.041311156Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {})
STDERR: 2023-02-13T23:41:00.041314764Z [job run_docker.cwl] completed permanentFail
STDERR: 2023-02-13T23:41:00.041317769Z WARNING:cwltool:[job run_docker.cwl] completed permanentFail
STDERR: 2023-02-13T23:41:00.041321637Z Traceback (most recent call last):
STDERR: 2023-02-13T23:41:00.041324884Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2023-02-13T23:41:00.041328307Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2023-02-13T23:41:00.041331526Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2023-02-13T23:41:00.041335147Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2023-02-13T23:41:00.041338337Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run
STDERR: 2023-02-13T23:41:00.041341524Z return self.run(fileStore)
STDERR: 2023-02-13T23:41:00.041344624Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2023-02-13T23:41:00.041347769Z raise cwltool.errors.WorkflowException(status)
STDERR: 2023-02-13T23:41:00.041350568Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2023-02-13T23:41:00.041353466Z ERROR:toil.worker:Exiting the worker because of a failed job on host 915a75a9bc13
STDERR: 2023-02-13T23:41:00.041356807Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys to 0
STDERR: 2023-02-13T23:41:00.044068912Z <=========
```
I noticed from the log file that the error occurs in `run_docker.py` between the creation of the log file and the finished training section.
Plus, according to the code (run_docker.py), log file name should be "projectID__log.txt" however the log file I received was "projectID_logs.txt".
@sieberts Could you please confirm that your backend is okay? Hello,
I have also the same challenge. :(