Hi @vjbytes102 , I am currently running into an issue where "No 'predictions.csv' file written to /output" is being logged. I somehow cannot locate the error on my side even after adding the file statically. I am able to produce results using the debug solution proposed by you here https://www.synapse.org/#!Synapse:syn31472953/wiki/619724. Here is the file tree in my docker image ./ /input/ CODA_TB_Clinical_Meta_Info_Test.csv CODA_TB_Clinical_Meta_Info.csv CODA_TB_Solicited_Meta_Info.csv CODA_TB_Longitudnal_Meta_Info.csv meta_info.csv raw_test_data/ /output/ model.onnx predictions.csv ${preview?entityId=syn51037001} I checked the source code creating this error, and it looks like the error-raising code section has been commented out, but somehow it is still raised. Does someone have a solution for this? Complete error log STDERR: 2023-02-01T17:43:08.073426290Z f4d0a7ee0568 2023-02-01 17:43:08,073 MainThread INFO cwltool: Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/workflow.cwl' STDERR: 2023-02-01T17:43:12.710323194Z f4d0a7ee0568 2023-02-01 17:43:12,710 MainThread WARNING cwltool: Workflow checker warning: STDERR: 2023-02-01T17:43:12.710366024Z TB-DREAM-Challenge-Infra-main/workflow.cwl:72:9: Source 'docker_repository' of type ["null", STDERR: 2023-02-01T17:43:12.710370981Z "string"] may be incompatible STDERR: 2023-02-01T17:43:12.710374288Z TB-DREAM-Challenge-Infra-main/workflow.cwl:172:9: with sink 'docker_repository' of type "string" STDERR: 2023-02-01T17:43:12.710377989Z TB-DREAM-Challenge-Infra-main/workflow.cwl:73:9: Source 'docker_digest' of type ["null", "string"] STDERR: 2023-02-01T17:43:12.710381101Z may be incompatible STDERR: 2023-02-01T17:43:12.710384457Z TB-DREAM-Challenge-Infra-main/workflow.cwl:174:9: with sink 'docker_digest' of type "string" STDERR: 2023-02-01T17:43:14.261098403Z f4d0a7ee0568 2023-02-01 17:43:14,260 MainThread INFO toil: Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2023-02-01T17:43:15.224443541Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpn284448x/worker_log.txt STDERR: 2023-02-01T17:43:15.580953424Z f4d0a7ee0568 2023-02-01 17:43:15,580 MainThread INFO toil.leader: 0 jobs are running, 0 jobs are issued and waiting to run STDERR: 2023-02-01T17:43:15.582035210Z f4d0a7ee0568 2023-02-01 17:43:15,581 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-jaffon4r with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:15.582196130Z f4d0a7ee0568 2023-02-01 17:43:15,582 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-ngvba1tz with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:15.582365387Z f4d0a7ee0568 2023-02-01 17:43:15,582 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-n3x207bl with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:15.582496976Z f4d0a7ee0568 2023-02-01 17:43:15,582 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-_c70eyp0 with job batch system ID: 4 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:15.582629871Z f4d0a7ee0568 2023-02-01 17:43:15,582 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-ggxz193e with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:16.411563326Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpa2p_6yt8/worker_log.txt STDERR: 2023-02-01T17:43:16.437029223Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpdo1v8m6j/worker_log.txt STDERR: 2023-02-01T17:43:19.643757554Z f4d0a7ee0568 2023-02-01 17:43:19,642 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-ngvba1tz STDERR: 2023-02-01T17:43:20.268864756Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpv2rwl9b1/worker_log.txt STDERR: 2023-02-01T17:43:20.725319759Z f4d0a7ee0568 2023-02-01 17:43:20,724 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-jaffon4r STDERR: 2023-02-01T17:43:21.651926384Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpep4bo5jk/worker_log.txt STDERR: 2023-02-01T17:43:23.321857679Z f4d0a7ee0568 2023-02-01 17:43:23,321 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-_c70eyp0 STDERR: 2023-02-01T17:43:23.322830418Z f4d0a7ee0568 2023-02-01 17:43:23,322 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-bvx7guul with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:23.970592261Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpjr3zlzyo/worker_log.txt STDERR: 2023-02-01T17:43:24.751927990Z f4d0a7ee0568 2023-02-01 17:43:24,751 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-n3x207bl STDERR: 2023-02-01T17:43:24.753209347Z f4d0a7ee0568 2023-02-01 17:43:24,752 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-n5yod8sh with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:24.753490794Z f4d0a7ee0568 2023-02-01 17:43:24,753 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-1l3ly92a with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:25.695459392Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpqq14n_gu/worker_log.txt STDERR: 2023-02-01T17:43:26.108222353Z f4d0a7ee0568 2023-02-01 17:43:26,107 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-bvx7guul STDERR: 2023-02-01T17:43:26.109114483Z f4d0a7ee0568 2023-02-01 17:43:26,108 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x with job batch system ID: 9 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:26.109227826Z f4d0a7ee0568 2023-02-01 17:43:26,109 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-i4y7x5g8 with job batch system ID: 10 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:26.746208321Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmphthzm82t/worker_log.txt STDERR: 2023-02-01T17:43:27.068386800Z f4d0a7ee0568 2023-02-01 17:43:27,067 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-ggxz193e STDERR: 2023-02-01T17:43:27.880328584Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmp3fg2l1ya/worker_log.txt STDERR: 2023-02-01T17:43:28.359084745Z f4d0a7ee0568 2023-02-01 17:43:28,358 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-i4y7x5g8 STDERR: 2023-02-01T17:43:28.360053232Z f4d0a7ee0568 2023-02-01 17:43:28,359 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-bhb3niwu with job batch system ID: 11 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:28.944094204Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpm8mpsgt9/worker_log.txt STDERR: 2023-02-01T17:43:30.064586551Z f4d0a7ee0568 2023-02-01 17:43:30,064 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-1l3ly92a STDERR: 2023-02-01T17:43:30.925401821Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpyli9v8qm/worker_log.txt STDERR: 2023-02-01T17:43:31.908027352Z f4d0a7ee0568 2023-02-01 17:43:31,907 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-n5yod8sh STDERR: 2023-02-01T17:43:31.908940544Z f4d0a7ee0568 2023-02-01 17:43:31,908 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-nzldvhum with job batch system ID: 12 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:32.699390651Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/worker_log.txt STDERR: 2023-02-01T17:43:33.945224936Z f4d0a7ee0568 2023-02-01 17:43:33,944 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-bhb3niwu STDERR: 2023-02-01T17:43:34.675681320Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpc5hp3cnn/worker_log.txt STDERR: 2023-02-01T17:43:35.187686054Z f4d0a7ee0568 2023-02-01 17:43:35,162 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-nzldvhum STDERR: 2023-02-01T17:43:35.187709382Z f4d0a7ee0568 2023-02-01 17:43:35,163 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-1l3ly92a with job batch system ID: 13 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-01T17:43:36.233584860Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpji_dnzol/worker_log.txt STDERR: 2023-02-01T17:43:36.441822066Z f4d0a7ee0568 2023-02-01 17:43:36,440 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-1l3ly92a STDERR: 2023-02-01T17:44:57.587987372Z f4d0a7ee0568 2023-02-01 17:44:57,587 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x STDERR: 2023-02-01T17:44:57.588503599Z f4d0a7ee0568 2023-02-01 17:44:57,588 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x STDERR: 2023-02-01T17:44:57.588745015Z f4d0a7ee0568 2023-02-01 17:44:57,588 MainThread WARNING toil.leader: Log from job kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x follows: STDERR: 2023-02-01T17:44:57.588754299Z =========> STDERR: 2023-02-01T17:44:57.588758050Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2023-02-01T17:44:57.588761568Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2023-02-01T17:44:57.588765017Z [job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83$ python3 \ STDERR: 2023-02-01T17:44:57.588768957Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py \ STDERR: 2023-02-01T17:44:57.588772353Z -s \ STDERR: 2023-02-01T17:44:57.588775120Z 9731839 \ STDERR: 2023-02-01T17:44:57.588777792Z -p \ STDERR: 2023-02-01T17:44:57.588780490Z docker.synapse.org/syn51035605/steph_model \ STDERR: 2023-02-01T17:44:57.588783476Z -d \ STDERR: 2023-02-01T17:44:57.588797723Z sha256:503e3b14d1449a02643e894cdaadc3321d07a71e97f33d493122e5da750bb14c \ STDERR: 2023-02-01T17:44:57.588801116Z --store \ STDERR: 2023-02-01T17:44:57.588804135Z --status \ STDERR: 2023-02-01T17:44:57.588807519Z VALIDATED \ STDERR: 2023-02-01T17:44:57.588810014Z --parentid \ STDERR: 2023-02-01T17:44:57.588813225Z syn51036990 \ STDERR: 2023-02-01T17:44:57.588816081Z -c \ STDERR: 2023-02-01T17:44:57.588819117Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/tmpg293y8l2/stgf6cd72be-05dc-456d-8584-ff10dd739ebf/.synapseConfig \ STDERR: 2023-02-01T17:44:57.588822102Z -t \ STDERR: 2023-02-01T17:44:57.588824928Z 1 STDERR: 2023-02-01T17:44:57.588827908Z INFO:cwltool:[job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83$ python3 \ STDERR: 2023-02-01T17:44:57.588831478Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py \ STDERR: 2023-02-01T17:44:57.588835536Z -s \ STDERR: 2023-02-01T17:44:57.588838304Z 9731839 \ STDERR: 2023-02-01T17:44:57.588841049Z -p \ STDERR: 2023-02-01T17:44:57.588844828Z docker.synapse.org/syn51035605/steph_model \ STDERR: 2023-02-01T17:44:57.588848044Z -d \ STDERR: 2023-02-01T17:44:57.588850837Z sha256:503e3b14d1449a02643e894cdaadc3321d07a71e97f33d493122e5da750bb14c \ STDERR: 2023-02-01T17:44:57.588853708Z --store \ STDERR: 2023-02-01T17:44:57.588856729Z --status \ STDERR: 2023-02-01T17:44:57.588859565Z VALIDATED \ STDERR: 2023-02-01T17:44:57.588862444Z --parentid \ STDERR: 2023-02-01T17:44:57.588865319Z syn51036990 \ STDERR: 2023-02-01T17:44:57.588868008Z -c \ STDERR: 2023-02-01T17:44:57.588870625Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/tmpg293y8l2/stgf6cd72be-05dc-456d-8584-ff10dd739ebf/.synapseConfig \ STDERR: 2023-02-01T17:44:57.588873865Z -t \ STDERR: 2023-02-01T17:44:57.588876684Z 1 STDERR: 2023-02-01T17:44:57.588879560Z Welcome, Vijay Yadav! STDERR: 2023-02-01T17:44:57.588882271Z STDERR: 2023-02-01T17:44:57.588884893Z root STDERR: 2023-02-01T17:44:57.588887337Z mounting volumes STDERR: 2023-02-01T17:44:57.588889967Z checking for containers STDERR: 2023-02-01T17:44:57.588893029Z running container STDERR: 2023-02-01T17:44:57.588896205Z creating logfile STDERR: 2023-02-01T17:44:57.588902414Z STDERR: 2023-02-01T17:44:57.588905606Z ################################################## STDERR: 2023-02-01T17:44:57.588908751Z Uploading file to Synapse storage STDERR: 2023-02-01T17:44:57.588912043Z ################################################## STDERR: 2023-02-01T17:44:57.588915141Z STDERR: 2023-02-01T17:44:57.588918117Z STDERR: 2023-02-01T17:44:57.588920705Z ################################################## STDERR: 2023-02-01T17:44:57.588923601Z Uploading file to Synapse storage STDERR: 2023-02-01T17:44:57.588926761Z ################################################## STDERR: 2023-02-01T17:44:57.588929805Z STDERR: 2023-02-01T17:44:57.588932547Z STDERR: 2023-02-01T17:44:57.588935163Z UPGRADE AVAILABLE STDERR: 2023-02-01T17:44:57.588938223Z STDERR: 2023-02-01T17:44:57.588940749Z A more recent version of the Synapse Client (2.7.0) is available. Your version (2.5.1) can be upgraded by typing: STDERR: 2023-02-01T17:44:57.588943644Z pip install --upgrade synapseclient STDERR: 2023-02-01T17:44:57.588946569Z STDERR: 2023-02-01T17:44:57.588949512Z Python Synapse Client version 2.7.0 release notes STDERR: 2023-02-01T17:44:57.588952341Z STDERR: 2023-02-01T17:44:57.588955134Z https://python-docs.synapse.org/build/html/news.html STDERR: 2023-02-01T17:44:57.588958025Z STDERR: 2023-02-01T17:44:57.588960768Z finished training STDERR: 2023-02-01T17:44:57.588963553Z Traceback (most recent call last): STDERR: 2023-02-01T17:44:57.588966441Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py", line 252, in STDERR: 2023-02-01T17:44:57.588970705Z main(syn, args) STDERR: 2023-02-01T17:44:57.588973495Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py", line 222, in main STDERR: 2023-02-01T17:44:57.588977291Z raise Exception("No 'predictions.csv' file written to /output, " STDERR: 2023-02-01T17:44:57.588980453Z Exception: No 'predictions.csv' file written to /output, please check inference docker STDERR: 2023-02-01T17:44:57.588983808Z [job run_docker.cwl] Max memory used: 53MiB STDERR: 2023-02-01T17:44:57.588986793Z INFO:cwltool:[job run_docker.cwl] Max memory used: 53MiB STDERR: 2023-02-01T17:44:57.588989781Z [job run_docker.cwl] Job error: STDERR: 2023-02-01T17:44:57.588993261Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-01T17:44:57.588999929Z ERROR:cwltool:[job run_docker.cwl] Job error: STDERR: 2023-02-01T17:44:57.589003129Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-01T17:44:57.589006628Z [job run_docker.cwl] completed permanentFail STDERR: 2023-02-01T17:44:57.589009655Z WARNING:cwltool:[job run_docker.cwl] completed permanentFail STDERR: 2023-02-01T17:44:57.589012401Z Traceback (most recent call last): STDERR: 2023-02-01T17:44:57.589015273Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript STDERR: 2023-02-01T17:44:57.589018295Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2023-02-01T17:44:57.589021465Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner STDERR: 2023-02-01T17:44:57.589024657Z returnValues = self._run(jobGraph, fileStore) STDERR: 2023-02-01T17:44:57.589027830Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run STDERR: 2023-02-01T17:44:57.589030935Z return self.run(fileStore) STDERR: 2023-02-01T17:44:57.589034041Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run STDERR: 2023-02-01T17:44:57.589037277Z raise cwltool.errors.WorkflowException(status) STDERR: 2023-02-01T17:44:57.589040465Z cwltool.errors.WorkflowException: permanentFail STDERR: 2023-02-01T17:44:57.589043565Z ERROR:toil.worker:Exiting the worker because of a failed job on host f4d0a7ee0568 STDERR: 2023-02-01T17:44:57.589050202Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x to 0 STDERR: 2023-02-01T17:44:57.589063393Z <========= STDERR: 2023-02-01T17:44:57.589737375Z f4d0a7ee0568 2023-02-01 17:44:57,589 MainThread WARNING toil.leader: Job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x is completely failed STDERR: 2023-02-01T17:45:11.543560665Z f4d0a7ee0568 2023-02-01 17:45:11,543 MainThread INFO toil.leader: Finished toil run with 9 failed jobs. STDERR: 2023-02-01T17:45:11.543804847Z f4d0a7ee0568 2023-02-01 17:45:11,543 MainThread INFO toil.leader: Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-bvx7guul 'CWLWorkflow' kind-CWLWorkflow/instance-djld4mlb 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-ngvba1tz 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-n5yod8sh 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-bhb3niwu 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-_c70eyp0 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-n3x207bl 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-i4y7x5g8 STDERR: 2023-02-01T17:45:11.555179806Z Traceback (most recent call last): STDERR: 2023-02-01T17:45:11.555238430Z File "/usr/local/bin/toil-cwl-runner", line 8, in STDERR: 2023-02-01T17:45:11.555244136Z sys.exit(main()) STDERR: 2023-02-01T17:45:11.555247707Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 1691, in main STDERR: 2023-02-01T17:45:11.555251682Z outobj = toil.start(wf1) STDERR: 2023-02-01T17:45:11.555254439Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 829, in start STDERR: 2023-02-01T17:45:11.555258028Z return self._runMainLoop(rootJobGraph) STDERR: 2023-02-01T17:45:11.555261010Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 1115, in _runMainLoop STDERR: 2023-02-01T17:45:11.555264262Z return Leader(config=self.config, STDERR: 2023-02-01T17:45:11.555267494Z File "/usr/local/lib/python3.8/site-packages/toil/leader.py", line 269, in run STDERR: 2023-02-01T17:45:11.555270420Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2023-02-01T17:45:11.555281258Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/tmpdbhf0_58' contains 9 failed jobs: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-bvx7guul, 'CWLWorkflow' kind-CWLWorkflow/instance-djld4mlb, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-ngvba1tz, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-n5yod8sh, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-bhb3niwu, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-_c70eyp0, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-n3x207bl, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-i4y7x5g8 STDERR: 2023-02-01T17:45:11.555316309Z Log from job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x follows: STDERR: 2023-02-01T17:45:11.555320811Z =========> STDERR: 2023-02-01T17:45:11.555324339Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2023-02-01T17:45:11.555327526Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2023-02-01T17:45:11.555331023Z [job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83$ python3 \ STDERR: 2023-02-01T17:45:11.555406762Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py \ STDERR: 2023-02-01T17:45:11.555411203Z -s \ STDERR: 2023-02-01T17:45:11.555414218Z 9731839 \ STDERR: 2023-02-01T17:45:11.555417334Z -p \ STDERR: 2023-02-01T17:45:11.555420437Z docker.synapse.org/syn51035605/steph_model \ STDERR: 2023-02-01T17:45:11.555427326Z -d \ STDERR: 2023-02-01T17:45:11.555429943Z sha256:503e3b14d1449a02643e894cdaadc3321d07a71e97f33d493122e5da750bb14c \ STDERR: 2023-02-01T17:45:11.555433046Z --store \ STDERR: 2023-02-01T17:45:11.555435666Z --status \ STDERR: 2023-02-01T17:45:11.555438907Z VALIDATED \ STDERR: 2023-02-01T17:45:11.555441686Z --parentid \ STDERR: 2023-02-01T17:45:11.555444823Z syn51036990 \ STDERR: 2023-02-01T17:45:11.555447782Z -c \ STDERR: 2023-02-01T17:45:11.555451060Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/tmpg293y8l2/stgf6cd72be-05dc-456d-8584-ff10dd739ebf/.synapseConfig \ STDERR: 2023-02-01T17:45:11.555454670Z -t \ STDERR: 2023-02-01T17:45:11.555457893Z 1 STDERR: 2023-02-01T17:45:11.555473766Z INFO:cwltool:[job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83$ python3 \ STDERR: 2023-02-01T17:45:11.555478182Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py \ STDERR: 2023-02-01T17:45:11.555481926Z -s \ STDERR: 2023-02-01T17:45:11.555485158Z 9731839 \ STDERR: 2023-02-01T17:45:11.555488286Z -p \ STDERR: 2023-02-01T17:45:11.555491387Z docker.synapse.org/syn51035605/steph_model \ STDERR: 2023-02-01T17:45:11.555495169Z -d \ STDERR: 2023-02-01T17:45:11.555498577Z sha256:503e3b14d1449a02643e894cdaadc3321d07a71e97f33d493122e5da750bb14c \ STDERR: 2023-02-01T17:45:11.555693800Z --store \ STDERR: 2023-02-01T17:45:11.555697280Z --status \ STDERR: 2023-02-01T17:45:11.555700657Z VALIDATED \ STDERR: 2023-02-01T17:45:11.555703881Z --parentid \ STDERR: 2023-02-01T17:45:11.555707060Z syn51036990 \ STDERR: 2023-02-01T17:45:11.555710603Z -c \ STDERR: 2023-02-01T17:45:11.555713890Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/tmpg293y8l2/stgf6cd72be-05dc-456d-8584-ff10dd739ebf/.synapseConfig \ STDERR: 2023-02-01T17:45:11.555717697Z -t \ STDERR: 2023-02-01T17:45:11.555720717Z 1 STDERR: 2023-02-01T17:45:11.555724027Z Welcome, Vijay Yadav! STDERR: 2023-02-01T17:45:11.555727087Z STDERR: 2023-02-01T17:45:11.555730364Z root STDERR: 2023-02-01T17:45:11.555733507Z mounting volumes STDERR: 2023-02-01T17:45:11.555736763Z checking for containers STDERR: 2023-02-01T17:45:11.555740501Z running container STDERR: 2023-02-01T17:45:11.555749048Z creating logfile STDERR: 2023-02-01T17:45:11.555765611Z STDERR: 2023-02-01T17:45:11.555768731Z ################################################## STDERR: 2023-02-01T17:45:11.555772496Z Uploading file to Synapse storage STDERR: 2023-02-01T17:45:11.555775571Z ################################################## STDERR: 2023-02-01T17:45:11.555778733Z STDERR: 2023-02-01T17:45:11.555782421Z STDERR: 2023-02-01T17:45:11.555785567Z ################################################## STDERR: 2023-02-01T17:45:11.555788994Z Uploading file to Synapse storage STDERR: 2023-02-01T17:45:11.555793782Z ################################################## STDERR: 2023-02-01T17:45:11.555797352Z STDERR: 2023-02-01T17:45:11.555800287Z STDERR: 2023-02-01T17:45:11.555803308Z UPGRADE AVAILABLE STDERR: 2023-02-01T17:45:11.555806599Z STDERR: 2023-02-01T17:45:11.555809899Z A more recent version of the Synapse Client (2.7.0) is available. Your version (2.5.1) can be upgraded by typing: STDERR: 2023-02-01T17:45:11.555813418Z pip install --upgrade synapseclient STDERR: 2023-02-01T17:45:11.555816795Z STDERR: 2023-02-01T17:45:11.555820037Z Python Synapse Client version 2.7.0 release notes STDERR: 2023-02-01T17:45:11.555823523Z STDERR: 2023-02-01T17:45:11.555845100Z https://python-docs.synapse.org/build/html/news.html STDERR: 2023-02-01T17:45:11.555848904Z STDERR: 2023-02-01T17:45:11.555852197Z finished training STDERR: 2023-02-01T17:45:11.555855531Z Traceback (most recent call last): STDERR: 2023-02-01T17:45:11.555859080Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py", line 252, in STDERR: 2023-02-01T17:45:11.555863515Z main(syn, args) STDERR: 2023-02-01T17:45:11.555867162Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/node-dc2d6c5b-0156-4e6d-ab84-1ef0f40ea660-1cc402dd0e11d5ae18db04a6de87223d/tmpnkf0h32j/da0e0ef4-fb4a-4517-b860-7ad40d769c8e/th9oxjbsy/tmp-out1fr5ph83/run_docker.py", line 222, in main STDERR: 2023-02-01T17:45:11.555871340Z raise Exception("No 'predictions.csv' file written to /output, " STDERR: 2023-02-01T17:45:11.555874687Z Exception: No 'predictions.csv' file written to /output, please check inference docker STDERR: 2023-02-01T17:45:11.555878420Z [job run_docker.cwl] Max memory used: 53MiB STDERR: 2023-02-01T17:45:11.555881698Z INFO:cwltool:[job run_docker.cwl] Max memory used: 53MiB STDERR: 2023-02-01T17:45:11.555885157Z [job run_docker.cwl] Job error: STDERR: 2023-02-01T17:45:11.555888400Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-01T17:45:11.555930589Z ERROR:cwltool:[job run_docker.cwl] Job error: STDERR: 2023-02-01T17:45:11.555934270Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-01T17:45:11.555938247Z [job run_docker.cwl] completed permanentFail STDERR: 2023-02-01T17:45:11.555941303Z WARNING:cwltool:[job run_docker.cwl] completed permanentFail STDERR: 2023-02-01T17:45:11.555944811Z Traceback (most recent call last): STDERR: 2023-02-01T17:45:11.555948183Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript STDERR: 2023-02-01T17:45:11.555951889Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2023-02-01T17:45:11.556012951Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner STDERR: 2023-02-01T17:45:11.556017237Z returnValues = self._run(jobGraph, fileStore) STDERR: 2023-02-01T17:45:11.556020463Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run STDERR: 2023-02-01T17:45:11.556023775Z return self.run(fileStore) STDERR: 2023-02-01T17:45:11.556027001Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run STDERR: 2023-02-01T17:45:11.556030375Z raise cwltool.errors.WorkflowException(status) STDERR: 2023-02-01T17:45:11.556033521Z cwltool.errors.WorkflowException: permanentFail STDERR: 2023-02-01T17:45:11.556036840Z ERROR:toil.worker:Exiting the worker because of a failed job on host f4d0a7ee0568 STDERR: 2023-02-01T17:45:11.556040499Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/453b0911-de6a-40c0-ab6f-d8b9c9eb48db/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_453b0911-de6a-40c0-ab6f-d8b9c9eb48db_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-6prvq83x to 0 STDERR: 2023-02-01T17:45:11.556045297Z <=========