Dear challenge manager,
I have submitted many times my docker submission, which have been tested valid on my windows laptop, to the challenge.
I also have confirmed that the inferring process is successful due to no logs in the "9730114_log.txt" file.
However, I still could not get it scored VALID, no matter how hard I tried to debug.
Could you please help me fix the issue?
Sincerely yours,
Tsai-Min
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PS "9730114_logs.zip":
STDERR: 2022-12-30T05:53:26.485602396Z 67eb95e8a027 2022-12-30 05:53:26,485 MainThread INFO cwltool: Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/workflow.cwl'
STDERR: 2022-12-30T05:53:30.936060706Z 67eb95e8a027 2022-12-30 05:53:30,935 MainThread WARNING cwltool: Workflow checker warning:
STDERR: 2022-12-30T05:53:30.936092162Z TB-DREAM-Challenge-Infra-main/workflow.cwl:72:9: Source 'docker_repository' of type ["null",
STDERR: 2022-12-30T05:53:30.936096489Z "string"] may be incompatible
STDERR: 2022-12-30T05:53:30.936099719Z TB-DREAM-Challenge-Infra-main/workflow.cwl:172:9: with sink 'docker_repository' of type "string"
STDERR: 2022-12-30T05:53:30.936102709Z TB-DREAM-Challenge-Infra-main/workflow.cwl:73:9: Source 'docker_digest' of type ["null", "string"]
STDERR: 2022-12-30T05:53:30.936105885Z may be incompatible
STDERR: 2022-12-30T05:53:30.936109224Z TB-DREAM-Challenge-Infra-main/workflow.cwl:174:9: with sink 'docker_digest' of type "string"
STDERR: 2022-12-30T05:53:32.262450429Z 67eb95e8a027 2022-12-30 05:53:32,262 MainThread INFO toil: Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2022-12-30T05:53:32.939127148Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpjz3lct3b/worker_log.txt
STDERR: 2022-12-30T05:53:33.363764250Z 67eb95e8a027 2022-12-30 05:53:33,363 MainThread INFO toil.leader: 0 jobs are running, 0 jobs are issued and waiting to run
STDERR: 2022-12-30T05:53:33.365226681Z 67eb95e8a027 2022-12-30 05:53:33,365 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-qb0hl39_ with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:33.365383251Z 67eb95e8a027 2022-12-30 05:53:33,365 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-yf13dzqc with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:33.365535650Z 67eb95e8a027 2022-12-30 05:53:33,365 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-obqjym56 with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:33.365768272Z 67eb95e8a027 2022-12-30 05:53:33,365 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-380c44bj with job batch system ID: 4 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:33.365935783Z 67eb95e8a027 2022-12-30 05:53:33,365 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-1bv64lqa with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:34.187927096Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp1utzyh6v/worker_log.txt
STDERR: 2022-12-30T05:53:34.214139923Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpllveu_x1/worker_log.txt
STDERR: 2022-12-30T05:53:37.193327533Z 67eb95e8a027 2022-12-30 05:53:37,192 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-qb0hl39_
STDERR: 2022-12-30T05:53:37.194371358Z 67eb95e8a027 2022-12-30 05:53:37,194 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-5gcd9r17 with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:37.744669266Z 67eb95e8a027 2022-12-30 05:53:37,744 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-yf13dzqc
STDERR: 2022-12-30T05:53:37.745675096Z 67eb95e8a027 2022-12-30 05:53:37,745 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-tnka9dm1 with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:37.746002905Z 67eb95e8a027 2022-12-30 05:53:37,745 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-7qvus7u1 with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:38.167439062Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpixm6cfvc/worker_log.txt
STDERR: 2022-12-30T05:53:38.649836745Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp6i8gjdav/worker_log.txt
STDERR: 2022-12-30T05:53:38.982658910Z 67eb95e8a027 2022-12-30 05:53:38,982 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-5gcd9r17
STDERR: 2022-12-30T05:53:38.983836132Z 67eb95e8a027 2022-12-30 05:53:38,983 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-wpulkert with job batch system ID: 9 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:39.883332882Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpl1x63mnj/worker_log.txt
STDERR: 2022-12-30T05:53:40.486267244Z 67eb95e8a027 2022-12-30 05:53:40,485 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-obqjym56
STDERR: 2022-12-30T05:53:40.492356105Z 67eb95e8a027 2022-12-30 05:53:40,491 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-wx98cuab with job batch system ID: 10 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:41.409790376Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpsej3q680/worker_log.txt
STDERR: 2022-12-30T05:53:41.810058121Z 67eb95e8a027 2022-12-30 05:53:41,809 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-wpulkert
STDERR: 2022-12-30T05:53:41.810604830Z 67eb95e8a027 2022-12-30 05:53:41,810 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-5y7g7g39 with job batch system ID: 11 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:53:42.391219267Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpid6fc_t2/worker_log.txt
STDERR: 2022-12-30T05:53:42.881130066Z 67eb95e8a027 2022-12-30 05:53:42,880 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-380c44bj
STDERR: 2022-12-30T05:53:43.555058506Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp6bd8ml7y/worker_log.txt
STDERR: 2022-12-30T05:53:46.356208749Z 67eb95e8a027 2022-12-30 05:53:46,355 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-5y7g7g39
STDERR: 2022-12-30T05:53:47.146218253Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpgtcg8sa0/worker_log.txt
STDERR: 2022-12-30T05:53:51.043225877Z 67eb95e8a027 2022-12-30 05:53:51,042 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-7qvus7u1
STDERR: 2022-12-30T05:53:51.889024949Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp3z7rnjwh/worker_log.txt
STDERR: 2022-12-30T05:53:54.996881135Z 67eb95e8a027 2022-12-30 05:53:54,996 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-1bv64lqa
STDERR: 2022-12-30T05:53:55.907715737Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpydqsubac/worker_log.txt
STDERR: 2022-12-30T05:53:59.875996985Z 67eb95e8a027 2022-12-30 05:53:59,875 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-tnka9dm1
STDERR: 2022-12-30T05:53:59.877059283Z 67eb95e8a027 2022-12-30 05:53:59,876 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-2diyo_cb with job batch system ID: 12 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:54:01.083150944Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmph1eloztw/worker_log.txt
STDERR: 2022-12-30T05:54:01.601190025Z 67eb95e8a027 2022-12-30 05:54:01,600 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-2diyo_cb
STDERR: 2022-12-30T05:54:01.602214678Z 67eb95e8a027 2022-12-30 05:54:01,601 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-7qvus7u1 with job batch system ID: 13 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:54:02.653405988Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpkrjtdzg4/worker_log.txt
STDERR: 2022-12-30T05:54:02.861265644Z 67eb95e8a027 2022-12-30 05:54:02,860 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-7qvus7u1
STDERR: 2022-12-30T05:57:57.359333995Z 67eb95e8a027 2022-12-30 05:57:57,359 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-wx98cuab
STDERR: 2022-12-30T05:57:57.359940720Z 67eb95e8a027 2022-12-30 05:57:57,359 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/validate.cwl' validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_validate.cwl/instance-6b3tp61b with job batch system ID: 14 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:57:57.360037139Z 67eb95e8a027 2022-12-30 05:57:57,359 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/upload_to_synapse.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_upload_to_synapse.cwl/instance-v2hzldq5 with job batch system ID: 15 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:57:58.186177535Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpaf_p4y_k/worker_log.txt
STDERR: 2022-12-30T05:57:58.186199049Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpi9l9rfqp/worker_log.txt
STDERR: 2022-12-30T05:58:00.407890861Z 67eb95e8a027 2022-12-30 05:58:00,407 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/validate.cwl' validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_validate.cwl/instance-6b3tp61b
STDERR: 2022-12-30T05:58:00.408680535Z 67eb95e8a027 2022-12-30 05:58:00,408 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-j67fsap0 with job batch system ID: 16 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:58:00.978917339Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp1ynepdm0/worker_log.txt
STDERR: 2022-12-30T05:58:01.627013793Z 67eb95e8a027 2022-12-30 05:58:01,626 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/upload_to_synapse.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_upload_to_synapse.cwl/instance-v2hzldq5
STDERR: 2022-12-30T05:58:01.627658367Z 67eb95e8a027 2022-12-30 05:58:01,627 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-jp1jemdb with job batch system ID: 17 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:58:02.496297337Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpz_2wel8o/worker_log.txt
STDERR: 2022-12-30T05:58:04.283844749Z 67eb95e8a027 2022-12-30 05:58:04,283 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-j67fsap0
STDERR: 2022-12-30T05:58:04.918572533Z 67eb95e8a027 2022-12-30 05:58:04,918 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-jp1jemdb
STDERR: 2022-12-30T05:58:04.919190682Z 67eb95e8a027 2022-12-30 05:58:04,919 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-rlivu7nx with job batch system ID: 18 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:58:05.449590134Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmpb4i59rwn/worker_log.txt
STDERR: 2022-12-30T05:58:07.713812398Z 67eb95e8a027 2022-12-30 05:58:07,713 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-rlivu7nx
STDERR: 2022-12-30T05:58:07.714444817Z 67eb95e8a027 2022-12-30 05:58:07,714 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary with job batch system ID: 19 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2022-12-30T05:58:08.246916946Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp3hoq3oqp/worker_log.txt
STDERR: 2022-12-30T05:58:08.526101654Z 67eb95e8a027 2022-12-30 05:58:08,525 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary
STDERR: 2022-12-30T05:58:08.526678927Z 67eb95e8a027 2022-12-30 05:58:08,526 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary
STDERR: 2022-12-30T05:58:08.526860307Z 67eb95e8a027 2022-12-30 05:58:08,526 MainThread WARNING toil.leader: Log from job kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary follows:
STDERR: 2022-12-30T05:58:08.526869440Z =========>
STDERR: 2022-12-30T05:58:08.526872647Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2022-12-30T05:58:08.526875881Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2022-12-30T05:58:08.526879260Z Failed to evaluate expression:
STDERR: 2022-12-30T05:58:08.526882731Z Expression evaluation error:
STDERR: 2022-12-30T05:58:08.526886065Z Expecting value: line 1 column 1 (char 0)
STDERR: 2022-12-30T05:58:08.526889103Z script was:
STDERR: 2022-12-30T05:58:08.526892112Z 01 "use strict";
STDERR: 2022-12-30T05:58:08.526895692Z 02 var inputs = {
STDERR: 2022-12-30T05:58:08.526899004Z 03 "status": "INVALID",
STDERR: 2022-12-30T05:58:08.526902359Z 04 "previous_annotation_finished": true,
STDERR: 2022-12-30T05:58:08.526905609Z 05 "previous_email_finished": true
STDERR: 2022-12-30T05:58:08.526908831Z 06 };
STDERR: 2022-12-30T05:58:08.526912511Z 07 var self = null;
STDERR: 2022-12-30T05:58:08.526915443Z 08 var runtime = {
STDERR: 2022-12-30T05:58:08.526918226Z 09 "cores": 1,
STDERR: 2022-12-30T05:58:08.526921334Z 10 "ram": 100.0,
STDERR: 2022-12-30T05:58:08.526924719Z 11 "tmpdirSize": 1024,
STDERR: 2022-12-30T05:58:08.526928034Z 12 "outdirSize": 0.95367431640625,
STDERR: 2022-12-30T05:58:08.526931331Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/tmph56zmt6c",
STDERR: 2022-12-30T05:58:08.526935531Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp3hoq3oqp/dd4cbaa6-de9b-4c71-98e3-589c576e181f/t70307nkj/tmp-outvzc2_k0_"
STDERR: 2022-12-30T05:58:08.526939856Z 15 };
STDERR: 2022-12-30T05:58:08.526954911Z 16 (function(){
STDERR: 2022-12-30T05:58:08.526958387Z 17 if(inputs.status == "INVALID"){
STDERR: 2022-12-30T05:58:08.526961231Z 18 throw 'invalid submission';
STDERR: 2022-12-30T05:58:08.526964203Z 19 } else {
STDERR: 2022-12-30T05:58:08.526967023Z 20 return {finished: true};
STDERR: 2022-12-30T05:58:08.526969864Z 21 }
STDERR: 2022-12-30T05:58:08.526973057Z 22 })()
STDERR: 2022-12-30T05:58:08.526976714Z stdout was: ''
STDERR: 2022-12-30T05:58:08.526979425Z stderr was: 'invalid submission'
STDERR: 2022-12-30T05:58:08.526982434Z
STDERR: 2022-12-30T05:58:08.526985474Z WARNING:cwltool:Failed to evaluate expression:
STDERR: 2022-12-30T05:58:08.526988599Z Expression evaluation error:
STDERR: 2022-12-30T05:58:08.526991755Z Expecting value: line 1 column 1 (char 0)
STDERR: 2022-12-30T05:58:08.526994835Z script was:
STDERR: 2022-12-30T05:58:08.526998094Z 01 "use strict";
STDERR: 2022-12-30T05:58:08.527001357Z 02 var inputs = {
STDERR: 2022-12-30T05:58:08.527004384Z 03 "status": "INVALID",
STDERR: 2022-12-30T05:58:08.527007454Z 04 "previous_annotation_finished": true,
STDERR: 2022-12-30T05:58:08.527010468Z 05 "previous_email_finished": true
STDERR: 2022-12-30T05:58:08.527013587Z 06 };
STDERR: 2022-12-30T05:58:08.527016231Z 07 var self = null;
STDERR: 2022-12-30T05:58:08.527019235Z 08 var runtime = {
STDERR: 2022-12-30T05:58:08.527022129Z 09 "cores": 1,
STDERR: 2022-12-30T05:58:08.527025170Z 10 "ram": 100.0,
STDERR: 2022-12-30T05:58:08.527028325Z 11 "tmpdirSize": 1024,
STDERR: 2022-12-30T05:58:08.527031378Z 12 "outdirSize": 0.95367431640625,
STDERR: 2022-12-30T05:58:08.527034432Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/tmph56zmt6c",
STDERR: 2022-12-30T05:58:08.527038334Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp3hoq3oqp/dd4cbaa6-de9b-4c71-98e3-589c576e181f/t70307nkj/tmp-outvzc2_k0_"
STDERR: 2022-12-30T05:58:08.527041733Z 15 };
STDERR: 2022-12-30T05:58:08.527044379Z 16 (function(){
STDERR: 2022-12-30T05:58:08.527047300Z 17 if(inputs.status == "INVALID"){
STDERR: 2022-12-30T05:58:08.527050373Z 18 throw 'invalid submission';
STDERR: 2022-12-30T05:58:08.527053496Z 19 } else {
STDERR: 2022-12-30T05:58:08.527056961Z 20 return {finished: true};
STDERR: 2022-12-30T05:58:08.527060247Z 21 }
STDERR: 2022-12-30T05:58:08.527063357Z 22 })()
STDERR: 2022-12-30T05:58:08.527066455Z stdout was: ''
STDERR: 2022-12-30T05:58:08.527072759Z stderr was: 'invalid submission'
STDERR: 2022-12-30T05:58:08.527076503Z
STDERR: 2022-12-30T05:58:08.527079738Z Traceback (most recent call last):
STDERR: 2022-12-30T05:58:08.527082714Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2022-12-30T05:58:08.527086185Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2022-12-30T05:58:08.527089167Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2022-12-30T05:58:08.527092247Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2022-12-30T05:58:08.527095402Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run
STDERR: 2022-12-30T05:58:08.527098561Z return self.run(fileStore)
STDERR: 2022-12-30T05:58:08.527101364Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2022-12-30T05:58:08.527104514Z raise cwltool.errors.WorkflowException(status)
STDERR: 2022-12-30T05:58:08.527107421Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2022-12-30T05:58:08.527110634Z ERROR:toil.worker:Exiting the worker because of a failed job on host 67eb95e8a027
STDERR: 2022-12-30T05:58:08.527113939Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary with ID kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary to 0
STDERR: 2022-12-30T05:58:08.527118565Z <=========
STDERR: 2022-12-30T05:58:08.527477128Z 67eb95e8a027 2022-12-30 05:58:08,527 MainThread WARNING toil.leader: Job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary with ID kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary is completely failed
STDERR: 2022-12-30T05:58:25.795375383Z 67eb95e8a027 2022-12-30 05:58:25,795 MainThread INFO toil.leader: Finished toil run with 15 failed jobs.
STDERR: 2022-12-30T05:58:25.795477392Z 67eb95e8a027 2022-12-30 05:58:25,795 MainThread INFO toil.leader: Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-wpulkert 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-j67fsap0 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-qb0hl39_ 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-wx98cuab 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/upload_to_synapse.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_upload_to_synapse.cwl/instance-v2hzldq5 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-jp1jemdb 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-yf13dzqc 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/validate.cwl' validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_validate.cwl/instance-6b3tp61b 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-obqjym56 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-5gcd9r17 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-tnka9dm1 'CWLWorkflow' kind-CWLWorkflow/instance-znq8s8qy 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-rlivu7nx 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-5y7g7g39
STDERR: 2022-12-30T05:58:25.811420201Z Traceback (most recent call last):
STDERR: 2022-12-30T05:58:25.811448224Z File "/usr/local/bin/toil-cwl-runner", line 8, in
STDERR: 2022-12-30T05:58:25.811452904Z sys.exit(main())
STDERR: 2022-12-30T05:58:25.811456419Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 1691, in main
STDERR: 2022-12-30T05:58:25.811460194Z outobj = toil.start(wf1)
STDERR: 2022-12-30T05:58:25.811463396Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 829, in start
STDERR: 2022-12-30T05:58:25.811466703Z return self._runMainLoop(rootJobGraph)
STDERR: 2022-12-30T05:58:25.811469903Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 1115, in _runMainLoop
STDERR: 2022-12-30T05:58:25.811487988Z return Leader(config=self.config,
STDERR: 2022-12-30T05:58:25.811490885Z File "/usr/local/lib/python3.8/site-packages/toil/leader.py", line 269, in run
STDERR: 2022-12-30T05:58:25.811494035Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
STDERR: 2022-12-30T05:58:25.811567145Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/tmpuz447z3u' contains 15 failed jobs: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-wpulkert, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-j67fsap0, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-qb0hl39_, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-wx98cuab, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/upload_to_synapse.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_upload_to_synapse.cwl/instance-v2hzldq5, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-jp1jemdb, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-yf13dzqc, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/validate.cwl' validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_validate.cwl/instance-6b3tp61b, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-obqjym56, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_89b9af3a-7e0d-41a8-82a8-085eec3abfd8_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-5gcd9r17, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-tnka9dm1, 'CWLWorkflow' kind-CWLWorkflow/instance-znq8s8qy, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-rlivu7nx, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-5y7g7g39
STDERR: 2022-12-30T05:58:25.811588692Z Log from job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary follows:
STDERR: 2022-12-30T05:58:25.811592616Z =========>
STDERR: 2022-12-30T05:58:25.811595898Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2022-12-30T05:58:25.811599420Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2022-12-30T05:58:25.811602912Z Failed to evaluate expression:
STDERR: 2022-12-30T05:58:25.811606294Z Expression evaluation error:
STDERR: 2022-12-30T05:58:25.811609358Z Expecting value: line 1 column 1 (char 0)
STDERR: 2022-12-30T05:58:25.811612745Z script was:
STDERR: 2022-12-30T05:58:25.811615759Z 01 "use strict";
STDERR: 2022-12-30T05:58:25.811619151Z 02 var inputs = {
STDERR: 2022-12-30T05:58:25.811622198Z 03 "status": "INVALID",
STDERR: 2022-12-30T05:58:25.811625657Z 04 "previous_annotation_finished": true,
STDERR: 2022-12-30T05:58:25.811629442Z 05 "previous_email_finished": true
STDERR: 2022-12-30T05:58:25.811632687Z 06 };
STDERR: 2022-12-30T05:58:25.811635995Z 07 var self = null;
STDERR: 2022-12-30T05:58:25.811639053Z 08 var runtime = {
STDERR: 2022-12-30T05:58:25.811642167Z 09 "cores": 1,
STDERR: 2022-12-30T05:58:25.811645183Z 10 "ram": 100.0,
STDERR: 2022-12-30T05:58:25.811648274Z 11 "tmpdirSize": 1024,
STDERR: 2022-12-30T05:58:25.811651406Z 12 "outdirSize": 0.95367431640625,
STDERR: 2022-12-30T05:58:25.811654739Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/tmph56zmt6c",
STDERR: 2022-12-30T05:58:25.811658375Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp3hoq3oqp/dd4cbaa6-de9b-4c71-98e3-589c576e181f/t70307nkj/tmp-outvzc2_k0_"
STDERR: 2022-12-30T05:58:25.811662457Z 15 };
STDERR: 2022-12-30T05:58:25.811665465Z 16 (function(){
STDERR: 2022-12-30T05:58:25.811668438Z 17 if(inputs.status == "INVALID"){
STDERR: 2022-12-30T05:58:25.811671463Z 18 throw 'invalid submission';
STDERR: 2022-12-30T05:58:25.811674902Z 19 } else {
STDERR: 2022-12-30T05:58:25.811678350Z 20 return {finished: true};
STDERR: 2022-12-30T05:58:25.811681631Z 21 }
STDERR: 2022-12-30T05:58:25.811688016Z 22 })()
STDERR: 2022-12-30T05:58:25.811691168Z stdout was: ''
STDERR: 2022-12-30T05:58:25.811694989Z stderr was: 'invalid submission'
STDERR: 2022-12-30T05:58:25.811698249Z
STDERR: 2022-12-30T05:58:25.811701189Z WARNING:cwltool:Failed to evaluate expression:
STDERR: 2022-12-30T05:58:25.811704496Z Expression evaluation error:
STDERR: 2022-12-30T05:58:25.811707748Z Expecting value: line 1 column 1 (char 0)
STDERR: 2022-12-30T05:58:25.811711068Z script was:
STDERR: 2022-12-30T05:58:25.811714249Z 01 "use strict";
STDERR: 2022-12-30T05:58:25.811717704Z 02 var inputs = {
STDERR: 2022-12-30T05:58:25.811721026Z 03 "status": "INVALID",
STDERR: 2022-12-30T05:58:25.811724366Z 04 "previous_annotation_finished": true,
STDERR: 2022-12-30T05:58:25.811727651Z 05 "previous_email_finished": true
STDERR: 2022-12-30T05:58:25.811731353Z 06 };
STDERR: 2022-12-30T05:58:25.811734169Z 07 var self = null;
STDERR: 2022-12-30T05:58:25.811781074Z 08 var runtime = {
STDERR: 2022-12-30T05:58:25.811784456Z 09 "cores": 1,
STDERR: 2022-12-30T05:58:25.811787403Z 10 "ram": 100.0,
STDERR: 2022-12-30T05:58:25.811790686Z 11 "tmpdirSize": 1024,
STDERR: 2022-12-30T05:58:25.811794558Z 12 "outdirSize": 0.95367431640625,
STDERR: 2022-12-30T05:58:25.811798115Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/tmph56zmt6c",
STDERR: 2022-12-30T05:58:25.811801951Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/89b9af3a-7e0d-41a8-82a8-085eec3abfd8/node-d93b0a00-1c7f-4418-a576-99f29aa8c4d2-1cc402dd0e11d5ae18db04a6de87223d/tmp3hoq3oqp/dd4cbaa6-de9b-4c71-98e3-589c576e181f/t70307nkj/tmp-outvzc2_k0_"
STDERR: 2022-12-30T05:58:25.811805864Z 15 };
STDERR: 2022-12-30T05:58:25.811809482Z 16 (function(){
STDERR: 2022-12-30T05:58:25.811812765Z 17 if(inputs.status == "INVALID"){
STDERR: 2022-12-30T05:58:25.811816255Z 18 throw 'invalid submission';
STDERR: 2022-12-30T05:58:25.811819571Z 19 } else {
STDERR: 2022-12-30T05:58:25.811823012Z 20 return {finished: true};
STDERR: 2022-12-30T05:58:25.811826200Z 21 }
STDERR: 2022-12-30T05:58:25.811829603Z 22 })()
STDERR: 2022-12-30T05:58:25.811832739Z stdout was: ''
STDERR: 2022-12-30T05:58:25.811836280Z stderr was: 'invalid submission'
STDERR: 2022-12-30T05:58:25.811839517Z
STDERR: 2022-12-30T05:58:25.811843002Z Traceback (most recent call last):
STDERR: 2022-12-30T05:58:25.811846463Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2022-12-30T05:58:25.811849996Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2022-12-30T05:58:25.811857404Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2022-12-30T05:58:25.811860988Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2022-12-30T05:58:25.811945720Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run
STDERR: 2022-12-30T05:58:25.811953830Z return self.run(fileStore)
STDERR: 2022-12-30T05:58:25.811957105Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2022-12-30T05:58:25.811960888Z raise cwltool.errors.WorkflowException(status)
STDERR: 2022-12-30T05:58:25.811964172Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2022-12-30T05:58:25.811967826Z ERROR:toil.worker:Exiting the worker because of a failed job on host 67eb95e8a027
STDERR: 2022-12-30T05:58:25.812028629Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary with ID kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-iu_zdary to 0
STDERR: 2022-12-30T05:58:25.812033937Z <=========
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Created by TSAI-MIN (??) CHEN (?) chentsaimin @vjbytes102 Thanks for your kind help.
My problem was solved in the latest submission "9730193."
Sincerely yours,
Tsai-Min @chentsaimin As per the competition guidelines, probability score should be between [0 to 1], but in your 'predictions.csv' few of participants have negative probability score. @vjbytes102 Thanks for your response.
I have added the extra code below to check the "probability" column in the "predictions.csv" file and the equality between the number of unique input participants and the output number of participants:
```
# comfirm that probability of predictions.csv exists
print(pd.read_csv('./output/predictions.csv')['probability'])
# confirm that the number of unique input participants is equal to the output number of participants.
print(len(pd.unique(pd.read_csv(input_path+'meta_info.csv')['participant'])), len(pd.read_csv('./output/predictions.csv')['participant']))
```
Although the "9730192_log.txt" file shows the 'probability' column correctly and there are no duplicate participant IDs as below, the status is still INVALID:
```
0 0.079849
1 0.256183
2 0.020971
3 0.057339
4 0.131064
...
243 0.252737
244 0.192485
245 0.729994
246 0.393373
247 0.495730
Name: probability, Length: 248, dtype: float64
248 248
```
Could you please help me fix the issue?
Sincerely yours,
Tsai-Min @chentsaimin Your 'predictions.csv' has duplicate participant IDs. Make sure you have one entry with a probability score for each participant IDs. @vjbytes102 Thanks for your response.
I have added the extra code below to check the "probability" column in the "predictions.csv" file:
```
# comfirm that probability of predictions.csv exists
print(pd.read_csv('./output/predictions.csv')['probability'])
```
Although the "9730131_log.txt" file shows the 'probability' column correctly as below, the status is still INVALID:
```
0 0.083965
1 0.056358
2 0.099225
3 0.253560
4 0.258150
...
2107 0.454865
2108 0.490748
2109 0.476869
2110 0.470725
2111 0.542333
Name: probability, Length: 2112, dtype: float64
```
Could you please help me fix the issue?
Sincerely yours,
Tsai-Min @chentsaimin I looked into your latest submission, 'probability' column is not available in the "prediction.csv". Looks like your inference script is not pushing predicted probability score in the 'prediction.csv' file.
Drop files to upload
(solved )[submissionID:9730114]No Logs, but still INVALID page is loading…