Hello Again, My Submission is getting a bit further now I changed the header rows, this is my new Error log, submission id is 9749224 STDERR: 2024-08-30T16:56:24.913969867Z 7fdebdb849b5 2024-08-30 16:56:24,912 MainThread INFO cwltool: Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/MIDI-B-De-identification-main/workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/MIDI-B-De-identification-main/workflow.cwl' STDERR: 2024-08-30T16:56:32.264015553Z 7fdebdb849b5 2024-08-30 16:56:32,263 MainThread WARNING cwltool: Workflow checker warning: STDERR: 2024-08-30T16:56:32.264072823Z MIDI-B-De-identification-main/workflow.cwl:78:9: Source 'filepath' of type ["null", "File"] may be STDERR: 2024-08-30T16:56:32.264083233Z incompatible STDERR: 2024-08-30T16:56:32.264089918Z MIDI-B-De-identification-main/workflow.cwl:86:9: with sink 'compressed_file' of type "File" STDERR: 2024-08-30T16:56:34.777863128Z 7fdebdb849b5 2024-08-30 16:56:34,777 MainThread INFO toil: Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2024-08-30T16:56:36.244863602Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpdwsfav4_/worker_log.txt STDERR: 2024-08-30T16:56:36.673452596Z 7fdebdb849b5 2024-08-30 16:56:36,671 MainThread INFO toil.leader: 0 jobs are running, 0 jobs are issued and waiting to run STDERR: 2024-08-30T16:56:36.673494364Z 7fdebdb849b5 2024-08-30 16:56:36,672 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.1_cwl_get_submission.cwl/instance-_n5ahrfy with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-08-30T16:56:36.673504065Z 7fdebdb849b5 2024-08-30 16:56:36,672 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.1_cwl_set_permissions.cwl/instance-9uwj6y02 with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-08-30T16:56:36.673511639Z 7fdebdb849b5 2024-08-30 16:56:36,673 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.1_cwl_set_permissions.cwl/instance-4p_lkfey with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-08-30T16:56:38.990961031Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmp03jyu3_g/worker_log.txt STDERR: 2024-08-30T16:56:39.565604344Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmp6h0i5_vd/worker_log.txt STDERR: 2024-08-30T16:56:39.835829943Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmp0ki3255x/worker_log.txt STDERR: 2024-08-30T16:56:44.413867052Z 7fdebdb849b5 2024-08-30 16:56:44,412 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.1_cwl_set_permissions.cwl/instance-9uwj6y02 STDERR: 2024-08-30T16:56:45.421419201Z 7fdebdb849b5 2024-08-30 16:56:45,421 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.1_cwl_set_permissions.cwl/instance-4p_lkfey STDERR: 2024-08-30T16:58:34.081542234Z 7fdebdb849b5 2024-08-30 16:58:34,081 Thread-48 WARNING toil.statsAndLogging: Got message from job at time 08-30-2024 16:58:34: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-r8w6n_n7/cleanup/file-iyanu0_6/stream used 702.08% (7.0 GB [7545569280B] used, 1.0 GB [1074741824B] requested) at the end of its run. STDERR: 2024-08-30T16:58:34.624497201Z 7fdebdb849b5 2024-08-30 16:58:34,622 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.1_cwl_get_submission.cwl/instance-_n5ahrfy STDERR: 2024-08-30T16:58:34.624540045Z 7fdebdb849b5 2024-08-30 16:58:34,624 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/MIDI-B-De-identification-main/steps/test.cwl' /bin/bash -c kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_480c8c1c-62a8-40e5-a525-84677bf53473_MIDI-B-De-identification-main_steps_test.cwl/instance-tdra1sw0 with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 15.6 G STDERR: 2024-08-30T16:58:36.334702994Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/worker_log.txt STDERR: 2024-08-30T17:03:43.751334938Z 7fdebdb849b5 2024-08-30 17:03:43,749 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/MIDI-B-De-identification-main/steps/test.cwl' /bin/bash -c kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_480c8c1c-62a8-40e5-a525-84677bf53473_MIDI-B-De-identification-main_steps_test.cwl/instance-tdra1sw0 STDERR: 2024-08-30T17:03:43.751389772Z 7fdebdb849b5 2024-08-30 17:03:43,750 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/MIDI-B-De-identification-main/steps/test.cwl' /bin/bash -c kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_480c8c1c-62a8-40e5-a525-84677bf53473_MIDI-B-De-identification-main_steps_test.cwl/instance-tdra1sw0 STDERR: 2024-08-30T17:03:43.751401493Z 7fdebdb849b5 2024-08-30 17:03:43,750 MainThread WARNING toil.leader: Log from job kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_480c8c1c-62a8-40e5-a525-84677bf53473_MIDI-B-De-identification-main_steps_test.cwl/instance-tdra1sw0 follows: STDERR: 2024-08-30T17:03:43.751408603Z =========> STDERR: 2024-08-30T17:03:43.751414264Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2024-08-30T17:03:43.751432250Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2024-08-30T17:03:43.751440106Z [job test.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2024-08-30T17:03:43.751446633Z WARNING:cwltool:[job test.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2024-08-30T17:03:43.751453132Z [job test.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd$ docker \ STDERR: 2024-08-30T17:03:43.751459803Z run \ STDERR: 2024-08-30T17:03:43.751465870Z -i \ STDERR: 2024-08-30T17:03:43.751473744Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd,target=/aKXkoT \ STDERR: 2024-08-30T17:03:43.751480883Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wkemk9nk3vc,target=/tmp \ STDERR: 2024-08-30T17:03:43.751487973Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tmpawzigolw.tmp,target=/var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224,readonly \ STDERR: 2024-08-30T17:03:43.751494894Z --workdir=/aKXkoT \ STDERR: 2024-08-30T17:03:43.751500690Z --read-only=true \ STDERR: 2024-08-30T17:03:43.751506668Z --user=0:0 \ STDERR: 2024-08-30T17:03:43.751512498Z --rm \ STDERR: 2024-08-30T17:03:43.751518086Z --env=TMPDIR=/tmp \ STDERR: 2024-08-30T17:03:43.751523914Z --env=HOME=/aKXkoT \ STDERR: 2024-08-30T17:03:43.751529467Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wked73zwndy/20240830165836-614849.cid \ STDERR: 2024-08-30T17:03:43.751535730Z docker.synapse.org/syn53065762/validate_score:v12 \ STDERR: 2024-08-30T17:03:43.751541291Z /bin/bash \ STDERR: 2024-08-30T17:03:43.751546688Z -c \ STDERR: 2024-08-30T17:03:43.751552195Z 'python /usr/local/bin/MIDI_validation_script/run_validation.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 && STDERR: 2024-08-30T17:03:43.751558297Z python /usr/local/bin/MIDI_validation_script/run_reports.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 # && STDERR: 2024-08-30T17:03:43.751569781Z # mkdir dciodvfy && STDERR: 2024-08-30T17:03:43.751575992Z # python /usr/local/bin/MIDI_validation_script/run_dciodvfy.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224' \ STDERR: 2024-08-30T17:03:43.751581872Z /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 STDERR: 2024-08-30T17:03:43.751588446Z INFO:cwltool:[job test.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd$ docker \ STDERR: 2024-08-30T17:03:43.751595424Z run \ STDERR: 2024-08-30T17:03:43.751601102Z -i \ STDERR: 2024-08-30T17:03:43.751606744Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd,target=/aKXkoT \ STDERR: 2024-08-30T17:03:43.751613497Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wkemk9nk3vc,target=/tmp \ STDERR: 2024-08-30T17:03:43.751619854Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tmpawzigolw.tmp,target=/var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224,readonly \ STDERR: 2024-08-30T17:03:43.751626852Z --workdir=/aKXkoT \ STDERR: 2024-08-30T17:03:43.751632492Z --read-only=true \ STDERR: 2024-08-30T17:03:43.751638126Z --user=0:0 \ STDERR: 2024-08-30T17:03:43.751643932Z --rm \ STDERR: 2024-08-30T17:03:43.751649410Z --env=TMPDIR=/tmp \ STDERR: 2024-08-30T17:03:43.751655052Z --env=HOME=/aKXkoT \ STDERR: 2024-08-30T17:03:43.751661006Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wked73zwndy/20240830165836-614849.cid \ STDERR: 2024-08-30T17:03:43.751667871Z docker.synapse.org/syn53065762/validate_score:v12 \ STDERR: 2024-08-30T17:03:43.751673655Z /bin/bash \ STDERR: 2024-08-30T17:03:43.751679290Z -c \ STDERR: 2024-08-30T17:03:43.751685019Z 'python /usr/local/bin/MIDI_validation_script/run_validation.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 && STDERR: 2024-08-30T17:03:43.751691537Z python /usr/local/bin/MIDI_validation_script/run_reports.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 # && STDERR: 2024-08-30T17:03:43.751698145Z # mkdir dciodvfy && STDERR: 2024-08-30T17:03:43.751705044Z # python /usr/local/bin/MIDI_validation_script/run_dciodvfy.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224' \ STDERR: 2024-08-30T17:03:43.751715957Z /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 STDERR: 2024-08-30T17:03:43.751722305Z 2024-08-30 17:01:07,220 - [INFO] - Run Started STDERR: 2024-08-30T17:03:43.751728289Z 2024-08-30 17:01:07,220 - [INFO] - Initialization Started STDERR: 2024-08-30T17:03:43.751734233Z 2024-08-30 17:01:07,225 - [INFO] - Validation Result DB Created STDERR: 2024-08-30T17:03:43.751740036Z 2024-08-30 17:01:30,118 - [INFO] - Answer Key Imported: 23921 Records STDERR: 2024-08-30T17:03:43.751746175Z 2024-08-30 17:01:30,355 - [INFO] - UID Mapping Imported: 24967 Records STDERR: 2024-08-30T17:03:43.751752210Z 2024-08-30 17:01:30,365 - [INFO] - PatID Mapping Imported: 216 Records STDERR: 2024-08-30T17:03:43.751757724Z 2024-08-30 17:01:30,365 - [INFO] - Initialization Complete STDERR: 2024-08-30T17:03:43.751763156Z 2024-08-30 17:01:30,366 - [INFO] - Directory Indexing Started STDERR: 2024-08-30T17:03:43.751768790Z { STDERR: 2024-08-30T17:03:43.751773876Z "config_file": { STDERR: 2024-08-30T17:03:43.751779487Z "run_name": "MIDI_1_1_Testing", STDERR: 2024-08-30T17:03:43.751785295Z "input_data_path": "submission/mappings", STDERR: 2024-08-30T17:03:43.751791055Z "output_data_path": "results", STDERR: 2024-08-30T17:03:43.751796715Z "answer_db_file": "/usr/local/bin/MIDI_validation_script/midi_1_1_answer_data_1.db", STDERR: 2024-08-30T17:03:43.751802644Z "uid_mapping_file": "submission/mappings/uid_mapping.csv", STDERR: 2024-08-30T17:03:43.751808142Z "patid_mapping_file": "submission/mappings/patient_id_mapping.csv", STDERR: 2024-08-30T17:03:43.751813785Z "multiprocessing": "True", STDERR: 2024-08-30T17:03:43.751819340Z "multiprocessing_cpus": "5", STDERR: 2024-08-30T17:03:43.751825156Z "log_path": "logs", STDERR: 2024-08-30T17:03:43.751830701Z "log_level": "info", STDERR: 2024-08-30T17:03:43.751836379Z "report_series": "True" STDERR: 2024-08-30T17:03:43.751842039Z } STDERR: 2024-08-30T17:03:43.751847491Z } STDERR: 2024-08-30T17:03:43.751852926Z Indexing File Batches: 0%| | 0/2 [00:00 STDERR: 2024-08-30T17:03:51.679957856Z sys.exit(main()) STDERR: 2024-08-30T17:03:51.679984925Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 1691, in main STDERR: 2024-08-30T17:03:51.679990423Z outobj = toil.start(wf1) STDERR: 2024-08-30T17:03:51.679995891Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 829, in start STDERR: 2024-08-30T17:03:51.680001375Z return self._runMainLoop(rootJobGraph) STDERR: 2024-08-30T17:03:51.680015747Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 1115, in _runMainLoop STDERR: 2024-08-30T17:03:51.680021795Z return Leader(config=self.config, STDERR: 2024-08-30T17:03:51.680027055Z File "/usr/local/lib/python3.8/site-packages/toil/leader.py", line 269, in run STDERR: 2024-08-30T17:03:51.680033022Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2024-08-30T17:03:51.680203779Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/tmpdky6s6ci' contains 3 failed jobs: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/MIDI-B-De-identification-main/steps/test.cwl' /bin/bash -c kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_480c8c1c-62a8-40e5-a525-84677bf53473_MIDI-B-De-identification-main_steps_test.cwl/instance-tdra1sw0, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.1_cwl_get_submission.cwl/instance-_n5ahrfy, 'CWLWorkflow' kind-CWLWorkflow/instance-r8w6n_n7 STDERR: 2024-08-30T17:03:51.680217186Z Log from job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/MIDI-B-De-identification-main/steps/test.cwl' /bin/bash -c kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_480c8c1c-62a8-40e5-a525-84677bf53473_MIDI-B-De-identification-main_steps_test.cwl/instance-tdra1sw0 follows: STDERR: 2024-08-30T17:03:51.680223708Z =========> STDERR: 2024-08-30T17:03:51.680228701Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2024-08-30T17:03:51.680234288Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2024-08-30T17:03:51.680239851Z [job test.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2024-08-30T17:03:51.680263791Z WARNING:cwltool:[job test.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2024-08-30T17:03:51.680270472Z [job test.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd$ docker \ STDERR: 2024-08-30T17:03:51.680277061Z run \ STDERR: 2024-08-30T17:03:51.680282626Z -i \ STDERR: 2024-08-30T17:03:51.680290096Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd,target=/aKXkoT \ STDERR: 2024-08-30T17:03:51.680296933Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wkemk9nk3vc,target=/tmp \ STDERR: 2024-08-30T17:03:51.680312969Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tmpawzigolw.tmp,target=/var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224,readonly \ STDERR: 2024-08-30T17:03:51.680319690Z --workdir=/aKXkoT \ STDERR: 2024-08-30T17:03:51.680325094Z --read-only=true \ STDERR: 2024-08-30T17:03:51.680330189Z --user=0:0 \ STDERR: 2024-08-30T17:03:51.680335259Z --rm \ STDERR: 2024-08-30T17:03:51.680340380Z --env=TMPDIR=/tmp \ STDERR: 2024-08-30T17:03:51.680345317Z --env=HOME=/aKXkoT \ STDERR: 2024-08-30T17:03:51.680350569Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wked73zwndy/20240830165836-614849.cid \ STDERR: 2024-08-30T17:03:51.680357269Z docker.synapse.org/syn53065762/validate_score:v12 \ STDERR: 2024-08-30T17:03:51.680362702Z /bin/bash \ STDERR: 2024-08-30T17:03:51.680367828Z -c \ STDERR: 2024-08-30T17:03:51.680372980Z 'python /usr/local/bin/MIDI_validation_script/run_validation.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 && STDERR: 2024-08-30T17:03:51.680378772Z python /usr/local/bin/MIDI_validation_script/run_reports.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 # && STDERR: 2024-08-30T17:03:51.680384508Z # mkdir dciodvfy && STDERR: 2024-08-30T17:03:51.680389397Z # python /usr/local/bin/MIDI_validation_script/run_dciodvfy.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224' \ STDERR: 2024-08-30T17:03:51.680394859Z /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 STDERR: 2024-08-30T17:03:51.680401294Z INFO:cwltool:[job test.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd$ docker \ STDERR: 2024-08-30T17:03:51.680408189Z run \ STDERR: 2024-08-30T17:03:51.680414189Z -i \ STDERR: 2024-08-30T17:03:51.680419947Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tzhpb6h3x/tmp-out3en8idgd,target=/aKXkoT \ STDERR: 2024-08-30T17:03:51.680426072Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wkemk9nk3vc,target=/tmp \ STDERR: 2024-08-30T17:03:51.680432231Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/tmpawzigolw.tmp,target=/var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224,readonly \ STDERR: 2024-08-30T17:03:51.680442330Z --workdir=/aKXkoT \ STDERR: 2024-08-30T17:03:51.680447466Z --read-only=true \ STDERR: 2024-08-30T17:03:51.680452420Z --user=0:0 \ STDERR: 2024-08-30T17:03:51.680457628Z --rm \ STDERR: 2024-08-30T17:03:51.680462617Z --env=TMPDIR=/tmp \ STDERR: 2024-08-30T17:03:51.680467583Z --env=HOME=/aKXkoT \ STDERR: 2024-08-30T17:03:51.680472758Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/480c8c1c-62a8-40e5-a525-84677bf53473/node-e5004aa6-43f1-4145-a583-d4f320fcef29-1cc402dd0e11d5ae18db04a6de87223d/tmpc2pxpfef/70913cd8-516a-4229-848b-aca91ad80d36/t6ses_wked73zwndy/20240830165836-614849.cid \ STDERR: 2024-08-30T17:03:51.680478939Z docker.synapse.org/syn53065762/validate_score:v12 \ STDERR: 2024-08-30T17:03:51.680483998Z /bin/bash \ STDERR: 2024-08-30T17:03:51.680489070Z -c \ STDERR: 2024-08-30T17:03:51.680494180Z 'python /usr/local/bin/MIDI_validation_script/run_validation.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 && STDERR: 2024-08-30T17:03:51.680499753Z python /usr/local/bin/MIDI_validation_script/run_reports.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 # && STDERR: 2024-08-30T17:03:51.680505611Z # mkdir dciodvfy && STDERR: 2024-08-30T17:03:51.680511338Z # python /usr/local/bin/MIDI_validation_script/run_dciodvfy.py /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224' \ STDERR: 2024-08-30T17:03:51.680517249Z /var/lib/cwl/stg5448e507-caf4-4220-b338-a5ed0df3da62/submission-9749224 STDERR: 2024-08-30T17:03:51.680523533Z 2024-08-30 17:01:07,220 - [INFO] - Run Started STDERR: 2024-08-30T17:03:51.680529119Z 2024-08-30 17:01:07,220 - [INFO] - Initialization Started STDERR: 2024-08-30T17:03:51.680534828Z 2024-08-30 17:01:07,225 - [INFO] - Validation Result DB Created STDERR: 2024-08-30T17:03:51.680540469Z 2024-08-30 17:01:30,118 - [INFO] - Answer Key Imported: 23921 Records STDERR: 2024-08-30T17:03:51.680546138Z 2024-08-30 17:01:30,355 - [INFO] - UID Mapping Imported: 24967 Records STDERR: 2024-08-30T17:03:51.680551558Z 2024-08-30 17:01:30,365 - [INFO] - PatID Mapping Imported: 216 Records STDERR: 2024-08-30T17:03:51.680556751Z 2024-08-30 17:01:30,365 - [INFO] - Initialization Complete STDERR: 2024-08-30T17:03:51.680562208Z 2024-08-30 17:01:30,366 - [INFO] - Directory Indexing Started STDERR: 2024-08-30T17:03:51.680567306Z { STDERR: 2024-08-30T17:03:51.680572358Z "config_file": { STDERR: 2024-08-30T17:03:51.680577588Z "run_name": "MIDI_1_1_Testing", STDERR: 2024-08-30T17:03:51.680582750Z "input_data_path": "submission/mappings", STDERR: 2024-08-30T17:03:51.680588373Z "output_data_path": "results", STDERR: 2024-08-30T17:03:51.680593751Z "answer_db_file": "/usr/local/bin/MIDI_validation_script/midi_1_1_answer_data_1.db", STDERR: 2024-08-30T17:03:51.680602977Z "uid_mapping_file": "submission/mappings/uid_mapping.csv", STDERR: 2024-08-30T17:03:51.680608456Z "patid_mapping_file": "submission/mappings/patient_id_mapping.csv", STDERR: 2024-08-30T17:03:51.680613941Z "multiprocessing": "True", STDERR: 2024-08-30T17:03:51.680619557Z "multiprocessing_cpus": "5", STDERR: 2024-08-30T17:03:51.680625010Z "log_path": "logs", STDERR: 2024-08-30T17:03:51.680630448Z "log_level": "info", STDERR: 2024-08-30T17:03:51.680636104Z "report_series": "True" STDERR: 2024-08-30T17:03:51.680641952Z } STDERR: 2024-08-30T17:03:51.680647375Z } STDERR: 2024-08-30T17:03:51.680653034Z Indexing File Batches: 0%| | 0/2 [00:00Created by Chris Ablett ChrisAb
@phmoer thanks for your help. It works now. Hopefully we are qualified for the next stage.
It looks to be working, normally I would have received an error by now. We will probably score only average as it is our 1st pass and I was not sure how to handle some of the PHI and private tags, for example, PHI in the Creator field, unfortunately I won't get a chance to make a 2nd submission in validation phase but we can make the required changes for the next phase. It is late for me now so I will check in the morning and hopefully the workflow completes. Regards, Chris
Hopefully, everything goes well. I want to remind you that today is the last day of the validation phase. The submission challenge will be closed after that.
I think I might have found the issue, some of the DICOM Files were duplicated for the same SOP Instance UID: 1.2.840.113654.2.174.69.74377345215629523413962636888524887184.dcm 1.2.840.113654.2.174.69.74377345215629523413962636888524887184[1].dcm I will delete these and have another try. Regards, Chris
Thanks @phmoer The 2nd directory structure is the one intended and as far as I know it looks Ok. Here is an example of 1 patient with 3 different Studies (all with 1 Series): C:\data\1007891030\1.2.840.113654.2.174.69.223715460560568928821754397970058025692\1.2.840.113654.2.174.69.87719002715665682864355257580026653301 C:\data\1007891030\1.2.840.113654.2.174.69.26145205136261889078622778969331016940\1.2.840.113654.2.174.69.141868078502216206777550465912051376943 C:\data\1007891030\1.2.840.113654.2.174.69.280967519456398613221188932867682619596\1.2.840.113654.2.174.69.69929887880784572423262823003651662811 Here is an example of 1 Study with 2 Series: C:\data\1396842432\1.2.840.113654.2.174.69.127782434570282007883500899563760212955\1.2.840.113654.2.174.69.12378017947619366094870036820509311274 C:\data\1396842432\1.2.840.113654.2.174.69.127782434570282007883500899563760212955\1.2.840.113654.2.174.69.170396840069663740274338396809462381712 The only issue I can see is maybe the name of the DICOM Files is the SOP_Instance_UID.dcm , for example, 1.2.840.113654.2.174.69.100283709184763085724207096030636521503.dcm Would that cause the issue ? If that format is Ok do you have an example of a Study causing a problem ? Regards, Chris
When I check the log, the data path is showing: "input_data_path": "submission/mappings". The correct data structure should be either one of the following: ??? data ? ??? instance001.dcm ? ??? instance002.dcm ? ...... ? ??? instance100.dcm ??? mappings ? ??? patient_id_mapping.csv ? ??? uid_mapping.csv or ??? data ? ??? patientID_1 ? ? ??? StudyUID ? ? ? ??? seriesUID ? ? ? ? ??? instance001.dcm ? ? ? ? ??? instance002.dcm ? ...... ? ??? patientID_n ? ? ??? StudyUID ? ? ? ??? seriesUID ? ? ? ? ??? instance001.dcm ? ? ? ? ??? instance002.dcm ??? mappings ? ??? patient_id_mapping.csv ? ??? uid_mapping.csv

Drop files to upload

New Submission Error = KeyError: 'instance'0 page is loading…