Thank-you for a great competion.
I'm trying to submit a test dataset of 10 validation studies but getting this sql error: _An unexpected error occurred: Execution failed on sql 'select * from validation_results': no such table: validation_results_
2024-08-30 04:27:16,720 - [INFO] - Run Started
STDERR: 2024-08-30T04:27:39.735756449Z 2024-08-30 04:27:16,720 - [INFO] - Initialization Started
STDERR: 2024-08-30T04:27:39.735763470Z 2024-08-30 04:27:16,721 - [INFO] - Validation Result DB Created
STDERR: 2024-08-30T04:27:39.735767118Z 2024-08-30 04:27:35,916 - [INFO] - Answer Key Imported: 23921 Records
STDERR: 2024-08-30T04:27:39.735770667Z 2024-08-30 04:27:35,964 - [ERROR] - Error:
STDERR: 2024-08-30T04:27:39.735774187Z Traceback (most recent call last):
STDERR: 2024-08-30T04:27:39.735777638Z File "/usr/local/bin/MIDI_validation_script/run_validation.py", line 203, in main
STDERR: 2024-08-30T04:27:39.735781329Z helper = validation_helper(config, log_path, log_level)
STDERR: 2024-08-30T04:27:39.735784875Z File "/usr/local/bin/MIDI_validation_script/modules/validation_helper.py", line 76, in __init__
STDERR: 2024-08-30T04:27:39.735788585Z uid_file.drop_duplicates(subset='id_old', keep='first', inplace=True)
STDERR: 2024-08-30T04:27:39.735792219Z File "/usr/local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
STDERR: 2024-08-30T04:27:39.735795974Z return func(*args, **kwargs)
STDERR: 2024-08-30T04:27:39.735800157Z File "/usr/local/lib/python3.9/site-packages/pandas/core/frame.py", line 6063, in drop_duplicates
STDERR: 2024-08-30T04:27:39.735803963Z duplicated = self.duplicated(subset, keep=keep)
STDERR: 2024-08-30T04:27:39.735807477Z File "/usr/local/lib/python3.9/site-packages/pandas/core/frame.py", line 6197, in duplicated
STDERR: 2024-08-30T04:27:39.735811089Z raise KeyError(diff)
STDERR: 2024-08-30T04:27:39.735814519Z KeyError: Index(['id_old'], dtype='object')
STDERR: 2024-08-30T04:27:39.735818110Z 2024-08-30 04:27:35,982 - [INFO] - Run Complete - Duration: (0, 20)
STDERR: 2024-08-30T04:27:39.735821718Z {
STDERR: 2024-08-30T04:27:39.735825490Z "config_file": {
STDERR: 2024-08-30T04:27:39.735830575Z "run_name": "MIDI_1_1_Testing",
STDERR: 2024-08-30T04:27:39.735837132Z "input_data_path": "submission",
STDERR: 2024-08-30T04:27:39.735850343Z "output_data_path": "results",
STDERR: 2024-08-30T04:27:39.735850343Z "answer_db_file": "/usr/local/bin/MIDI_validation_script/midi_1_1_answer_data_1.db",
STDERR: 2024-08-30T04:27:39.735862123Z "uid_mapping_file": "submission/MIDI-B-2024-VAL/mappings/uid_mapping.csv",
STDERR: 2024-08-30T04:27:39.735864351Z "patid_mapping_file": "submission/MIDI-B-2024-VAL/mappings/patient_id_mapping.csv",
STDERR: 2024-08-30T04:27:39.735872694Z "multiprocessing": "True",
STDERR: 2024-08-30T04:27:39.735872694Z "multiprocessing_cpus": "5",
STDERR: 2024-08-30T04:27:39.735876335Z "log_path": "logs",
STDERR: 2024-08-30T04:27:39.735880000Z "log_level": "info",
STDERR: 2024-08-30T04:27:39.735883694Z "report_series": "True"
STDERR: 2024-08-30T04:27:39.735887278Z }
STDERR: 2024-08-30T04:27:39.735890710Z }
STDERR: 2024-08-30T04:27:39.735897430Z 2024-08-30 04:27:39,022 - [INFO] - Report Generation Started
STDERR: 2024-08-30T04:27:39.735901063Z 2024-08-30 04:27:39,022 - [INFO] - Initialization Started
STDERR: 2024-08-30T04:27:39.735904673Z **2024-08-30 04:27:39,024 - [ERROR] - An unexpected error occurred: Execution failed on sql 'select * from validation_results': no such table: validation_results**
STDERR: 2024-08-30T04:27:39.735908413Z 2024-08-30 04:27:39,024 - [INFO] - Reports Generation Complete - Duration: (0, 1)
STDERR: 2024-08-30T04:27:39.735912065Z {
STDERR: 2024-08-30T04:27:39.735915441Z "config_file": {
STDERR: 2024-08-30T04:27:39.735918976Z "run_name": "MIDI_1_1_Testing",
STDERR: 2024-08-30T04:27:39.735922535Z "input_data_path": "submission",
STDERR: 2024-08-30T04:27:39.735926172Z "output_data_path": "results",
STDERR: 2024-08-30T04:27:39.735929780Z "answer_db_file": "/usr/local/bin/MIDI_validation_script/midi_1_1_answer_data_1.db",
STDERR: 2024-08-30T04:27:39.735933473Z "uid_mapping_file": "submission/MIDI-B-2024-VAL/mappings/uid_mapping.csv",
STDERR: 2024-08-30T04:27:39.735937146Z "patid_mapping_file": "submission/MIDI-B-2024-VAL/mappings/patient_id_mapping.csv",
STDERR: 2024-08-30T04:27:39.735940943Z "multiprocessing": "True",
STDERR: 2024-08-30T04:27:39.735944463Z "multiprocessing_cpus": "5",
STDERR: 2024-08-30T04:27:39.735948060Z "log_path": "logs",
STDERR: 2024-08-30T04:27:39.735951620Z "log_level": "info",
STDERR: 2024-08-30T04:27:39.735955239Z "report_series": "True"
STDERR: 2024-08-30T04:27:39.735960065Z }
STDERR: 2024-08-30T04:27:39.735965035Z }
STDERR: 2024-08-30T04:27:39.735979353Z [job test.cwl] Max memory used: 3284MiB
STDERR: 2024-08-30T04:27:39.735981684Z INFO:cwltool:[job test.cwl] Max memory used: 3284MiB
STDERR: 2024-08-30T04:27:39.735989051Z [job test.cwl] Job error:
STDERR: 2024-08-30T04:27:39.735995169Z ("Error collecting output for parameter 'discrepancy_internal':\nMIDI-B-De-identification-main/steps/test.cwl:61:7: Did not find output file with glob pattern: '['results/MIDI_1_1_Testing/discrepancy_report_internal.csv']'", {})
STDERR: 2024-08-30T04:27:39.736002872Z ERROR:cwltool:[job test.cwl] Job error:
STDERR: 2024-08-30T04:27:39.736002872Z ("Error collecting output for parameter 'discrepancy_internal':\nMIDI-B-De-identification-main/steps/test.cwl:61:7: Did not find output file with glob pattern: '['results/MIDI_1_1_Testing/discrepancy_report_internal.csv']'", {})
STDERR: 2024-08-30T04:27:39.736006819Z [job test.cwl] completed permanentFail
STDERR: 2024-08-30T04:27:39.736010304Z WARNING:cwltool:[job test.cwl] completed permanentFail
STDERR: 2024-08-30T04:27:39.736013781Z Traceback (most recent call last):
STDERR: 2024-08-30T04:27:39.736017274Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2024-08-30T04:27:39.736020914Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2024-08-30T04:27:39.736027831Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2024-08-30T04:27:39.736031541Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2024-08-30T04:27:39.736035045Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run
STDERR: 2024-08-30T04:27:39.736041589Z return self.run(fileStore)
STDERR: 2024-08-30T04:27:39.736045149Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2024-08-30T04:27:39.736048762Z raise cwltool.errors.WorkflowException(status)
STDERR: 2024-08-30T04:27:39.736052264Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2024-08-30T04:27:39.736055885Z ERROR:toil.worker:Exiting the worker because of a failed job on host 8a4c654925fc
Created by Michael Evans mdevans It might be an issue of compressing the file. We recommend two ways for the compression:
1. For MacOS, compress the folder using the command (not right-click to "compress ..."): tar --exclude="/." --exclude="/MACOSX" -cvzf NAMEOFTAR.tar /path/to/folder/you/want/to/archive*
2. For Windows or Linux, compress the folder using the command: zip -r foldername.zip path/to/the/folder Hi Linmin,
I tried another submission (file id: syn62707601) which had 5 de-id studies (each with <2 images) & mapping files in tar file which resulted in the same error.
Thanks,
Michael
Sorry to reply this late. Could you please provide the submission ID? I will look into the log for more details. BTW, if only submitting partial data, the performance will be impacted in the test phase (even though the score is still high in the validation phase).
Drop files to upload
Error: No such table: validation_reuslts page is loading…