Hi there,
We are working on evaluating the validation dataset, our single image shape is [240, 240, 155], however we encountered Memory Error after we submitted validation files.
Please find the detailed error messages attached below.
Thanks for helping.
NGresearch.
STDERR: 2021-08-03T09:07:07.088120326Z [22] Failed to execute script Hausdorff95
STDERR: 2021-08-03T09:07:07.088122539Z Traceback (most recent call last):
STDERR: 2021-08-03T09:07:07.088124663Z File "Hausdorff95.py", line 34, in
STDERR: 2021-08-03T09:07:07.088127138Z File "site-packages/medpy/metric/binary.py", line 396, in hd95
STDERR: 2021-08-03T09:07:07.088129479Z File "site-packages/medpy/metric/binary.py", line 1224, in __surface_distances
STDERR: 2021-08-03T09:07:07.088131803Z File "site-packages/scipy/ndimage/morphology.py", line 2202, in distance_transform_edt
STDERR: 2021-08-03T09:07:07.088134086Z MemoryError: Unable to allocate 204. MiB for an array with shape (3, 155, 240, 240) and data type float64
STDERR: 2021-08-03T09:07:07.088136376Z terminate called after throwing an instance of 'std::out_of_range'
STDERR: 2021-08-03T09:07:07.088138683Z what(): basic_string::erase: __pos (which is 18446744073709551615) > this->size() (which is 0)
STDERR: 2021-08-03T09:07:07.088143169Z /usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py:120: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
STDERR: 2021-08-03T09:07:07.088145781Z warnings.warn(msg)
STDERR: 2021-08-03T09:07:07.088147907Z Traceback (most recent call last):
STDERR: 2021-08-03T09:07:07.088150013Z File "/usr/local/bin/score.py", line 141, in
STDERR: 2021-08-03T09:07:07.088152304Z main()
STDERR: 2021-08-03T09:07:07.088154404Z File "/usr/local/bin/score.py", line 115, in main
STDERR: 2021-08-03T09:07:07.088156600Z results = score(preds, golds, args.captk_path)
STDERR: 2021-08-03T09:07:07.088158774Z File "/usr/local/bin/score.py", line 102, in score
STDERR: 2021-08-03T09:07:07.088160935Z run_captk(captk_path, pred, gold, tmp_output)
STDERR: 2021-08-03T09:07:07.088163045Z File "/usr/local/bin/score.py", line 63, in run_captk
STDERR: 2021-08-03T09:07:07.088165258Z subprocess.check_call(cmd)
STDERR: 2021-08-03T09:07:07.088167375Z File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
STDERR: 2021-08-03T09:07:07.088169637Z raise CalledProcessError(retcode, cmd)
STDERR: 2021-08-03T09:07:07.088171809Z subprocess.CalledProcessError: Command '['/work/CaPTk/bin/Utilities', '-i', 'RSNA_ASNR_MICCAI_BraTS2021_ValidationGT/BraTS2021_00001_seg.nii.gz', '-lsb', 'BraTS2021_00001.nii.gz', '-o', 'tmp.csv']' died with .
STDERR: 2021-08-03T09:07:07.088174472Z [job score.cwl] Max memory used: 954MiB
STDERR: 2021-08-03T09:07:07.088176984Z INFO:cwltool:[job score.cwl] Max memory used: 954MiB
STDERR: 2021-08-03T09:07:07.088179152Z [job score.cwl] Job error:
STDERR: 2021-08-03T09:07:07.088181295Z ("Error collecting output for parameter 'results':\nbrats-dream-challenge-infra-main/score.cwl:46:7: Did not find output file with glob pattern: '['results.json']'", {})
STDERR: 2021-08-03T09:07:07.088183869Z ERROR:cwltool:[job score.cwl] Job error:
STDERR: 2021-08-03T09:07:07.088185991Z ("Error collecting output for parameter 'results':\nbrats-dream-challenge-infra-main/score.cwl:46:7: Did not find output file with glob pattern: '['results.json']'", {})
STDERR: 2021-08-03T09:07:07.088188506Z [job score.cwl] completed permanentFail
STDERR: 2021-08-03T09:07:07.088190639Z WARNING:cwltool:[job score.cwl] completed permanentFail
STDERR: 2021-08-03T09:07:07.088192779Z Traceback (most recent call last):
STDERR: 2021-08-03T09:07:07.088194881Z File "/usr/local/lib/python3.6/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2021-08-03T09:07:07.088197171Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2021-08-03T09:07:07.088199399Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2021-08-03T09:07:07.088201675Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2021-08-03T09:07:07.088206294Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1329, in _run
STDERR: 2021-08-03T09:07:07.088208600Z return self.run(fileStore)
STDERR: 2021-08-03T09:07:07.088210772Z File "/usr/local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2021-08-03T09:07:07.088213047Z raise cwltool.errors.WorkflowException(status)
STDERR: 2021-08-03T09:07:07.088215180Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2021-08-03T09:07:07.088217341Z ERROR:toil.worker:Exiting the worker because of a failed job on host 3485a45736c9
STDERR: 2021-08-03T09:07:07.088219679Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/cde2a488-3ff3-4215-8c81-9fcbe3327a7f/brats-dream-challenge-infra-main/score.cwl' score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_cde2a488-3ff3-4215-8c81-9fcbe3327a7f_brats-dream-challenge-infra-main_score.cwl/instance-4c7kxvzh with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_cde2a488-3ff3-4215-8c81-9fcbe3327a7f_brats-dream-challenge-infra-main_score.cwl/instance-4c7kxvzh to 0
Created by NGresearch Hi @NGresearch,
The issue is sorted out now. I could verify that your submission is evaluated now.
Ujjwal. I get the same error. I upload the files as a single .zip archive. Every *.nii.gz file has a shape (240, 240, 155) and data type uint8. Yes. I did it the same way. Place all the *nii.gz in a single zip and submitted. Did you have any error (invalid submission) while submitting? Could I ask about your folder structure? I contained *.nii.gz files inside of a single zip and submitted that zip for evaluation. Is it correct? I am also having this error. Same error is being output when I tried submitting one of the given seg.nii.gz file. Same error here, files were saved as int16 but it seems they are loaded as float64?
Drop files to upload
Submission Evaluation INVALID due to MemoryError ----Failed to execute script Hausdorff95 page is loading…