@vchung My submission syn26235035 was shown as INVALID. The log file (9715220_log.txt) shows "ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)" and "RuntimeError: DataLoader worker (pid 18) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.". Can you please take a look?
This docker image runs fine on my computer with a 1080Ti GPU, using about 7 GB GPU memory and 12 GB RAM. Thanks.
Created by James Yuan deepX_MS @conner and team --
I brought up your request for a late submission with the Challenge Organizers and they are in agreement to accept your model for the final run against the test data. Please provide us with the image name and tag that you would like to be considered as your final submission, and I will add it to the queue for the final test data. @conner ,
I have forwarded your request to the Challenge Organizers! @vchung
Thanks for your kind help. It works when we change num_worker=2 to num_workers=0. However, we missed the deadline. Is there any chance to resubmit the docker image? The former 2 submissions were both invalid because of the wrong setting of num_workers. Hi @conner ,
Thank you for providing your details. The default shared memory space (shm-size) when running Docker containers is 64Mb rather than 61Gb. Fellow participants who encountered your error were able to fix it by setting num_workers=0 rather than num_numbers=2. Hi @vchung
We meet the same trouble during the final submitting.
Our submission ID is `9716414`.
The log file `9716415_log.txt` shows `ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).`
However, this docker runs normally on the local device. (We use NVIDIA RTX2080Ti GPU (11GB) and set `-c` to 8 and `--shm-size` to 61G (Actually used less than 10 GB). That is the shared memory is set to 61GB which should meet the hardware requirement(https://www.synapse.org/#!Synapse:syn25829067/discussion/threadId=8406&replyId=25942)) The consuming time is about 74 s per patient during inference locally.
We guess this problem is caused by the configuration of num_workers. We set `num_workers=2` for each model. Could you help us solve this problem or if we could have another chance to modify the configuration and resubmit it?
I'd appreciate it if you can help us or provide more information because the information of log file is limited.
Thank you.
Thanks! That fixes my memory issue too. @zjiang please try to set num_workers=0 . It works for me. @vchung Thanks for the information. This issue was resolved by setting num_workers=0! I was able to successfully test my docker online and obtain scores, thank you! Hi @deepX ,
Apologies for the confusion. The message received in the email is just a general message to indicate that prediction files were not generated, either due to exceeding the time limit or other. The next best step to find the reason why was, as you have done, to check the log file.
Another fellow participant had encountered your error, and he provided a workaround [here](https://www.synapse.org/#!Synapse:syn25829067/discussion/threadId=8392). Maybe this can help? Otherwise, please let us know the amount of shared memory size your model requires. Hi deepX,
Thank you for posting this issue. We've got a similar issue, I post the issue and some possible reasons on a new thread.
Best,
Zjiang It took about 120 seconds to complete one case on my local computer. , but from the email I received, it shows :
Your submission (id: 9716219) is invalid, below are the invalid reasons:
No *.nii.gz files found; please check whether running the Docker container locally will result in a NIfTI file within the time constaint.
Drop files to upload
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm) page is loading…