Hi @vchung and everyone!
I tested my docker locally and it works normally.
But when I submitted it, and I got the error No *.nii.gz files found; please check whether running the Docker container locally will result in a NIfTI file within the time constraint.
I tried several times but it's still not worked.
Please help me.
Thank you.
Submission ID: 9716117
Created by Bair Tuchinov nsu_btr @nsu_btr ,
Thank you for the follow-up! We have made note of your final submission :) Dear @vchung
Final submission is 9716416 @nsu_btr ,
Sorry about that -- the submission end date has been resolved; please let me know if you are still not able to submit.
With respect to submission 9716260, I locally ran it on a machine with the same specs as the submission system, and received the following:
```
...
using precomputed Gaussian
prediction done
force_separate_z: None interpolation order: 1
no resampling necessary
inference done. Now waiting for the segmentation export to finish...
WARNING! Cannot run postprocessing because the postprocessing file is missing. Make sure to run consolidate_folds in the output folder of the model first!
The folder you need to run this in is /usr/local/bin/nnUNet_trained_models/nnUNet/3d_fullres/Task300_BraTS2021/nnUNetTrainerV2BraTSRegions_DA3__nnUNetPlansv2.1
Traceback (most recent call last):
File "/usr/local/bin/brats_script.py", line 289, in
main()
File "/usr/local/bin/brats_script.py", line 239, in main
remove = test(input, output)
File "/usr/local/bin/predict_DMF.py", line 140, in test
assert torch.cuda.is_available(), "Currently, we only support CUDA version"
AssertionError: Currently, we only support CUDA version
```
If it helps, the driver and CUDA versions on the instance are 470.57.02 and 11.4, respectively.
EDIT:
The model also took about 30 minutes to run on the machine:
```
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c41026311543 docker.synapse.org/syn26024301/nsu-model1 "python3 /usr/local/?" 46 minutes ago Exited (1) 15 minutes ago nsu_btr
``` Dear @vchung ,
From the last email:
"Due to the technical difficulties faced earlier during the final Docker submission phase, we have decided to extend the original September 23rd 23:59 EST deadline. You now have until September 24th 23:59 EST to submit"
When I tried the submit new docker, the result "It is currently outside of the time range allowed for submissions."
In my case, what I need to do?
1. Wait you checking on your local machine my submission ID 9716260
2. How i can submit today the new docker? @vchung
I tested the docker on CPU and it took about 5mins per case so I thought if I use a GPU to get a result, the time goes under 4mins per case.
When I tried to just get a output not a Docker image using our local computer, I used RTX3090(24Gb memory) and it took just 4~5 sec per case.
Is there any particular way to choose specific GPU number you provided when I test the Docker?
Cause in my code 'predict3d.py' there is a section to choose device which is GPU number.
@vchung
Im tested the docker on Titan X with 12 Gb (GPU memory). Time 2 min 46 second. @nsu_btr , @yoonseokchoi -
Apologies for the confusion. The message received in the email is just a general message to indicate that prediction files were not generated, either due to exceeding the time limit or other. The next best step to find the reason why was, as you have done, to check the log file.
As the log file suggests, the submitted models were taking longer than the time allowed (4 minutes + 2.5 minutes buffer time), so the workflow stopped running the submission. This resulted in no NIfTI file generated. About a month ago, an email was sent out with details regarding this phase:
> When you submit your docker image, we will run them against 3-5 cases from the validation data to make sure your containers run, then we will run them against the unseen test set. Your submissions will have a time limit of 4 minutes per case (About 26 hours total).
When you run your models on your local machine, how long did it take? What are your GPU specs? The submission system is currently using a Tesla V100 with 16 GiB vRAM (GPU memory). Me too.... I have no idea why this happen Dear @vchung
New ID submission 9716260.
I tested my new docker locally and it works normally.
But when I submitted it, and I got the error "No *.nii.gz files found; please check whether running the Docker container locally will result in a NIfTI file within the time constraint"
From the log file:
This worker has ended successfully, no errors to report
Warnings:
=========
Time limit of 390s reached for case 00001 - stopping submission...
Please check and help me.
Thank you!
Thank you again, @vchung!
@nsu_btr ,
`/output` has read-write permissions. If it helps any, the Docker run command used by the workflow is something like this:
```bash
docker run \
--rm \
--network=none \
--runtime="nvidia" \
-v /path/to/input:/input:ro \
-v /path/to/output:/output:rw \
```
where `/path/to/input` is one of the case folders, e.g. **BraTS2021_00001**.
Hope this helps! thank you, @vchung!
Does /output write-only directory? Dear @kondratevakate ,
To better help you, what is the ID of the submission you are asking for more details on? Dear @nsu_btr ,
I checked the [log file](https://www.synapse.org/#!Synapse:syn26226724) of submission 9716117; looks like your model is trying to rename a file in the mounted `/input` directory, which is read-only.
Hope this helps! + the same
Drop files to upload
No *.nii.gz files found; please check whether running the Docker container locally will result in a NIfTI file within the time constraint. page is loading…