Dear all,
I am trying to run the command:
```
medperf test run \
--offline --no-cache \
--demo_dataset_url synapse:syn53368010 \
--demo_dataset_hash "45b1a5827f34a40ea63c74db374fc872" \
-p ./prep_segmentation \
-e ./eval_segmentation \
-m ../brats2024/mlcube/mlcube.yaml
```
but I am having this error:
```
? Invalid resource input: mock_url. A Resource must be a url or
in the following format: ':'. Run
`medperf mlcube submit --help` for more details.
```
What could be the cause?
Thank you for your time.
Best regards,
André Ferreira
Created by André Filipe Sousa Ferreira ShadowTwin41 @ShadowTwin41 got it. Thanks! @abhijeetparida
Go to your project (left side, second icon), select all, and then the one you created for this task. On top left you see "Project SynID:", that's the synapseID. Where do I find the synapseID of my project? @vchung,
Yes, it was missing the argument --gpus=all!
Now everything is working.
I really appreciated your help, thank you very much! @ShadowTwin41 ,
Great question! You are correct - there has been an update since last year. To expose the GPUs to the medperf CLI, please include `--gpus=all` to the command:
```sh
medperf --gpus=all test run \
--offline --no-cache \
--demo_dataset_url synapse:syn53368010 \
--demo_dataset_hash "18534824fb01ca92fb4eed991cb19d0f9445821bfd935b58e4935ac276c62d2c" \
-p ./prep_segmentation \
-e ./eval_segmentation \
-m ??FILEPATH_TO_YOUR_MLCUBE
```
EDIT: fixed command order @vchung,
Thank you very much! Now it's working until it start performing the inferences (running my docker image).
For some reason, when I run the test "medperf test run" the docker does not have access to the GPU, but when I use "mlcube run --task infer" everything works well and the results appear correctly in the output folder.
Is there any configuration of the medperf test run that ensures that GPUs are visible?
Thank you very much! > I did run 'medperf auth synapse_login' before and the login was successful
Ahh, this is actually due to an update on the synapseclient side - thank you for bringing this to my attention. Instead of `medperf auth synapse_login`, can you try:
```
synapse config
```
I will update the tutorial with the same shortly.
> I believe this is from the BraTS2023 challenge and does not apply in this challenge, or the testing data is named distinctly from the validation and training data?
Good question. The testing datasets are distinct from the training and validation datasets, so the 3-digit timepoint note is correct. @vchung
Thank you very much!
Yes, indeed pushing the image to Synapse solved that problem. Maybe it should be included in that part that we need to do:
```
docker push docker.synapse.org/??SYNID/??IMAGE_NAME:??TAG
```
However, another problem appeared:
```
? There was an attempt to download resources from the Synapse platform, but couldn't find Synapse credentials.
Did you run 'medperf auth synapse_login' before?.
```
I did run 'medperf auth synapse_login' before and the login was successful as it says my full name and "? Done!". However, the error keeps showing. This is the full output, it might help:
```
MedPerf 0.1.0
Retrieving Model cube
2024-04-11 10:56:28 andreferreira-MS-7D98 mlcube.__main__[61343] INFO Configuring the MLCube for `docker` platform.
2024-04-11 10:56:28 andreferreira-MS-7D98 mlcube_docker.docker_run[61343] INFO Pulling container image.
1.0: Pulling from syn55066325/faking_it_docker_low_disk
Digest: sha256:dc1aa3be2b6c9ddea8687ae82d13608fcbb784a55511e7a86269c566cb2593bd
Status: Image is up to date for docker.synapse.org/syn55066325/faking_it_docker_low_disk:1.0
docker.synapse.org/syn55066325/faking_it_docker_low_disk:1.0
What's Next?
1. Sign in to your Docker account ? docker login
2. View a summary of image vulnerabilities and recommendations ? docker scout quickview docker.synapse.org/syn55066325/faking_it_docker_low_disk:1.0
2024-04-11 10:56:30 andreferreira-MS-7D98 mlcube.__main__[61343] INFO The MLCube has been successfully configured for `docker` platform.
> Model cube download complete
Retrieving Evaluator cube
2024-04-11 10:56:31 andreferreira-MS-7D98 mlcube.__main__[61526] INFO Configuring the MLCube for `docker` platform.
2024-04-11 10:56:31 andreferreira-MS-7D98 mlcube_docker.docker_run[61526] INFO Pulling container image.
0.0.0: Pulling from mlcommons/brats2023-mocked-metrics
Digest: sha256:58177b2cd6cd81a6fd7dc3f48919005b9d8dbbaa6844c3ead6f10635c13ca6c9
Status: Image is up to date for mlcommons/brats2023-mocked-metrics:0.0.0
docker.io/mlcommons/brats2023-mocked-metrics:0.0.0
What's Next?
1. Sign in to your Docker account ? docker login
2. View a summary of image vulnerabilities and recommendations ? docker scout quickview mlcommons/brats2023-mocked-metrics:0.0.0
2024-04-11 10:56:33 andreferreira-MS-7D98 mlcube.__main__[61526] INFO The MLCube has been successfully configured for `docker` platform.
> Evaluator cube download complete
? There was an attempt to download resources from the Synapse platform, but couldn't find Synapse credentials.
Did you run 'medperf auth synapse_login' before?.
```
I have another question, in the submission tutorial (https://www.synapse.org/#!Synapse:syn52939291/wiki/626233) is a note saying "Note that the input filenames now include a 3-digit timepoint. Timepoints are required in your prediction filenames if they are present in the input dataset.". I believe this is from the BraTS2023 challenge and does not apply in this challenge, or the testing data is named distinctly from the validation and training data?
Thank you very much! @ShadowTwin41,
Thank you for providing your steps. I will reach out to MedPerf for possible solutions - sorry again for the inconvenience!
In the meantime, if you were to push your image to Synapse, would that resolve the error? @vchung,
I think the tag is correct. In the error message it says "(docker=docker, image=docker.synapse.org/syn51713565/faking_it_docker_low_disk:1.00)" where 1.00 is the tag, so I believe that is correct.
Doing docker login docker.synapse.org didn't solve the problem. Now I'm getting:
```
2024-04-10 21:45:01 andreferreira-MS-7D98 mlcube.__main__[10750] INFO Configuring the MLCube for `docker` platform.
2024-04-10 21:45:01 andreferreira-MS-7D98 mlcube_docker.docker_run[10750] INFO Pulling container image.
Error response from daemon: manifest for docker.synapse.org/syn51713565/faking_it_docker_low_disk:1.0 not found: manifest unknown: manifest unknown
2024-04-10 21:45:03 andreferreira-MS-7D98 mlcube.__main__[10750] ERROR DockerRun runner failed to configure MLCube. Error occurred while pulling docker image (docker=docker, image=docker.synapse.org/syn51713565/faking_it_docker_low_disk:1.0).
? There was an error while retrieving the MLCube image.
```
As far as I understand, the problem comes from the fact that medperf is trying to get this image from the registry, but this image is local, and it's not in any registry.
I tracked down the error and it happens when in line:
```
self._get_image_from_registry()
```
in the file:
```
medperf/cli/medperf/entities/cube.py
```
It seems like the following medperf tries to run
```
docker pull docker.synapse.org/syn51713565/faking_it_docker_low_disk:1.0
```
but since the image is just locally, it will not work.
Am I missing something?
I do the following steps:
**1. ** mlcube configure -Pdocker.build_strategy=always
which creates:
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.synapse.org/syn51713565/faking_it_docker_low_disk 1.00 3c003c9a0ab5 7 hours ago 19.5GB
**2** docker login docker.synapse.org
**3** medperf auth synapse_login
**4.** medperf test run --offline --no-cache --demo_dataset_url synapse:syn53368010 --demo_dataset_hash "920f00d146eadc10663d265c12176e978d1a001d64127adfd084c2bef308fb27" -p ./prep_segmentation -e ./eval_segmentation -m /home/shadowtwin/Desktop/AI_work/BraTS2024_GOAT/faking_it_brats2024/mlcube
the mlcube is the same used to create the docker image, and it contains:
```
name: faking_it_brats2024_goat
description: Participation for the BraTS2024 goat challenge
authors:
- {name: Andre Ferreira}
platform:
accelerator_count: 1
docker:
# Image name
image: docker.synapse.org/syn51713565/faking_it_docker_low_disk:1.0
# Docker build context relative to $MLCUBE_ROOT. Default is `build`.
build_context: "../project"
# Docker file name within docker build context, default is `Dockerfile`.
build_file: "Dockerfile"
gpu_args: --gpus=all --shm-size=2G
tasks:
infer:
# Computes predictions on input data
parameters:
inputs: {
data_path: data/,
#parameters_file: parameters.yaml,
#nnUNet_results: additional_files/checkpoints,
# Feel free to include other files required for inference.
# These files MUST go inside the additional_files path.
# e.g. model weights
# weights: additional_files/weights.pt,
}
outputs: {output_path: {type: directory, default: predictions}}
```
Thank you for your help!
@ShadowTwin41 ,
> The "FILEPATH_TO_YOUR_MLCUBE" is the path to the folder that contains the file "mlcube.yaml" correct?
Yes, that's correct!
> Also, do I need to run first
> ```
> mlcube configure -Pdocker.build_strategy=always
> ```
> before running the medperf test?
Yes, based on [this tutorial](https://docs.medperf.org/mlcubes/mlcube_models/#build-your-mlcube), I believe this command will build the Docker image and make the MLCube ready for use.
> I am getting the following error now
> ```
> ...
> ? There was an error while retrieving the MLCube image.
> ```
Based on this particular error message:
```
Error response from daemon: pull access denied for docker.synapse.org/syn51713565/faking_it_docker_low_disk,
repository does not exist or may require 'docker login': denied: requested access to the resource is denied
```
My first thoughts are:
1. there is a missing tag in your "mlcube.yaml" for `docker.image`. When the tag is omitted, Docker will use `latest` by default, but perhaps there is no `latest` tag for your Docker image? You can check with `docker images` and seeing what value is under the "TAG" column.
2. you need to sign in to the Synapse Docker registry first, with: `docker login docker.synapse.org`.
I'm not sure if these will help, so let me know~ @vchung,
Thank you very much for your help!
I still have some doubts:
The "FILEPATH_TO_YOUR_MLCUBE" is the path to the folder that contains the file "mlcube.yaml" correct?
Also, do I need to run first
```
mlcube configure -Pdocker.build_strategy=always
```
before running the medperf test?
I am getting the following error now:
First I run the test
```
medperf test run --offline --no-cache --demo_dataset_url synapse:syn53368010 --demo_dataset_hash "920f00d146eadc10663d265c12176e978d1a001d64127adfd084c2bef308fb27" -p ./prep_segmentation -e ./eval_segmentation -m /home/shadowtwin/Desktop/AI_work/BraTS2024_GOAT/faking_it_brats2024/mlcube
```
and I get the outout:
```
MedPerf 0.1.0
Retrieving Model cube
2024-04-10 14:53:05 andreferreira-MS-7D98 mlcube.__main__[48686] INFO Configuring the MLCube for `docker` platform.
2024-04-10 14:53:05 andreferreira-MS-7D98 mlcube_docker.docker_run[48686] INFO Pulling container image.
Error response from daemon: pull access denied for docker.synapse.org/syn51713565/faking_it_docker_low_disk, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
2024-04-10 14:53:08 andreferreira-MS-7D98 mlcube.__main__[48686] ERROR DockerRun runner failed to configure MLCube. Error occurred while pulling docker image (docker=docker, image=docker.synapse.org/syn51713565/faking_it_docker_low_disk:1.00).
? There was an error while retrieving the MLCube image.
```
I can run the command:
```
mlcube run --task infer data_path=/home/shadowtwin/Desktop/AI_work/BraTS2024_GOAT/testing_data output_path=/home/shadowtwin/Desktop/AI_work/BraTS2024_GOAT/output_path
```
with success.
Could you help me?
@ShadowTwin41 ,
The issues have been resolved! IIRC, you were also a BraTS 2023 participant, so you may need to update your version of the MedPerf CLI. To update, please:
- Change to the `medperf` directory then pull the latest changes:
```sh
git pull
```
- Switch to the virtual environment (if you're using one), then re-install the MedPerf CLI from source:
```sh
pip install --force-reinstall -e ./cli
```
- Remove the previous (outdated) MedPerf configurations:
```sh
rm ~/mlcube.yaml && rm -rf ~/.medperf
```
- Set the logging level to "debug":
```sh
medperf profile set --loglevel=debug
```
The sample dataset has also been updated, so to test your MLCube, please use the following command:
```sh
medperf --gpus=all test run \
--offline --no-cache \
--demo_dataset_url synapse:syn53368010 \
--demo_dataset_hash "18534824fb01ca92fb4eed991cb19d0f9445821bfd935b58e4935ac276c62d2c" \
-p ./prep_segmentation \
-e ./eval_segmentation \
-m ??FILEPATH_TO_YOUR_MLCUBE
```
The [submission tutorial](https://www.synapse.org/#!Synapse:syn52939291/wiki/626233) has been updated accordingly, so if you prefer, you can read the notes there. Once again, thank you for letting us know of the issue and for your patience!
EDIT: update test command @vchung,
Thank you very much for your time! @ShadowTwin41 ,
MedPerf has resolved the original issue but unfortunately, another error has arisen and they are looking into it. Sorry for the wait! Dear @vchung,
Did you find the solution for this problem?
Thank you for your time.
Best regards,
André Ferreira @ShadowTwin41 -
Quick update: I was able to reproduce the error on my end as well, and have reached out to MedPerf regarding the issue. We will return with updates as soon as possible.
Thank you for your patience! @ShadowTwin41 -
Thank you bringing this up! I will take a look shortly, and will let you know~