# inpainting challenge
Please follow us on [Twitter](https://twitter.com/BraTS_inpaint) and take a look at the [challenge website](https://www.synapse.org/#!Synapse:syn51156910/wiki/622357).
# Announcements
** 29 September 2023**: Submitted papers can be published in the BrainLes proceedings - we will followup soon!
** 28 September 2023**: We have notified the top-3 teams with instructions now. Please check your inboxes!
**25 August 2023**: Reach out to us if you face trouble creating your ML cube submissions
**22 August 2023**: Deadline extension for ML cube submission until the 25th. More info [here](https://www.synapse.org/#!Synapse:syn51156910/discussion/threadId=10465)
**28 July 2023**: another deadline extension and monetary awards - more info [here](https://www.synapse.org/#!Synapse:syn51156910/discussion/threadId=10387)! New [schedule](https://www.synapse.org/#!Synapse:syn51156910/wiki/)
**20 July 2023**: deadlines extended! New challenge timeline [here](https://www.synapse.org/#!Synapse:syn51156910/wiki/)
**19 July 2023**: bugs fixed in the metrics [evaluation script](https://github.com/BraTS-inpainting/2023_challenge/blob/main/evaluation/evaluation_utils.py)
**11 July 2023**: the validation phase of the [inpainting challenge](https://www.synapse.org/#!Synapse:syn51156910/wiki/622357) has officially begun! Evaluate your model following the [tutorial](https://www.synapse.org/#!Synapse:syn51156910/wiki/622885)!
**5 June 2023**: The challenge is officially launched. Git repos and website are online. The manuscript will be updated soon.
**24 May 2023**: The challenge data is released with a soft launch.
# Questions and Answers
tba
Created by Florian Kofler neuronflow Dear participants,
We will contact MICCAI to remind them about the payment of prize money.
Further, please see Ujjwal's announcement regarding your papers:
https://www.synapse.org/#!Synapse:syn51156910/discussion/threadId=10699
Best,
Florian Hi @jazin ,
Sorry for the late response. I was distracted by the CVPR deadline.
Our metrics are meant to evaluate 3D volumes as you find them in the challenges.
The code is available here:
https://github.com/BraTS-inpainting/inpainting
You can install them as a package from pypi:
https://pypi.org/project/inpainting/
The documentation is, unfortunately a bit vague.
Our challenge features a hidden validation and test set. Participants uploaded their model predictions for the validation phase to get scored and an ML cube version of their algorithm to be evaluated on the test set.
A 20% subset of the training set should be partially comparable to our test set.
Hi there,
I hope this message finds you well. First and foremost, I would like to express my gratitude for coordinating this challenge; it's been an enriching experience.
I have a few inquiries regarding the dataset and methodology, and I would appreciate your clarification on the following points:
In my approach to 2-D image inpainting, I concatenate all slices to form a 3D volume. Is it acceptable to employ a 2D algorithm and subsequently construct a 3D volume for evaluation using metrics such as PSNR and SSIM?
While reviewing the results section, I noticed the announcement of various metrics. However, the ground truth for the validation dataset seems to be missing. Could you please confirm whether the validation and testing phases were executed by the organizers or the participants?
If I train my network using 80% of the provided training data and test it with the remaining 20%, would my results be comparable to the ones presented in your showcase?
I eagerly await your response and appreciate your time and assistance in addressing these queries.
Best Regards,
Javad @neuronflow
Hi. Sorry for the late reply. I have checked the inbox and replied the email. If there are any emails I did not reply, could you please send the emails to juexin.zhang@nottingham.edu.cn. Dear Ying Weng Team,
We have trouble contacting you. Please check your inboxes and get back to us asap.
@YingWengTeam @kechen @juexinzhang @yingweng All teams passing our rigorous paper review will be invited to publish their papers in the proceedings of the BrainLesion workshop [here last year's edition: https://link.springer.com/book/10.1007/978-3-031-33842-7 ]
We will reach out soon with instructions. We have notified the top-3 teams with instructions now. Please check your inboxes! Dear @juexinzhang we will contact the top-3 performing teams in the upcoming days. Dear Organizers,
Has the evaluation of all teams' MLCubes on the testing data been finalized? Dear participants,
do you face trouble creating your ML cube submissions? Please reach out to us now for assistance. @kechen0207 ,
Submission ID 9739682 has been invalidated and closed per your request. You may resubmit your Docker image and updated config tarball whenever you're ready. @vchung can you please do this? :) @neuronflow Since the deadline has been extended. We would like to submit a better mlcube package. Can you kindly invalidate our submission with ID 9739682. @kechen0207 no worries, up to 24h of GPU time should be no problem (for more, we can discuss), and I expect the GPUs in the testing setup to be slightly quicker than your 3080.
ANNOUNCEMENT:
** 22 August 2023 **: Deadline extension for ML cube submission until the 25th. More info [here](https://www.synapse.org/#!Synapse:syn51156910/discussion/threadId=10465)
@neuronflow regarding the 30 minutes mentioned by @juexinzhang. More specifically, it is run on 250 cases from the training set and the inference is run on a single GPU machine (Nvidia 3080 10G). For the testing set, as you mentioned 568 cases, it would be approximately slightly over 1 hour on a machine with 3080 10G. Also if you provide V100 or some GPU at a similar level, it would still be slightly over 1 hour. Is it acceptable? @juexinzhang on which hardware? 30min for a single case?
So predicting the full test set of 568 cases would take 284h? Hi! I'm wondering if there is a time limit for the inference of our model. Is less than 30 minutes reasonable for inference? Thanks for your comment @juexinzhang!
@khue if the problem has not been solved, please send us your operating system and `pip --version`.
EDIT: and `conda --version` @hasank can you please have a look? @khue
This problem is caused by Cython version >= 3.0, try
```
pip install cython<3.0.0
```
and
```
pip install --no-build-isolation pyyaml==5.4.1
```
. Hi @vchung,
It might be late now but I was unable to install `medperf` although I followed [this instruction](https://docs.medperf.org/getting_started/installation/) step by step.
I got this error when running the command `pip install -e ./cli`.
I might not be able to fix it in time, but I want to solve it for the future challenge.
Please help me!
```
Obtaining file:///Users/khueluu/Desktop/BraTS23/medperf/cli
Preparing metadata (setup.py) ... done
Collecting mlcube@ git+https://github.com/mlcommons/mlcube@fb371c960938b495e939bf38b161199d529cf912#subdirectory=mlcube (from medperf==0.1.0)
Cloning https://github.com/mlcommons/mlcube (to revision fb371c960938b495e939bf38b161199d529cf912) to /private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-install-ih66egi9/mlcube_a33b5882bee241c3a0fe8a45b8ced07e
Running command git clone --filter=blob:none --quiet https://github.com/mlcommons/mlcube /private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-install-ih66egi9/mlcube_a33b5882bee241c3a0fe8a45b8ced07e
Running command git rev-parse -q --verify 'sha^fb371c960938b495e939bf38b161199d529cf912'
Running command git fetch -q https://github.com/mlcommons/mlcube fb371c960938b495e939bf38b161199d529cf912
Running command git checkout -q fb371c960938b495e939bf38b161199d529cf912
Resolved https://github.com/mlcommons/mlcube to commit fb371c960938b495e939bf38b161199d529cf912
Preparing metadata (setup.py) ... done
Collecting mlcube-docker@ git+https://github.com/mlcommons/mlcube@fb371c960938b495e939bf38b161199d529cf912#subdirectory=runners/mlcube_docker (from medperf==0.1.0)
Cloning https://github.com/mlcommons/mlcube (to revision fb371c960938b495e939bf38b161199d529cf912) to /private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-install-ih66egi9/mlcube-docker_6d30877869fe42d6a264792152eac35c
Running command git clone --filter=blob:none --quiet https://github.com/mlcommons/mlcube /private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-install-ih66egi9/mlcube-docker_6d30877869fe42d6a264792152eac35c
Running command git rev-parse -q --verify 'sha^fb371c960938b495e939bf38b161199d529cf912'
Running command git fetch -q https://github.com/mlcommons/mlcube fb371c960938b495e939bf38b161199d529cf912
Running command git checkout -q fb371c960938b495e939bf38b161199d529cf912
Resolved https://github.com/mlcommons/mlcube to commit fb371c960938b495e939bf38b161199d529cf912
Preparing metadata (setup.py) ... done
Collecting mlcube-singularity@ git+https://github.com/mlcommons/mlcube@fb371c960938b495e939bf38b161199d529cf912#subdirectory=runners/mlcube_singularity (from medperf==0.1.0)
Cloning https://github.com/mlcommons/mlcube (to revision fb371c960938b495e939bf38b161199d529cf912) to /private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-install-ih66egi9/mlcube-singularity_79bd879567c947fa81b441cdc544147a
Running command git clone --filter=blob:none --quiet https://github.com/mlcommons/mlcube /private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-install-ih66egi9/mlcube-singularity_79bd879567c947fa81b441cdc544147a
Running command git rev-parse -q --verify 'sha^fb371c960938b495e939bf38b161199d529cf912'
Running command git fetch -q https://github.com/mlcommons/mlcube fb371c960938b495e939bf38b161199d529cf912
Running command git checkout -q fb371c960938b495e939bf38b161199d529cf912
Resolved https://github.com/mlcommons/mlcube to commit fb371c960938b495e939bf38b161199d529cf912
Preparing metadata (setup.py) ... done
Collecting typer~=0.6.0 (from medperf==0.1.0)
Downloading typer-0.6.1-py3-none-any.whl (38 kB)
Collecting rich~=12.5.0 (from medperf==0.1.0)
Downloading rich-12.5.1-py3-none-any.whl (235 kB)
????????????????????????????????????? 235.6/235.6 kB 846.1 kB/s eta 0:00:00
Collecting PyYAML<6,>=5.4 (from medperf==0.1.0)
Downloading PyYAML-5.4.1.tar.gz (175 kB)
????????????????????????????????????????? 175.1/175.1 kB 2.8 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
? exit code: 1
??> [62 lines of output]
/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
!!
********************************************************************************
The license_file parameter is deprecated, use license_files instead.
By 2023-Oct-30, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
********************************************************************************
!!
parsed = self.parsers.get(option_name, lambda x: x)(value)
running egg_info
writing lib3/PyYAML.egg-info/PKG-INFO
writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
writing top-level names to lib3/PyYAML.egg-info/top_level.txt
Traceback (most recent call last):
File "/Users/khueluu/anaconda3/envs/medperf-env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/Users/khueluu/anaconda3/envs/medperf-env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/khueluu/anaconda3/envs/medperf-env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "", line 271, in
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1233, in run_command
super().run_command(command)
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 319, in run
self.find_sources()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 327, in find_sources
mm.run()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 549, in run
self.add_defaults()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 587, in add_defaults
sdist.add_defaults(self)
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
super().add_defaults()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
self._add_defaults_ext()
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
self.filelist.extend(build_ext.get_source_files())
File "", line 201, in get_source_files
File "/private/var/folders/s5/bwng7ztn4r10vbl4dl51sp_w0000gn/T/pip-build-env-czx_o0f1/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
raise AttributeError(attr)
AttributeError: cython_sources
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
? exit code: 1
??> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
``` Thank you. The problem is fixed. @msadi002,
Your submissions have been invalidated. You may now resubmit as a team. Let us know if you experience any issues. could you please removing my previous submission. So, I can submit as a part of team.
Thank you @msadi002 ,
As noted in the Participation Overview (under "Submission"), once you have submitted as an individual, you may not thereafter submit as part of a team. Unfortunately, because you have already submitted to the inpainting queue as an individual, the submission system will no longer allow you to submit as part of a team. That being said, we can resolve this by invalidating and removing your previous submissions (n = 9). Let me know if you would like to go with this option.
Another alternative would be to submit as a team once the MLCube Submission Phase begins. For this, you must report the _team ID_ in your short paper submission rather than your user ID. That way, the proper submitting permissions are granted to the team.
Hope this helps! @vchung can you please have a look? Hi, I am getting this error while submitting my validation file to challenge:
I am submitting as an individual
I am submitting as part of a team
Select Team
Vision Lab ODU23
Register a different team
Contributors
@msadi002
Has a conflicting submission.
@waliafarzana
@Monibor
@atemt001
@kiftekha
The following Team members are ineligible to contribute to a Submission in the specified Evaluation at this time [3429716]
Thank you Thanks a lot! @lulululu ,
Thank you for your patience! The issue has been addressed and I have restarted your submission ID 9739041 - you should be getting notified shortly. If you are still experiencing issues, please let us know! @lulululu ,
Apologies for the inconvenience! I will look into the issue and try to resolve ASAP. @vchung @albrecht can you please have a look at this? :) Dear organizers,
We just received this error message with our last submission attempts:
```
...
STDERR: 2023-08-11T12:08:14.098465657Z OSError: [Errno 28] No space left on device: '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/37e11fd7-3642-454b-abcb-c6d467299fdc/node-24c1b727-393b-4e32-be20-a29b3445a76c-1cc402dd0e11d5ae18db04a6de87223d/tmpmfjvtdbc/4629d165-e3e8-4e47-a044-8fa625edc93f/ti0l7umyn/tmp-outacrzsqii/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip' -> '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/37e11fd7-3642-454b-abcb-c6d467299fdc/tmpwagoxaud/files/for-job/kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-mg8eajf0/file-t5ox5m0v/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip.d0e90438-c2b2-4066-b8cc-a27a1bad63f9.tmp.zip'
STDERR: 2023-08-11T12:08:14.098468744Z ERROR:toil.worker:Exiting the worker because of a failed job on host 4d7b3be85642
STDERR: 2023-08-11T12:08:14.098473324Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-mg8eajf0 with ID kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-mg8eajf0 to 0
STDERR: 2023-08-11T12:08:14.098476130Z <=========
```
If we understand this correctly then there is currently no space left for the evaluations on your server - could you have a look at this?
Thank you in advance! :) It is mandatory to submit the short summary paper to be eligible for the testing phase of the challenge. You can always OPT-OUT for the publication of the paper. @juexinzhang ,
As noted in the Participation Overview (under "Submission"), once you have submitted as an individual, you may not thereafter submit as part of a team. That being said \-- and @ujjwalbaid can confirm -- you may continue the testing phase as part of a team if you report the team ID in your short paper submission.
Hope that helps!
EDIT: fixing format
@ChunnanMiao
From our side, it is totally acceptable to benchmark a method submitted to MIA in the challenge. However, make sure your short-paper contains an adequate method description, so readers of the short paper can understand your method without reading the MIA submission (which they cannot access).
Also, make sure to align with MIA's submission rules.
@ujjwalbaid did I overlook something? We note that a short paper submission is required in order to be eligible for the next phase of the challenge, that is: evaluating your models against the unseen test cases.
The method we use has been submitted on Medical Image Analysis. Does it have to be a brand new paper when we submit the paper, and can we use the submitted content? @juexinzhang Sorry that you are facing technical issues. @vchung can you please have a look?
To clarify the rules:
We will accept one submission per team.
One person cannot be part of multiple teams.
A valid submission is defined as a short paper + algorithm as a (working!) ML cube. Hi. I have submitted my results several times as an individual in the validation phase, but I have found that I am not able to submit my results as part of my team. I would like to know if I will be able to submit my results as a team in the testing phase. Additionally, I would like to submit the short paper as a team. Is this in line with the rules? @msadi002 sorry, I overlooked your message. We follow the time schedule [here](https://www.synapse.org/#!Synapse:syn51156910/wiki/)
Update: Another deadline extension and monetary awards!
https://www.synapse.org/#!Synapse:syn51156910/discussion/threadId=10387
Happy inpainting! Dear
Could you please let us know the short paper due date ?
@kechen0207 ,
Apologies for the inconvenience! The bug has been addressed on our end, and your submissions (9737119, 9737122, 9737130) were re-run successfully earlier today. Your results are available on the [leaderboard here](https://www.synapse.org/#!Synapse:syn51156910/wiki/622878), in addition to having been sent to your registered email.
Let us know if you are still having issues! Hi,
I have followed the tutorial you mentioned.
My generated files have names as below:
```
BraTS-GLI-00006-000-t1n-inference.nii.gz
BraTS-GLI-00014-000-t1n-inference.nii.gz
BraTS-GLI-00016-001-t1n-inference.nii.gz
BraTS-GLI-00018-000-t1n-inference.nii.gz
BraTS-GLI-00021-000-t1n-inference.nii.gz
```
Also checked with CaPTk. Has a dimension of 240x240x155 and origin at [0, -239, 0].
@neuronflow @vchung Hi,
Thanks for your interest in our challenge. Can you confirm you are following [this tutorial](https://www.synapse.org/#!Synapse:syn51156910/wiki/622885)?
@vchung can you have a look, please?
Hi,
I have met a problem when submitting validation results for scoring. The official script for generating the results has been used and the file format matches the requirements of the official submission guidelines. However, I always got the result 'invalid'. Tried several times and always got failure logs as below. Is it the problem on my side or on the server side? Looks more likely a server side error ?@neuronflow
```
STDERR: 2023-07-23T15:47:09.058565730Z d891faf4fc9e 2023-07-23 15:47:09,058 MainThread INFO cwltool: Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/validation-phase.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/validation-phase.cwl'
STDERR: 2023-07-23T15:47:12.321766287Z d891faf4fc9e 2023-07-23 15:47:12,321 MainThread WARNING cwltool: Workflow checker warning:
STDERR: 2023-07-23T15:47:12.321788927Z brats2023-main/wf-inpainting/validation-phase.cwl:60:9: Source 'filepath' of type ["null", "File"]
STDERR: 2023-07-23T15:47:12.321791884Z may be incompatible
STDERR: 2023-07-23T15:47:12.321793995Z brats2023-main/wf-inpainting/validation-phase.cwl:94:9: with sink 'input_file' of type "File"
STDERR: 2023-07-23T15:47:12.321796284Z brats2023-main/wf-inpainting/validation-phase.cwl:60:9: Source 'filepath' of type ["null", "File"]
STDERR: 2023-07-23T15:47:12.321798371Z may be incompatible
STDERR: 2023-07-23T15:47:12.321800379Z brats2023-main/wf-inpainting/validation-phase.cwl:172:9: with sink 'input_file' of type "File"
STDERR: 2023-07-23T15:47:12.497314622Z d891faf4fc9e 2023-07-23 15:47:12,497 MainThread INFO toil: Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2023-07-23T15:47:12.969717518Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmp_mo8hddp/worker_log.txt
STDERR: 2023-07-23T15:47:13.138798675Z d891faf4fc9e 2023-07-23 15:47:13,138 MainThread INFO toil.leader: 0 jobs are running, 0 jobs are issued and waiting to run
STDERR: 2023-07-23T15:47:13.139380628Z d891faf4fc9e 2023-07-23 15:47:13,139 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-fy177x78 with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:13.139461163Z d891faf4fc9e 2023-07-23 15:47:13,139 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-96md_nuk with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:13.139535812Z d891faf4fc9e 2023-07-23 15:47:13,139 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-l2nran7f with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:13.139625936Z d891faf4fc9e 2023-07-23 15:47:13,139 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_set_permissions.cwl/instance-39037fkn with job batch system ID: 4 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:13.616676565Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpznob888d/worker_log.txt
STDERR: 2023-07-23T15:47:13.618025186Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmps9p7fv_z/worker_log.txt
STDERR: 2023-07-23T15:47:13.618437186Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpjnu7ix05/worker_log.txt
STDERR: 2023-07-23T15:47:13.619506240Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmph7o_i5er/worker_log.txt
STDERR: 2023-07-23T15:47:16.601746627Z d891faf4fc9e 2023-07-23 15:47:16,601 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_set_permissions.cwl/instance-39037fkn
STDERR: 2023-07-23T15:47:16.823679642Z d891faf4fc9e 2023-07-23 15:47:16,823 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-96md_nuk
STDERR: 2023-07-23T15:47:17.278093927Z d891faf4fc9e 2023-07-23 15:47:17,277 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-l2nran7f
STDERR: 2023-07-23T15:47:17.278583304Z d891faf4fc9e 2023-07-23 15:47:17,278 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_validate.cwl/instance-qg204ky5 with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:17.748615290Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmprc0mbcbv/worker_log.txt
STDERR: 2023-07-23T15:47:18.998677183Z d891faf4fc9e 2023-07-23 15:47:18,998 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_validate.cwl/instance-qg204ky5
STDERR: 2023-07-23T15:47:18.999311410Z d891faf4fc9e 2023-07-23 15:47:18,999 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-qp4wg386 with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:18.999387280Z d891faf4fc9e 2023-07-23 15:47:18,999 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-3a5dn3ff with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:19.470328003Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmp405qh3ek/worker_log.txt
STDERR: 2023-07-23T15:47:19.474551642Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmp1o03pidl/worker_log.txt
STDERR: 2023-07-23T15:47:22.379088033Z d891faf4fc9e 2023-07-23 15:47:22,378 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-3a5dn3ff
STDERR: 2023-07-23T15:47:22.530370245Z d891faf4fc9e 2023-07-23 15:47:22,530 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-qp4wg386
STDERR: 2023-07-23T15:47:22.530954862Z d891faf4fc9e 2023-07-23 15:47:22,530 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-vps9pv8a with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:47:22.998201360Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmp85rnqdir/worker_log.txt
STDERR: 2023-07-23T15:47:23.191109347Z d891faf4fc9e 2023-07-23 15:47:23,190 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-vps9pv8a
STDERR: 2023-07-23T15:48:02.086776269Z d891faf4fc9e 2023-07-23 15:48:02,086 Thread-47 WARNING toil.statsAndLogging: Got message from job at time 07-23-2023 15:48:02: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-eot4q3bt/cleanup/file-s1j9nlos/stream used 219.89% (2.2 GB [2363260928B] used, 1.0 GB [1074741824B] requested) at the end of its run.
STDERR: 2023-07-23T15:48:02.388428642Z d891faf4fc9e 2023-07-23 15:48:02,388 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-fy177x78
STDERR: 2023-07-23T15:48:02.388915282Z d891faf4fc9e 2023-07-23 15:48:02,388 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr with job batch system ID: 9 and cores: 1, disk: 1.0 G, and memory: 100.0 M
STDERR: 2023-07-23T15:48:02.900421425Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/worker_log.txt
STDERR: 2023-07-23T15:48:18.426355136Z d891faf4fc9e 2023-07-23 15:48:18,426 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr
STDERR: 2023-07-23T15:48:18.426788309Z d891faf4fc9e 2023-07-23 15:48:18,426 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr
STDERR: 2023-07-23T15:48:18.426959772Z d891faf4fc9e 2023-07-23 15:48:18,426 MainThread WARNING toil.leader: Log from job kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr follows:
STDERR: 2023-07-23T15:48:18.426965410Z =========>
STDERR: 2023-07-23T15:48:18.426967916Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2023-07-23T15:48:18.426970061Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2023-07-23T15:48:18.426972282Z [job score.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.
STDERR: 2023-07-23T15:48:18.426974541Z WARNING:cwltool:[job score.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.
STDERR: 2023-07-23T15:48:18.426977005Z [job score.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1$ docker \
STDERR: 2023-07-23T15:48:18.426979410Z run \
STDERR: 2023-07-23T15:48:18.426981444Z -i \
STDERR: 2023-07-23T15:48:18.426983500Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1,target=/gWnXqH \
STDERR: 2023-07-23T15:48:18.426992218Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js23r0om2v,target=/tmp \
STDERR: 2023-07-23T15:48:18.426995016Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpt7kj4_l5.tmp,target=/var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig,readonly \
STDERR: 2023-07-23T15:48:18.426997466Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpl47nigpc.tmp,target=/var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148,readonly \
STDERR: 2023-07-23T15:48:18.426999819Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpvpk9b6_8.tmp,target=/var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip,readonly \
STDERR: 2023-07-23T15:48:18.427002374Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpc30sksec.tmp,target=/var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip,readonly \
STDERR: 2023-07-23T15:48:18.427004826Z --workdir=/gWnXqH \
STDERR: 2023-07-23T15:48:18.427006854Z --read-only=true \
STDERR: 2023-07-23T15:48:18.427008749Z --user=0:0 \
STDERR: 2023-07-23T15:48:18.427010776Z --rm \
STDERR: 2023-07-23T15:48:18.427012598Z --env=TMPDIR=/tmp \
STDERR: 2023-07-23T15:48:18.427014457Z --env=HOME=/gWnXqH \
STDERR: 2023-07-23T15:48:18.427016407Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js1jnc595f/20230723154802-990343.cid \
STDERR: 2023-07-23T15:48:18.427018761Z docker.synapse.org/syn51156910/inpainting-evaluation:v2 \
STDERR: 2023-07-23T15:48:18.427020776Z /app/score.py \
STDERR: 2023-07-23T15:48:18.427022777Z --parent_id \
STDERR: 2023-07-23T15:48:18.427024661Z syn52137018 \
STDERR: 2023-07-23T15:48:18.427044101Z -s \
STDERR: 2023-07-23T15:48:18.427046517Z /var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig \
STDERR: 2023-07-23T15:48:18.427048713Z -p \
STDERR: 2023-07-23T15:48:18.427050593Z /var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148 \
STDERR: 2023-07-23T15:48:18.427055602Z -g \
STDERR: 2023-07-23T15:48:18.427057587Z /var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip \
STDERR: 2023-07-23T15:48:18.427059733Z -m \
STDERR: 2023-07-23T15:48:18.427061997Z /var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip \
STDERR: 2023-07-23T15:48:18.427064078Z -o \
STDERR: 2023-07-23T15:48:18.427065965Z results.json
STDERR: 2023-07-23T15:48:18.427067870Z INFO:cwltool:[job score.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1$ docker \
STDERR: 2023-07-23T15:48:18.427070223Z run \
STDERR: 2023-07-23T15:48:18.427072228Z -i \
STDERR: 2023-07-23T15:48:18.427074156Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1,target=/gWnXqH \
STDERR: 2023-07-23T15:48:18.427076570Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js23r0om2v,target=/tmp \
STDERR: 2023-07-23T15:48:18.427078877Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpt7kj4_l5.tmp,target=/var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig,readonly \
STDERR: 2023-07-23T15:48:18.427081209Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpl47nigpc.tmp,target=/var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148,readonly \
STDERR: 2023-07-23T15:48:18.427085244Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpvpk9b6_8.tmp,target=/var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip,readonly \
STDERR: 2023-07-23T15:48:18.427088063Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpc30sksec.tmp,target=/var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip,readonly \
STDERR: 2023-07-23T15:48:18.427090595Z --workdir=/gWnXqH \
STDERR: 2023-07-23T15:48:18.427092519Z --read-only=true \
STDERR: 2023-07-23T15:48:18.427096979Z --user=0:0 \
STDERR: 2023-07-23T15:48:18.427098974Z --rm \
STDERR: 2023-07-23T15:48:18.427100923Z --env=TMPDIR=/tmp \
STDERR: 2023-07-23T15:48:18.427102788Z --env=HOME=/gWnXqH \
STDERR: 2023-07-23T15:48:18.427104732Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js1jnc595f/20230723154802-990343.cid \
STDERR: 2023-07-23T15:48:18.427106965Z docker.synapse.org/syn51156910/inpainting-evaluation:v2 \
STDERR: 2023-07-23T15:48:18.427108958Z /app/score.py \
STDERR: 2023-07-23T15:48:18.427110854Z --parent_id \
STDERR: 2023-07-23T15:48:18.427112804Z syn52137018 \
STDERR: 2023-07-23T15:48:18.427114703Z -s \
STDERR: 2023-07-23T15:48:18.427116654Z /var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig \
STDERR: 2023-07-23T15:48:18.427118547Z -p \
STDERR: 2023-07-23T15:48:18.427120353Z /var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148 \
STDERR: 2023-07-23T15:48:18.427122247Z -g \
STDERR: 2023-07-23T15:48:18.427124099Z /var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip \
STDERR: 2023-07-23T15:48:18.427126103Z -m \
STDERR: 2023-07-23T15:48:18.427127959Z /var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip \
STDERR: 2023-07-23T15:48:18.427130017Z -o \
STDERR: 2023-07-23T15:48:18.427131827Z results.json
STDERR: 2023-07-23T15:48:18.427133650Z [38;5;6m [38;5;5m15:48:16.01 [0m
STDERR: 2023-07-23T15:48:18.427135633Z [38;5;6m [38;5;5m15:48:16.01 [0m[1mWelcome to the Bitnami pytorch container[0m
STDERR: 2023-07-23T15:48:18.427137687Z [38;5;6m [38;5;5m15:48:16.01 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
STDERR: 2023-07-23T15:48:18.427139833Z [38;5;6m [38;5;5m15:48:16.01 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
STDERR: 2023-07-23T15:48:18.427142052Z [38;5;6m [38;5;5m15:48:16.01 [0m
STDERR: 2023-07-23T15:48:18.427144008Z
STDERR: 2023-07-23T15:48:18.427145865Z Traceback (most recent call last):
STDERR: 2023-07-23T15:48:18.427147766Z File "/app/score.py", line 72, in
STDERR: 2023-07-23T15:48:18.427150034Z pred_lst: list[str], label: str) -> pd.DataFrame:
STDERR: 2023-07-23T15:48:18.427152012Z TypeError: 'type' object is not subscriptable
STDERR: 2023-07-23T15:48:18.427153885Z [job score.cwl] Max memory used: 0MiB
STDERR: 2023-07-23T15:48:18.427155763Z INFO:cwltool:[job score.cwl] Max memory used: 0MiB
STDERR: 2023-07-23T15:48:18.427159985Z [job score.cwl] Job error:
STDERR: 2023-07-23T15:48:18.427162014Z ("Error collecting output for parameter 'results':\nbrats2023-main/wf-inpainting/steps/score.cwl:27:5: Did not find output file with glob pattern: '['results.json']'", {})
STDERR: 2023-07-23T15:48:18.427164346Z ERROR:cwltool:[job score.cwl] Job error:
STDERR: 2023-07-23T15:48:18.427166282Z ("Error collecting output for parameter 'results':\nbrats2023-main/wf-inpainting/steps/score.cwl:27:5: Did not find output file with glob pattern: '['results.json']'", {})
STDERR: 2023-07-23T15:48:18.427168492Z [job score.cwl] completed permanentFail
STDERR: 2023-07-23T15:48:18.427170355Z WARNING:cwltool:[job score.cwl] completed permanentFail
STDERR: 2023-07-23T15:48:18.427172309Z WARNING:toil.fileStores.abstractFileStore:LOG-TO-MASTER: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-eot4q3bt/cleanup/file-cgdgw6uc/stream used 221.51% (2.2 GB [2380627968B] used, 1.0 GB [1074741824B] requested) at the end of its run.
STDERR: 2023-07-23T15:48:18.427174713Z Traceback (most recent call last):
STDERR: 2023-07-23T15:48:18.427176643Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2023-07-23T15:48:18.427178712Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2023-07-23T15:48:18.427180765Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2023-07-23T15:48:18.427182808Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2023-07-23T15:48:18.427184704Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run
STDERR: 2023-07-23T15:48:18.427186757Z return self.run(fileStore)
STDERR: 2023-07-23T15:48:18.427188630Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2023-07-23T15:48:18.427190692Z raise cwltool.errors.WorkflowException(status)
STDERR: 2023-07-23T15:48:18.427193845Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2023-07-23T15:48:18.427195910Z ERROR:toil.worker:Exiting the worker because of a failed job on host d891faf4fc9e
STDERR: 2023-07-23T15:48:18.427201190Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr to 0
STDERR: 2023-07-23T15:48:18.427204228Z <=========
STDERR: 2023-07-23T15:48:18.435107767Z d891faf4fc9e 2023-07-23 15:48:18,434 MainThread WARNING toil.leader: Job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr is completely failed
STDERR: 2023-07-23T15:48:31.628053860Z d891faf4fc9e 2023-07-23 15:48:31,627 MainThread INFO toil.leader: Finished toil run with 9 failed jobs.
STDERR: 2023-07-23T15:48:31.628078659Z d891faf4fc9e 2023-07-23 15:48:31,627 MainThread INFO toil.leader: Failed jobs at end of the run: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-l2nran7f 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-vps9pv8a 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-96md_nuk 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_validate.cwl/instance-qg204ky5 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-qp4wg386 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-3a5dn3ff 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-fy177x78 'CWLWorkflow' kind-CWLWorkflow/instance-eot4q3bt
STDERR: 2023-07-23T15:48:31.644076258Z Traceback (most recent call last):
STDERR: 2023-07-23T15:48:31.644096091Z File "/usr/local/bin/toil-cwl-runner", line 8, in
STDERR: 2023-07-23T15:48:31.644100314Z sys.exit(main())
STDERR: 2023-07-23T15:48:31.644103342Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 1691, in main
STDERR: 2023-07-23T15:48:31.644107142Z outobj = toil.start(wf1)
STDERR: 2023-07-23T15:48:31.644110493Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 829, in start
STDERR: 2023-07-23T15:48:31.644113697Z return self._runMainLoop(rootJobGraph)
STDERR: 2023-07-23T15:48:31.644116863Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 1115, in _runMainLoop
STDERR: 2023-07-23T15:48:31.644120597Z return Leader(config=self.config,
STDERR: 2023-07-23T15:48:31.644135617Z File "/usr/local/lib/python3.8/site-packages/toil/leader.py", line 269, in run
STDERR: 2023-07-23T15:48:31.644138834Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
STDERR: 2023-07-23T15:48:31.644143519Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/tmp53mog6m4' contains 9 failed jobs: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-l2nran7f, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-vps9pv8a, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-96md_nuk, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_validate.cwl/instance-qg204ky5, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-qp4wg386, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-3a5dn3ff, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-fy177x78, 'CWLWorkflow' kind-CWLWorkflow/instance-eot4q3bt
STDERR: 2023-07-23T15:48:31.644152864Z Log from job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr follows:
STDERR: 2023-07-23T15:48:31.644156564Z =========>
STDERR: 2023-07-23T15:48:31.644159878Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
STDERR: 2023-07-23T15:48:31.644163218Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
STDERR: 2023-07-23T15:48:31.644166733Z [job score.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.
STDERR: 2023-07-23T15:48:31.644170320Z WARNING:cwltool:[job score.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.
STDERR: 2023-07-23T15:48:31.644177954Z [job score.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1$ docker \
STDERR: 2023-07-23T15:48:31.644182043Z run \
STDERR: 2023-07-23T15:48:31.644185273Z -i \
STDERR: 2023-07-23T15:48:31.644188540Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1,target=/gWnXqH \
STDERR: 2023-07-23T15:48:31.644192597Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js23r0om2v,target=/tmp \
STDERR: 2023-07-23T15:48:31.644196705Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpt7kj4_l5.tmp,target=/var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig,readonly \
STDERR: 2023-07-23T15:48:31.644201185Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpl47nigpc.tmp,target=/var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148,readonly \
STDERR: 2023-07-23T15:48:31.644205317Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpvpk9b6_8.tmp,target=/var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip,readonly \
STDERR: 2023-07-23T15:48:31.644210158Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpc30sksec.tmp,target=/var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip,readonly \
STDERR: 2023-07-23T15:48:31.644214380Z --workdir=/gWnXqH \
STDERR: 2023-07-23T15:48:31.644217333Z --read-only=true \
STDERR: 2023-07-23T15:48:31.644220528Z --user=0:0 \
STDERR: 2023-07-23T15:48:31.644223767Z --rm \
STDERR: 2023-07-23T15:48:31.644228883Z --env=TMPDIR=/tmp \
STDERR: 2023-07-23T15:48:31.644232068Z --env=HOME=/gWnXqH \
STDERR: 2023-07-23T15:48:31.644235571Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js1jnc595f/20230723154802-990343.cid \
STDERR: 2023-07-23T15:48:31.644243590Z docker.synapse.org/syn51156910/inpainting-evaluation:v2 \
STDERR: 2023-07-23T15:48:31.644246952Z /app/score.py \
STDERR: 2023-07-23T15:48:31.644249940Z --parent_id \
STDERR: 2023-07-23T15:48:31.644253191Z syn52137018 \
STDERR: 2023-07-23T15:48:31.644256471Z -s \
STDERR: 2023-07-23T15:48:31.644260048Z /var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig \
STDERR: 2023-07-23T15:48:31.644263309Z -p \
STDERR: 2023-07-23T15:48:31.644266357Z /var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148 \
STDERR: 2023-07-23T15:48:31.644269603Z -g \
STDERR: 2023-07-23T15:48:31.644272813Z /var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip \
STDERR: 2023-07-23T15:48:31.644276415Z -m \
STDERR: 2023-07-23T15:48:31.644279638Z /var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip \
STDERR: 2023-07-23T15:48:31.644282737Z -o \
STDERR: 2023-07-23T15:48:31.644285568Z results.json
STDERR: 2023-07-23T15:48:31.644288811Z INFO:cwltool:[job score.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1$ docker \
STDERR: 2023-07-23T15:48:31.644292826Z run \
STDERR: 2023-07-23T15:48:31.644296326Z -i \
STDERR: 2023-07-23T15:48:31.644299393Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tc8s8u5_2/tmp-out0r6j9lg1,target=/gWnXqH \
STDERR: 2023-07-23T15:48:31.644303641Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js23r0om2v,target=/tmp \
STDERR: 2023-07-23T15:48:31.644307819Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpt7kj4_l5.tmp,target=/var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig,readonly \
STDERR: 2023-07-23T15:48:31.644311834Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpl47nigpc.tmp,target=/var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148,readonly \
STDERR: 2023-07-23T15:48:31.644315930Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpvpk9b6_8.tmp,target=/var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip,readonly \
STDERR: 2023-07-23T15:48:31.644324040Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tmpc30sksec.tmp,target=/var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip,readonly \
STDERR: 2023-07-23T15:48:31.644328537Z --workdir=/gWnXqH \
STDERR: 2023-07-23T15:48:31.644331833Z --read-only=true \
STDERR: 2023-07-23T15:48:31.644335211Z --user=0:0 \
STDERR: 2023-07-23T15:48:31.644338533Z --rm \
STDERR: 2023-07-23T15:48:31.644341537Z --env=TMPDIR=/tmp \
STDERR: 2023-07-23T15:48:31.644345320Z --env=HOME=/gWnXqH \
STDERR: 2023-07-23T15:48:31.644348839Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/node-cab0e43a-c14f-4174-a3d6-06a5a30d7ba5-1cc402dd0e11d5ae18db04a6de87223d/tmpgmk1fd9a/7b53e899-8bd6-4914-ab4e-1fa398b6dfa9/tcks_k0js1jnc595f/20230723154802-990343.cid \
STDERR: 2023-07-23T15:48:31.644352818Z docker.synapse.org/syn51156910/inpainting-evaluation:v2 \
STDERR: 2023-07-23T15:48:31.644356209Z /app/score.py \
STDERR: 2023-07-23T15:48:31.644360123Z --parent_id \
STDERR: 2023-07-23T15:48:31.644363219Z syn52137018 \
STDERR: 2023-07-23T15:48:31.644366549Z -s \
STDERR: 2023-07-23T15:48:31.644369826Z /var/lib/cwl/stg5d798126-62c5-43aa-a844-e5d6797504ec/.synapseConfig \
STDERR: 2023-07-23T15:48:31.644373066Z -p \
STDERR: 2023-07-23T15:48:31.644376616Z /var/lib/cwl/stgd4e21fc8-acfa-49a1-a4bb-031c87ca2d12/submission-9737148 \
STDERR: 2023-07-23T15:48:31.644380456Z -g \
STDERR: 2023-07-23T15:48:31.644384040Z /var/lib/cwl/stg2655a38a-74f4-46d0-a698-da0a383a0eb0/ASNR-MICCAI-BraTS2023-GLI-Challenge-ValidationData.zip \
STDERR: 2023-07-23T15:48:31.644387754Z -m \
STDERR: 2023-07-23T15:48:31.644391103Z /var/lib/cwl/stg92b74c96-77f6-4cd1-a77d-491d1d059697/ASNR-MICCAI-BraTS2023-Local-Synthesis-Challenge-Validation-Server-Minimal.zip \
STDERR: 2023-07-23T15:48:31.644394446Z -o \
STDERR: 2023-07-23T15:48:31.644397893Z results.json
STDERR: 2023-07-23T15:48:31.644401073Z [38;5;6m [38;5;5m15:48:16.01 [0m
STDERR: 2023-07-23T15:48:31.644404855Z [38;5;6m [38;5;5m15:48:16.01 [0m[1mWelcome to the Bitnami pytorch container[0m
STDERR: 2023-07-23T15:48:31.644408811Z [38;5;6m [38;5;5m15:48:16.01 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
STDERR: 2023-07-23T15:48:31.644413057Z [38;5;6m [38;5;5m15:48:16.01 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
STDERR: 2023-07-23T15:48:31.644420847Z [38;5;6m [38;5;5m15:48:16.01 [0m
STDERR: 2023-07-23T15:48:31.644424506Z
STDERR: 2023-07-23T15:48:31.644427811Z Traceback (most recent call last):
STDERR: 2023-07-23T15:48:31.644431135Z File "/app/score.py", line 72, in
STDERR: 2023-07-23T15:48:31.644434756Z pred_lst: list[str], label: str) -> pd.DataFrame:
STDERR: 2023-07-23T15:48:31.644438442Z TypeError: 'type' object is not subscriptable
STDERR: 2023-07-23T15:48:31.644442394Z [job score.cwl] Max memory used: 0MiB
STDERR: 2023-07-23T15:48:31.644445715Z INFO:cwltool:[job score.cwl] Max memory used: 0MiB
STDERR: 2023-07-23T15:48:31.644449319Z [job score.cwl] Job error:
STDERR: 2023-07-23T15:48:31.644452471Z ("Error collecting output for parameter 'results':\nbrats2023-main/wf-inpainting/steps/score.cwl:27:5: Did not find output file with glob pattern: '['results.json']'", {})
STDERR: 2023-07-23T15:48:31.644456248Z ERROR:cwltool:[job score.cwl] Job error:
STDERR: 2023-07-23T15:48:31.644459687Z ("Error collecting output for parameter 'results':\nbrats2023-main/wf-inpainting/steps/score.cwl:27:5: Did not find output file with glob pattern: '['results.json']'", {})
STDERR: 2023-07-23T15:48:31.644463690Z [job score.cwl] completed permanentFail
STDERR: 2023-07-23T15:48:31.644467423Z WARNING:cwltool:[job score.cwl] completed permanentFail
STDERR: 2023-07-23T15:48:31.644471933Z WARNING:toil.fileStores.abstractFileStore:LOG-TO-MASTER: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-eot4q3bt/cleanup/file-cgdgw6uc/stream used 221.51% (2.2 GB [2380627968B] used, 1.0 GB [1074741824B] requested) at the end of its run.
STDERR: 2023-07-23T15:48:31.644476662Z Traceback (most recent call last):
STDERR: 2023-07-23T15:48:31.644479920Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript
STDERR: 2023-07-23T15:48:31.644483461Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
STDERR: 2023-07-23T15:48:31.644487164Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner
STDERR: 2023-07-23T15:48:31.644491063Z returnValues = self._run(jobGraph, fileStore)
STDERR: 2023-07-23T15:48:31.644494664Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run
STDERR: 2023-07-23T15:48:31.644498224Z return self.run(fileStore)
STDERR: 2023-07-23T15:48:31.644501600Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run
STDERR: 2023-07-23T15:48:31.644505371Z raise cwltool.errors.WorkflowException(status)
STDERR: 2023-07-23T15:48:31.644508522Z cwltool.errors.WorkflowException: permanentFail
STDERR: 2023-07-23T15:48:31.644511966Z ERROR:toil.worker:Exiting the worker because of a failed job on host d891faf4fc9e
STDERR: 2023-07-23T15:48:31.644515546Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f929daa5-e3a9-452b-ba3b-1c72e2e0b722/brats2023-main/wf-inpainting/steps/score.cwl' /app/score.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_f929daa5-e3a9-452b-ba3b-1c72e2e0b722_brats2023-main_wf-inpainting_steps_score.cwl/instance-3ael6vsr to 0
STDERR: 2023-07-23T15:48:31.644523960Z <=========
```
Two announcements:
First:
@lulululu thanks for asking! Yes, we adjust our timeline to the timeline of the BraTS organization. The website with the new deadlines should be up to date by now. Apologies for the confusion.
Second:
We fixed a bug in our evaluation script, participants who already uploaded a submission to the validation set should have received an email. Dear all,
thanks a lot for organizing the challenge!
On the BraTS challenge overview page (https://www.synapse.org/#!Synapse:syn51156910/wiki/) the validation / short paper submission deadline was changed from July 31 to August 7 and the submission deadline of MLCubes from August 7 to August 14. However, on the inpainting-specific page (https://www.synapse.org/#!Synapse:syn51156910/wiki/622357) still July 31 appears as deadline for validation / short paper submission, August 7 as deadline for MLcubes submission.
When is the final date to submit validation files and the short paper? And when is the MLcubes submission deadline?
Thank you in advance and have a great day.
Dear @BraTSChallengeParticipants, the validation phase of the [inpainting challenge](https://www.synapse.org/#!Synapse:syn51156910/wiki/622357) has officially begun! Evaluate your model following the [tutorial](https://www.synapse.org/#!Synapse:syn51156910/wiki/622885)! Thanks for clarifying all issues @neuronflow . Hi @King-HAW ,
Thank you for your interest in our challenge!
1. Yes, that is correct; the validation dataset contains tumors. Your task is to construct inpainting models that inpaint everywhere, where the inpainting mask is to synthesize a healthy version of the tumor brain. Note: the mask might also cover already healthy tissue or background. Your model can also synthesize beyond the mask if you believe that will give you more realistic images. We quantitatively evaluate only healthy brain areas because of the mentioned lack of ground truth. However, the model should inpaint everything that is masked as models are also evaluated qualitatively, and again, the goal of the challenge is to synthesize healthy brains.
2. Yes, you can use any publicly available and also internal datasets. But please document all additional datasets in your method description.
3. Yes, again, please document this in your method description. Hi there,
Many thanks for hosting this brain inpainting challenge. Here I have some questions about this challenge.
1. For the validation set, I notice that images in the validation set have tumors (obvious from the shifted midline). Also the masks for the validation images contain tumor area and healthy area. Since the ground-truth (healthy tissues) for the tumor is not available, how will the evaluation be conducted? Or the evaluation will only be conducted on the healthy area (inpainting in the tumor area can be ignored)?
2. Can I use additional publicly-available healthy T1 weighted brain MRI datasets for model training? Or using any additional datasets are not permitted.
3. Can I use the publicly-available pretrained model weights to assist model training?
Best,
Gary @BraTSChallengeParticipants @BraTS2023Participants
Ever wondered what the brain of a tumor patient looked like before they developed the disease? This question keeps doctors awake at night. Help them by joining the BraTS Inpainting Challenge!
Kindly consider spreading the word at your institutions and retweeting: https://twitter.com/BraTS_inpaint/status/1665651190737018880?s=20
Thank you for your time and support!
PS: In the upcoming days, we will release the validation data so you can start officially benchmarking your models. Hurray, thanks or reporting back and happy inpainting! :) @neuronflow Yes, it works! our challenge is now officially launched - see announcement in the first post! @Ruizhi_Zhu
Thanks for reporting this. I changed the URL to: https://www.synapse.org/#!Synapse:syn51156910/wiki/622347
Please let me know whether this works for you. Signup for the challenge, following the **instructions** for steps 1-3 here and request access to the data.
When I click the url, it shows 'Sorry, no access to this page.'.
Drop files to upload
[inpainting challenge] Announcements and Q&A page is loading…