I am not able to create files inside /output directory of my docker containers.
Example:
$ more Dockerfile
...
USER root
RUN apt-get update && apt-get -y install sudo
RUN echo 'Set disable_coredump false' > /etc/sudo.conf
RUN sudo rm -rf /output
RUN sudo mkdir /output
RUN sudo chmod -R a+rwx /output
RUN echo hello > /output/hi.txt
COPY beatAML.R /output/
ENTRYPOINT more /output/beatAML.R
It works locally, but fails when I submit to the Synapse:
more: stat of /output/beatAM.R failed: No such file or directory
more: stat of /output/hi.txt failed: No such file or directory
It looks to me that your python wrapper that executes the containers overrides the permissions as there is always this file run_docker.py inside it:
/:
total 4
drwx------ 4 root root 63 Jan 31 17:24 output
/output:
total 0
lrwxrwxrwx 1 root root 235 Jan 31 17:24 run_docker.py -> /var/lib/docker/volumes/workflow_orchestrator_shared/_data/26cb94ff-4b28-4915-b242-67403227a06d/toil-a88f68a6-1b22-4f96-9ce8-2159df338cbf-a5a85d161208265bec1c6fb020b29456/tmpr4aombj2/0ab776bb-1e84-4fe5-8c5c-2223413ba67c/tmp95c2qs4v.tmp
I wonder if this is done purposefully (i.e. it's part of the challenge) or your docker execution pipeline is perhaps broken?
Created by ymemari @ymemari,
Ah yes, the encoding error was on our part, so I apologize for that. Glad to hear you got some submissions in! And hey, a score is a score!
Let us know if you run into any other issues. Thank you again for participating!
Best,
Verena Sure, I can write into /outout/ using ENTRYPOINT, but I was mislead by the fact that it never worked with RUN or COPY...
Anyway, as you probably noticed, the actual problem was caused by this error:
STDERR: 2020-02-03T16:09:24.920287409Z 1555110c2944 2020-02-03 16:09:24,919 MainThread WARNING toil.leader: kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_a2ea7618-df61-44c7-a1a0-5e215f2f79aa_CTD2-BeatAML-Challenge-master_run_docker.cwl/instanced65duuqs UnicodeEncodeError: 'ascii' codec can't encode character '\u2018' in position 20: ordinal not in range(128)
which I was trying to solve by modifying the locales or PYTHONIOENCODING...
I noticed you resubmitting my jobs and fixing the outstanding problems yesterday. Many thanks!
I have now managed to make my first successful submission where I am literally at the bottom of the leaderboard, lol! @ymemari,
You are correct! You will want the predictions to be written to `/output/predictions.csv`, however, it shouldn't be your Docker image that creates the output file; it should be your Rscript. Is your current Rscript writing the predictions to `/output/predictions.csv`? If it was and you had run into errors, can you let me know what they were?
Best,
Verena Thanks for your response, but shouldn't the prediction file be written into /output/predictions.csv?
The above is only for demonstration to show that I am not able to write into /output neither using 'RUN echo' nor COPY, but I am not intending to store my R code in there. No matter I explicitly 'RUN mkdir /output' or not, or resort to excessive means as above, I am unable to create anything inside /output and then this apparently causes the job to fail:
STDERR: 2020-02-01T10:24:07.242606095Z 9670cd0203ad 2020-02-01 10:24:07,242 MainThread WARNING toil.leader: kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_502a057c-677b-4db6-ba45-e884b9cad71a_CTD2-BeatAML-Challenge-master_run_docker.cwl/instancefdms29qs ("Error collecting output for parameter 'predictions':\nCTD2-BeatAML-Challenge-master/run_docker.cwl:61:7: Did not find output file with glob pattern: '['predictions.csv']'", {})
Dear @ymemari,
Thank you for participating!
I see in your Dockerfile, you are creating an `output` directory where a text file and R script are then stored. The errors you are receiving (`No such file or directory`) are due to your directory getting overwritten when we mount `/input` and `/output` (so essentially, `/output` becomes empty).
If you would like to copy over your files/scripts, I suggest copying into a different directory.
Best,
Verena