Hello
i am trying to make a submission,for which i wanted to pull the demo image and maybe modify that(please let me know if there is a better way)
Anyway, in order to run this command `docker pull docker.synapse.org/syn20545111:hello-world` i think i have to login first by running this command `docker login docker.synapse.org`,for which i enter my synapse login and password details to which i get
Error response from daemon: Get https://docker.synapse.org/v2/: received unexpected HTTP status: 503
am i doing something wrong here?
Thanks
Created by decentmakeover @allawayr i will be reading the files using the template csv, is this right? or does it have to read using the test folder?
read_template = pd.read_csv('template.csv')
filenames = read_template['Patient_ID'].tolist() Oh also - you need to export the paths correctly in your run.sh - as shown by the example We used conda, to install tensorflow-gpu (see https://towardsdatascience.com/tensorflow-gpu-installation-made-easy-use-conda-instead-of-pip-52e5249374bc)
Which worked for us is anyone using pytorch docker? if so how did you guys configure it to run on cuda??
this is my DOCKERFILE
ROM nvidia/cuda:10.0-base-ubuntu16.04
# Install some basic utilities
RUN apt-get update && apt-get install -y \
curl \
ca-certificates \
sudo \
git \
bzip2 \
libx11-6 \
&& rm -rf /var/lib/apt/lists/*
# Create a working directory
RUN mkdir /app
WORKDIR /app
COPY bone_challenge /bone_challenge
RUN chmod -R 777 /bone_challenge
# Create a non-root user and switch to it
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
&& chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user
# All users can use /home/user as their home directory
ENV HOME=/home/user
# Install Miniconda
RUN curl -so ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh
ENV PATH=/home/user/miniconda/bin:$PATH
ENV CONDA_AUTO_UPDATE_CONDA=false
# Create a Python 3.6 environment
RUN /home/user/miniconda/bin/conda create -y --name py36 python=3.6.9 \
&& /home/user/miniconda/bin/conda clean -ya
ENV CONDA_DEFAULT_ENV=py36
ENV CONDA_PREFIX=/home/user/miniconda/envs/$CONDA_DEFAULT_ENV
ENV PATH=$CONDA_PREFIX/bin:$PATH
RUN /home/user/miniconda/bin/conda install conda-build=3.18.9=py36_3 \
&& /home/user/miniconda/bin/conda clean -ya
# CUDA 10.0-specific steps
RUN conda install -y -c pytorch \
cudatoolkit=10.0 \
"pytorch=1.2.0=py3.6_cuda10.0.130_cudnn7.6.2_0" \
"torchvision=0.4.0=py36_cu100" \
&& conda clean -ya
# Install Requests, a Python library for making HTTP requests
RUN conda install -y requests=2.19.1 \
&& conda clean -ya
# Install OpenCV3 Python bindings
RUN sudo apt-get update && sudo apt-get install -y --no-install-recommends \
libgtk2.0-0 \
libcanberra-gtk-module \
&& sudo rm -rf /var/lib/apt/lists/*
RUN conda install -y -c menpo opencv3=3.1.0 \
&& conda clean -ya
RUN conda install pandas
RUN conda install pillow=6.1
RUN conda install -c conda-forge pycocotools
RUN pip install scikit-image
RUN conda install tensorboard
RUN conda install future
# RUN conda install pycocotools
COPY run.sh /run.sh
RUN sudo chmod +x /run.sh
# Set the default command to python3
# RUN ["sleep","5h"]
ENTRYPOINT ["/bin/bash", "/run.sh"]
but when i run i get
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/cuda/__init__.py", line 178, in _lazy_init
_check_driver()
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/cuda/__init__.py", line 99, in _check_driver
http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx
any suggestions? Thanks @sds for passing along that information.
It looks like @dcentmakeover's question has been addressed, but let me know if you need additional help. @sds you have restored my faith in god,thanks. @dcentmakeover We had the same problem, the following information from [another thread](https://www.synapse.org/#!Synapse:syn20545111/discussion/threadId=6744) solved it for us;
Hi there, are you a Certified User on Synapse? You can find out by going to https://www.synapse.org/#!Profile:v/settings
There should be a green button that reads "Certification Quiz Passed". In order to upload data or docker containers to Synapse you must be a certified user.
@allawayr @ anyone i need some help here, i have created a dockerfile , my docker is ready but when i try to push i keep getting `denied: requested access to the resource is denied`, i have created a project called DevilsAngel and i am using the synapseid from that projecy to push.it looks somethign like this ` sudo docker push docker.synapse.org/syn21667793/submission:01`
any suggestions okay thanks Yes you have to make your own project, and under the project there is a Docker tab. i just created a new porject from the project page ,hoping it works @lars.ericson hey how did you create your projectid? i was using my synapse id which shows up at the top of the screen as my project id, i think thats why i am getting denied request, is that correct or did you make a seperate project id?if so ,how? oh, man this sucks, thanks for the comment.
ill start right now. To create a Docker container, you don't directly enter someone else's Docker container and do stuff to it. You make a Dockerfile which starts by pulling the other container, which has features you like, and then customize that Dockerfile to bring in your code and models. Or you could edit Roberts Dockerfile directly if you are following his pattern. Either way, you can't get anywhere if you have no Dockerfile.
You should watch the Docker tutorials from Docker and read some basic introductions to Docker, to better understand what it is providing and how the Docker ecosystem works. also @allawayr i get `denied: requested access to the resource is denied` when i try running `sudo docker push docker.synapse.org/syn20545111/version:3` but thats inside the dockerfile right? i do not have a dockerfile. The ENTRYPOINT in the last line of the Dockerfile I posted above is a clue. hello , i am almost there, i have built the docker but now, i dont have a dockerfile i just pulled a docker from dockerhub and modified it, so now i am confused how do i set an entrypoint, can an entrypoint only be set with a dockerfile or ....... any suggestions please hmmm okay thanks for the response. @decentmakeover, the config I show worked for me. Notice that it is built on top of a Tensorflow container and I do an Nvidia install and a Cuddenv install on top of that. The net result is that it works. @lars.ericson hello , i see you havent defined any cuda enviornment variables, also do i need to install cuda 10 or so in the docker or will it be automatically taken care of?
thanks @decentmakeover yes score.py looks in the /test directory for the images and then cogitates and then writes prediction.csv to the /output directory. @stadlerm has good suggestions! Particularly with regards to uploading to your personal project and NOT the challenge project - this happens somewhat often.
Regarding training, @james.costello says in [this thread](https://www.synapse.org/#!Synapse:syn20545111/discussion/threadId=6346) that pre-trained models can be submitted. However, they must be written in a way that they could score any hand/feet images provided in this format (not just hardcoded to the template patient IDs). This is particularly important so that the models can score the validation data in the final round.
However, we've configured the scoring server to provide computational resources and training data so that you can train on the server. This may be useful for folks that don't have a lot of local compute available to develop their models.
okay thanks for the comment. i am building my docker image now its taking time!!, I think there is still a requirement that the model must be able to be trained on the synapse servers (@allawayr can be maybe clear this up)
However, none of this should cause the errors you are experiencing - your error HTTP 503, indicates that you are not able to upload an image in the first place.
Make sure that your account is verified on synapse (see the challenge instructions on how to get verified), as that is required to upload files. Also make sure the synapse servers are reachable from your network. Lastly, make sure you are uploading to YOUR project, not the challenge project.
You need to create your own project on synapse, named after your team. You upload to this project, and then submit from there to the challenge submission lane
Generally, you should always be able to upload any image, and then you will get an email saing your image isn't valid, if your image doesn't write the necessary output file (different error than the one you are getting) @lars.ericson Hi Lars, I am guessing your score.py is a script which takes in test images and generates the csv file? If i understand correctly there is no need to train in the docker ?
Thanks @allawayr thanks for the comments, i am trying to build this now, so ill go through lars's comments and also check @lars.ericson Thank you so much,i will look into this. @decentmakeover, here is my **Dockerfile**:
```
FROM tensorflow/tensorflow:nightly-gpu-py3
# Updating the image
RUN apt-get update -y
RUN apt-get install -y apt-transport-https
RUN apt-get install emacs -y
RUN apt install nvidia-cuda-toolkit -y
# Configure Python
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get -y install tzdata
RUN apt install python3-opencv -y
RUN pip install pandas
RUN pip install scikit-learn
RUN pip install scikit-image
RUN pip install ipywidgets
RUN pip install cudnnenv
# Required: Create /train /test and /output directories
RUN mkdir /train
RUN mkdir /test
RUN mkdir /output
# Required: trained models
COPY model_035_release.tar.gz /model_035_release.tar.gz
RUN tar xvfz /model_035_release.tar.gz
# Main program
COPY run.sh /run.sh
# Required: code
COPY score.py /score.py
# Make model and runfiles executable
RUN chmod 775 /run.sh
# This is for the virtualenv defined above, if not using a virtualenv, this is not necessary
RUN chmod 755 /root #to make virtualenv accessible to singularity user
# Required: define an entrypoint. run.sh will run the model for us, but in a different configuration
# you could simply call the model file directly as an entrypoint
ENTRYPOINT ["/bin/bash", "/run.sh"]
```
and my **run.sh** file:
```
#!/bin/bash
python3 /score.py
```
To build the Docker image:
```
docker build -t docker.synapse.org/syn123456/model:001 .
```
To test it locally:
```
export STAGE=/home/catskills/Desktop/ra2/stage
docker run \
-v $STAGE/test:/test:ro \
-v $STAGE/train:/train:ro \
-v $STAGE/output:/output \
--gpus all \
docker.synapse.org/syn123456/model:001
```
Note that, running locally, you can copy files from inside the image into /output and then see them on your host machine. You can also copy on the host machine into /test and /train and see those files inside the running Docker image. Sometimes it is helpful to run the image, get inside it, and move stuff in and out from the host machine. To run the image locally do this:
```
export STAGE=/home/catskills/Desktop/ra2/stage
docker run -it \
-v $STAGE/test:/test:ro \
-v $STAGE/train:/train:ro \
-v $STAGE/output:/output \
--gpus all \
--entrypoint=/bin/bash docker.synapse.org/syn123456/model:001
```
To upload the image do something like this:
```
docker push docker.synapse.org/syn123456/model:001
```
It helps to read these links closely for understanding:
* [Docker Submission](https://www.synapse.org/#!Synapse:syn20545111/wiki/597249)
* [Getting started with Docker](https://docs.docker.com/get-started/)
Hi there,
I provided docker.synapse.org/syn20545111:hello-world so that people could take a look at it if they desired, but if you are using the hello-world container as a template, I would instead suggest running `git clone https://github.com/allaway/ra2-docker-demo` and editing the resulting files locally. After doing so, you can `docker build` the Dockerfile that directory to create and push that container.
I am sure how familiar with docker you are but you cannot simply run the image interactively and save your changes - the running docker container is ephemeral. Instead, you have to build an image from a directory containing the Dockerfile that installs dependencies and copies in any other required runfiles. You could theoretically start your Dockerfile with `FROM docker.synapse.org/syn20545111:hello-world`, but I would not recommend this. I would instead build from the smallest base possible (e.g. `FROM rocker/r-base@sha256:ec224c21eff00e6cd8016419fae2886596c76e80fb1ae042e657b3cd08ba30d8` as defined here: https://github.com/allaway/ra2-docker-demo/blob/master/Dockerfile) okay thanks ill give this a check. Hey - make sure to only login with your username, and not the synapse email
Also I guess, which goes without saying - make sure you spell the password correctly
Also, try deleting this file /home/{user}/.docker/config.json - docker stores your auth credentials here and it might be corrupted, some issue with them
Lastly, make sure to the login and push command with sudo - otherwise it might not be able to start services/upload files and so forth