It takes a long time - several days - to get the files pushed up to synapse. The problem seems, mainly, to be timeouts. The timeout seems to be set to 19 minutes. The problem is that, after timing out, it restarts the loading of those that have not been pushed. Is there a way to increase the timeout? It would make a big difference, both to the time taken to upload, and the amount of data I have to send - multiple re-loads take a lot of bandwidth. These are the lines from syslog - I hope they give an idea of what the problem might be: ``` Oct 19 20:47:54 govern dockerd[5587]: time="2016-10-19T20:47:54.536992602+02:00" level=error msg="Upload failed, retrying: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 19 20:49:20 govern dockerd[5801]: time="2016-10-19T20:49:20.621758124+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" Oct 19 20:49:39 govern dockerd[5801]: time="2016-10-19T20:49:39.167231781+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" Oct 19 20:49:39 govern dockerd[5801]: time="2016-10-19T20:49:39.167373727+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" Oct 19 20:49:39 govern dockerd[5801]: time="2016-10-19T20:49:39.167455026+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.062494633+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.071580857+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.071853723+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.089799599+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout" ```

Created by Peter Brooks fustbariclation
@fustbariclation, You might be constrained by your network upload metering. Members of our team also had some issues. Our solution is to use local machines for coding, and commits, but use a much better connected machine for building images, pulling, pushing. --r
> I've got it working, and tried it a few times now - it seems much the same When you say "got *it* working" are you referring to pushing to DockerHub? When you say "it seems much the same" are you saying that your experience trying to push images to DockerHub is "much the same" as trying to push to Synapse? Based on the session info you include with your comment, I think your answers to both these question is "yes". > I was hoping it was to upgrade things to fix bugs. If your experience with DockerHub is the same as with Synapse then I don't think there is anything we can change in Synapse to address this issue. That is, when you push to DockerHub the interaction is purely between your Docker installation and their server. Our system is not involved. I wonder if there might be more insight from the folks on the Docker forum ( https://forums.docker.com )? A quick search revealed this discussion thread: https://forums.docker.com/t/fata-0025-io-timeout-on-docker-image-push/1742/6 If you resolve the issue, please let us know!
I see that synapse was down for maintenance this morning - I was hoping it was to upgrade things to fix bugs. Unfortunately, things don't seem to have improved much for me. I'd be grateful for any suggestions to help with this! I've also moved to a completely new machine and re-installed everything in sight, to the correct revision, as well as checking all my network connections and configurations. I'm now getting this error, again, it had stopped happening yesterday: ``` net/http: TLS handshake timeout ``` I'm also getting these errors: ``` TCP 192.168.65.2:54970 > 50.17.241.148:443 proxy failed with flow proxy b: Connection refused ``` Yesterday, my 'docker push' commands were aborting after 1-2 hours, which was a great improvement. Today, it's back to 23 minutes - it was 19 minutes on synapse early last week, and 21 minutes on the docker site itself. Something seems to have happened to the upload speed too, I've checked my ISP, and it isn't that, earlier this week, it was varying between 5-7Mb/s - now it is mainly about 128Kb/s sometimes spiking at 1.41Mb/s. It seems to be throwing away segments that it has already uploaded and trying again. The following is with this repository: ``` [docker.synapse.org/syn7364819/blue_training] ``` On feature of the pushes that makes them take so long is that they seem to upload an entire segment. for example: ``` 67aca2b232ea: Pushing [================================================> ] 170 MB/177 MB 3ac0d1046245: Pushing [==================================================>] 86.27 MB/86.27 MB ``` Then, a few minutes later ( this is happening now, 11:18GMT, if you're looking in your log files) this: ``` 67aca2b232ea: Pushing [==================================================>] 180.6 MB 3ac0d1046245: Pushing [==================================================>] 86.27 MB/86.27 MB 2c2153fbd032: Pushing [==================================================>] 8.102 MB ``` Then it looks like it's worked: ``` 67aca2b232ea: Pushed ``` Then:: ``` Sun 23 Oct 2016 13:27:42 SAST Run time: 0 days 1 hours 23 minutes 0 seconds net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ``` When I restart, with the same image, though: ``` 67aca2b232ea: Pushing [=> ] 5.407 MB/177 MB ```
Any thoughts about this one? It is making it very difficult to make progress. The main problem is that 177Mb is just too big to load in 20 minutes - I'm not sure why, because it's easy to download five times that in less than 20 minutes. If it was possible to split the chunks, it'd solve the problem.
I've got it working, and tried it a few times now - it seems much the same. The timeout is 32 minutes, rather than 19. I've tried modifying various tcp timeout options, but it doesn't seem to have made much difference. As you can see, I've tried it a number of times to be sure. ``` + logger Starting docker push -- Oct 20 15:25:59 govern peterbrooks: Starting docker push + docker push fustbariclation/fustbariclation The push refers to a repository [docker.io/fustbariclation/fustbariclation] 93d8703dde2a: Layer already exists 934396e52c9d: Layer already exists 8efaddf4ca67: Layer already exists ffd427b1c79a: Layer already exists 92313c584362: Retrying in 1 second 4a4f58055766: Retrying in 1 second adb25476d2c1: Retrying in 1 second 655a6c01124a: Pushing [==========================================> ] 175.8 MB/205.9 MB 0de3b0993fca: Retrying in 1 second 2253b7eea5e6: Retrying in 1 second 05b90b2afe02: Retrying in 1 second ec96d31f8eaf: Retrying in 1 second e3ac10d00c60: Retrying in 1 second 2c2153fbd032: Retrying in 1 second d9a069c1d0fc: Retrying in 1 second a5eb0fc1decb: Retrying in 1 second b2ac5371e0f2: Retrying in 1 second 142a601d9793: Retrying in 1 second net/http: TLS handshake timeout + logger Docker push stopped -- Oct 20 15:57:54 govern peterbrooks: Docker push stopped ``` These are the tcp parameters I tried modifying in /etc/sysctl.conf -- but, as I said, it didn't seem to make any difference. If you've any suggestions for a good setting, that'd be great! ``` net.ipv4.tcp_mtu_probing=1 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.xfrm_aevent_etime = 50 net.ipv4.conf.docker0.accept_local = 0 net.ipv4.conf.docker0.accept_redirects = 1 net.ipv4.conf.docker0.accept_source_route = 1 net.ipv4.conf.docker0.arp_accept = 0 net.ipv4.conf.docker0.arp_announce = 0 net.ipv4.conf.docker0.arp_filter = 0 net.ipv4.conf.docker0.arp_ignore = 0 net.ipv4.conf.docker0.arp_notify = 0 net.ipv4.conf.docker0.bootp_relay = 0 net.ipv4.conf.docker0.disable_policy = 0 net.ipv4.conf.docker0.disable_xfrm = 0 net.ipv4.conf.docker0.force_igmp_version = 0 net.ipv4.conf.docker0.forwarding = 1 net.ipv4.conf.docker0.igmpv2_unsolicited_report_interval = 10000 net.ipv4.conf.docker0.igmpv3_unsolicited_report_interval = 1000 net.ipv4.conf.docker0.ignore_routes_with_linkdown = 0 net.ipv4.conf.docker0.log_martians = 0 net.ipv4.conf.docker0.mc_forwarding = 0 net.ipv4.conf.docker0.medium_id = 0 net.ipv4.conf.docker0.promote_secondaries = 0 net.ipv4.conf.docker0.proxy_arp = 0 net.ipv4.conf.docker0.proxy_arp_pvlan = 0 net.ipv4.conf.docker0.route_localnet = 0 net.ipv4.conf.docker0.rp_filter = 1 net.ipv4.conf.docker0.secure_redirects = 1 net.ipv4.conf.docker0.send_redirects = 1 net.ipv4.conf.docker0.shared_media = 1 net.ipv4.conf.docker0.src_valid_mark = 0 net.ipv4.conf.docker0.tag = 0 net.ipv4.ipfrag_time = 50 net.ipv4.neigh.default.base_reachable_time_ms = 30000 net.ipv4.neigh.default.delay_first_probe_time = 5 net.ipv4.neigh.default.gc_stale_time = 600 net.ipv4.neigh.default.locktime = 100 net.ipv4.neigh.default.retrans_time_ms = 3000 net.ipv4.neigh.docker0.anycast_delay = 400 net.ipv4.neigh.docker0.app_solicit = 0 net.ipv4.neigh.docker0.base_reachable_time_ms = 90000 net.ipv4.neigh.docker0.delay_first_probe_time = 1 net.ipv4.neigh.docker0.delay_first_probe_time = 5 net.ipv4.neigh.docker0.gc_stale_time = 600 net.ipv4.neigh.docker0.gc_stale_time = 600S net.ipv4.neigh.docker0.locktime = 100 net.ipv4.neigh.docker0.locktime = 1000 net.ipv4.neigh.docker0.mcast_resolicit = 0 net.ipv4.neigh.docker0.mcast_solicit = 3 net.ipv4.neigh.docker0.proxy_delay = 80 net.ipv4.neigh.docker0.proxy_qlen = 64 net.ipv4.neigh.docker0.retrans_time_ms = 9000 net.ipv4.neigh.docker0.ucast_solicit = 3 net.ipv4.neigh.docker0.unres_qlen = 31 net.ipv4.neigh.docker0.unres_qlen_bytes = 65536 net.ipv4.neigh.enp0s3.base_reachable_time_ms = 30000 net.ipv4.neigh.enp0s3.delay_first_probe_time = 5 net.ipv4.neigh.enp0s3.gc_stale_time = 60 net.ipv4.neigh.enp0s3.locktime = 100 net.ipv4.neigh.enp0s3.retrans_time_ms = 1000 net.ipv4.neigh.lo.base_reachable_time_ms = 30000 net.ipv4.neigh.lo.delay_first_probe_time = 5 net.ipv4.neigh.lo.gc_stale_time = 60 net.ipv4.neigh.lo.locktime = 100 net.ipv4.neigh.lo.retrans_time_ms = 1000 net.ipv4.route.gc_timeout = 300 net.ipv4.tcp_fack = 1 net.ipv4.tcp_fin_timeout = 60 net.ipv4.tcp_frto = 2 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_sack = 1 net.ipv4.tcp_thin_linear_timeouts = 0 net.ipv4.tcp_wmem = 4096 87380 16777216 net.ipv6.conf.docker0.accept_dad = 1 net.ipv6.conf.docker0.accept_ra = 1 net.ipv6.conf.docker0.accept_ra_defrtr = 1 net.ipv6.conf.docker0.accept_ra_from_local = 0 net.ipv6.conf.docker0.accept_ra_min_hop_limit = 1 net.ipv6.conf.docker0.accept_ra_mtu = 1 net.ipv6.conf.docker0.accept_ra_pinfo = 1 net.ipv6.conf.docker0.accept_ra_rt_info_max_plen = 0 net.ipv6.conf.docker0.accept_ra_rtr_pref = 1 net.ipv6.conf.docker0.accept_redirects = 1 net.ipv6.conf.docker0.accept_source_route = 0 net.ipv6.conf.docker0.autoconf = 1 net.ipv6.conf.docker0.dad_transmits = 1 net.ipv6.conf.docker0.disable_ipv6 = 0 net.ipv6.conf.docker0.force_mld_version = 0 net.ipv6.conf.docker0.force_tllao = 0 net.ipv6.conf.docker0.forwarding = 0 net.ipv6.conf.docker0.hop_limit = 64 net.ipv6.conf.docker0.ignore_routes_with_linkdown = 0 net.ipv6.conf.docker0.max_addresses = 16 net.ipv6.conf.docker0.max_desync_factor = 600 net.ipv6.conf.docker0.mc_forwarding = 0 net.ipv6.conf.docker0.mldv1_unsolicited_report_interval = 10000 net.ipv6.conf.docker0.mldv2_unsolicited_report_interval = 1000 net.ipv6.conf.docker0.mtu = 4096 net.ipv6.conf.docker0.ndisc_notify = 0 net.ipv6.conf.docker0.proxy_ndp = 0 net.ipv6.conf.docker0.regen_max_retry = 9 net.ipv6.conf.docker0.router_probe_interval = 600 net.ipv6.conf.docker0.router_solicitation_delay = 1 net.ipv6.conf.docker0.router_solicitation_interval = 4 net.ipv6.conf.docker0.router_solicitations = 3 net.ipv6.conf.docker0.suppress_frag_ndisc = 1 net.ipv6.conf.docker0.temp_prefered_lft = 86400 net.ipv6.conf.docker0.temp_valid_lft = 604800 net.ipv6.conf.docker0.use_oif_addrs_only = 0 net.ipv6.conf.docker0.use_tempaddr = 2 net.ipv6.neigh.docker0.anycast_delay = 100 net.ipv6.neigh.docker0.app_solicit = 0 net.ipv6.neigh.docker0.base_reachable_time_ms = 90000 net.ipv6.neigh.docker0.delay_first_probe_time = 5 net.ipv6.neigh.docker0.gc_stale_time = 600 net.ipv6.neigh.docker0.locktime = 0 net.ipv6.neigh.docker0.mcast_resolicit = 0 net.ipv6.neigh.docker0.mcast_solicit = 3 net.ipv6.neigh.docker0.proxy_delay = 80 net.ipv6.neigh.docker0.proxy_qlen = 128 net.ipv6.neigh.docker0.retrans_time_ms = 9000 net.ipv6.neigh.docker0.ucast_solicit = 3 net.ipv6.neigh.docker0.unres_qlen = 31 net.ipv6.neigh.docker0.unres_qlen_bytes = 65536 ```
Peter: I apologize for my misleading comment: The address hub.docker.com is used in your web browser but not with the Docker client. With Docker when you **omit** the registry address you are referring to DockerHub by default: ``` [~] docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username (brucehoff): brucehoff Password: Login Succeeded [~] ``` You name the image with your DockerHub user name followed by the repository name, e.g. ``` docker build -t fustbariclation/your-inspired-model . ``` Then push to DockerHub ``` docker push fustbariclation/your-inspired-model ``` Would you please let me know if this works for you?
I've created a repository there. Unfortunately I get a different problem - I can't find a solution on-line. This is how it goes. As you can see, everything seems to work, it even looks as if it's trying to push, but, after a couple of seconds it gives the error: ``` Index response didn't contain any endpoints ``` ``` root@govern:/media/sf_mammogram/blue# docker login hub.docker.com Username (fustbariclation): fustbariclation Password: Login Succeeded root@govern:/media/sf_mammogram/blue# docker build -t hub.docker.com/fustbariclation/fustbariclation . Sending build context to Docker daemon 110.1 kB Step 1 : FROM python ---> 1d0326469b55 Step 2 : RUN date ---> Using cache ---> 2c4f8b9e35be Step 3 : RUN pip install pydicom ---> Using cache ---> 1b14765d4688 Step 4 : RUN pip install Image ---> Using cache ---> 010062ae6c6d Step 5 : RUN pip install cm ---> Using cache ---> ba770065980c Step 6 : RUN pip install scipy ---> Using cache ---> 4ee14dc8f763 Step 7 : RUN pip install numpy ---> Using cache ---> 22a73112174a Step 8 : RUN pip install setuptools ---> Using cache ---> 8aff5e4216a0 Step 9 : RUN apt-get update --fix-missing && apt-get install -y --fix-missing r-base ---> Using cache ---> 3a07386fba61 Step 10 : COPY train.py /train.py ---> Using cache ---> 5cc4c88f89e0 Step 11 : COPY train.sh /train.sh ---> Using cache ---> 7e2843466f0d Step 12 : COPY test.sh /test.sh ---> Using cache ---> c67abe918ab7 Step 13 : COPY preprocess.sh /preprocess.sh ---> Using cache ---> e1833a96b9d8 Step 14 : RUN /bin/df -H ---> Using cache ---> e6985005a2bb Step 15 : RUN ls -lt ---> Using cache ---> 2382a1ab1087 Step 16 : RUN echo "Completed initialise phase - now running preprocess.sh" ---> Using cache ---> f99dd0f01931 Step 17 : RUN date ---> Using cache ---> 95a88fbfbe86 Step 18 : RUN /preprocess.sh ---> Using cache ---> c81bf9f489f9 Step 19 : RUN date ---> Using cache ---> 4810f9fcaf0e Step 20 : RUN echo "End of Dockerfile" ---> Using cache ---> 33778bef622c Successfully built 33778bef622c root@govern:/media/sf_mammogram/blue# docker push hub.docker.com/fustbariclation/fustbariclation:latest The push refers to a repository [hub.docker.com/fustbariclation/fustbariclation] 93d8703dde2a: Preparing 934396e52c9d: Preparing 8efaddf4ca67: Preparing ffd427b1c79a: Preparing 92313c584362: Preparing 4a4f58055766: Waiting adb25476d2c1: Waiting 655a6c01124a: Waiting 0de3b0993fca: Waiting 2253b7eea5e6: Waiting 05b90b2afe02: Waiting ec96d31f8eaf: Waiting e3ac10d00c60: Waiting 2c2153fbd032: Waiting d9a069c1d0fc: Waiting a5eb0fc1decb: Waiting b2ac5371e0f2: Waiting 142a601d9793: Waiting Index response didn't contain any endpoints ```
Peter: Do you have the same or different experience when you push to DockerHub (https://hub.docker.com/) rather than Synapse? The answer could help determine whether the problem lies with our Docker registry, with your client, or with the network connectivity. Thank you.

Push timeout page is loading…