It takes a long time - several days - to get the files pushed up to synapse.
The problem seems, mainly, to be timeouts. The timeout seems to be set to 19 minutes.
The problem is that, after timing out, it restarts the loading of those that have not been pushed.
Is there a way to increase the timeout? It would make a big difference, both to the time taken to upload, and the amount of data I have to send - multiple re-loads take a lot of bandwidth.
These are the lines from syslog - I hope they give an idea of what the problem might be:
```
Oct 19 20:47:54 govern dockerd[5587]: time="2016-10-19T20:47:54.536992602+02:00" level=error msg="Upload failed, retrying: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 19 20:49:20 govern dockerd[5801]: time="2016-10-19T20:49:20.621758124+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
Oct 19 20:49:39 govern dockerd[5801]: time="2016-10-19T20:49:39.167231781+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
Oct 19 20:49:39 govern dockerd[5801]: time="2016-10-19T20:49:39.167373727+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
Oct 19 20:49:39 govern dockerd[5801]: time="2016-10-19T20:49:39.167455026+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.062494633+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.071580857+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.071853723+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
Oct 19 20:50:21 govern dockerd[5801]: time="2016-10-19T20:50:21.089799599+02:00" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
```
Created by Peter Brooks fustbariclation @fustbariclation,
You might be constrained by your network upload metering. Members of our team also had some issues.
Our solution is to use local machines for coding, and commits, but use a much better connected machine for building images, pulling, pushing.
--r > I've got it working, and tried it a few times now - it seems much the same
When you say "got *it* working" are you referring to pushing to DockerHub? When you say "it seems much the same" are you saying that your experience trying to push images to DockerHub is "much the same" as trying to push to Synapse? Based on the session info you include with your comment, I think your answers to both these question is "yes".
> I was hoping it was to upgrade things to fix bugs.
If your experience with DockerHub is the same as with Synapse then I don't think there is anything we can change in Synapse to address this issue. That is, when you push to DockerHub the interaction is purely between your Docker installation and their server. Our system is not involved. I wonder if there might be more insight from the folks on the Docker forum ( https://forums.docker.com )? A quick search revealed this discussion thread:
https://forums.docker.com/t/fata-0025-io-timeout-on-docker-image-push/1742/6
If you resolve the issue, please let us know! I see that synapse was down for maintenance this morning - I was hoping it was to upgrade things to fix bugs. Unfortunately, things don't seem to have improved much for me.
I'd be grateful for any suggestions to help with this!
I've also moved to a completely new machine and re-installed everything in sight, to the correct revision, as well as checking all my network connections and configurations. I'm now getting this error, again, it had stopped happening yesterday:
```
net/http: TLS handshake timeout
```
I'm also getting these errors:
```
TCP 192.168.65.2:54970 > 50.17.241.148:443 proxy failed with flow proxy b: Connection refused
```
Yesterday, my 'docker push' commands were aborting after 1-2 hours, which was a great improvement. Today, it's back to 23 minutes - it was 19 minutes on synapse early last week, and 21 minutes on the docker site itself. Something seems to have happened to the upload speed too, I've checked my ISP, and it isn't that, earlier this week, it was varying between 5-7Mb/s - now it is mainly about 128Kb/s sometimes spiking at 1.41Mb/s.
It seems to be throwing away segments that it has already uploaded and trying again. The following is with this repository:
```
[docker.synapse.org/syn7364819/blue_training]
```
On feature of the pushes that makes them take so long is that they seem to upload an entire segment. for example:
```
67aca2b232ea: Pushing [================================================> ] 170 MB/177 MB
3ac0d1046245: Pushing [==================================================>] 86.27 MB/86.27 MB
```
Then, a few minutes later ( this is happening now, 11:18GMT, if you're looking in your log files) this:
```
67aca2b232ea: Pushing [==================================================>] 180.6 MB
3ac0d1046245: Pushing [==================================================>] 86.27 MB/86.27 MB
2c2153fbd032: Pushing [==================================================>] 8.102 MB
```
Then it looks like it's worked:
```
67aca2b232ea: Pushed
```
Then::
```
Sun 23 Oct 2016 13:27:42 SAST Run time: 0 days 1 hours 23 minutes 0 seconds
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
```
When I restart, with the same image, though:
```
67aca2b232ea: Pushing [=> ] 5.407 MB/177 MB
```
Any thoughts about this one? It is making it very difficult to make progress.
The main problem is that 177Mb is just too big to load in 20 minutes - I'm not sure why, because it's easy to download five times that in less than 20 minutes.
If it was possible to split the chunks, it'd solve the problem. I've got it working, and tried it a few times now - it seems much the same. The timeout is 32 minutes, rather than 19. I've tried modifying various tcp timeout options, but it doesn't seem to have made much difference. As you can see, I've tried it a number of times to be sure.
```
+ logger Starting docker push -- Oct 20 15:25:59 govern peterbrooks: Starting docker push
+ docker push fustbariclation/fustbariclation
The push refers to a repository [docker.io/fustbariclation/fustbariclation]
93d8703dde2a: Layer already exists
934396e52c9d: Layer already exists
8efaddf4ca67: Layer already exists
ffd427b1c79a: Layer already exists
92313c584362: Retrying in 1 second
4a4f58055766: Retrying in 1 second
adb25476d2c1: Retrying in 1 second
655a6c01124a: Pushing [==========================================> ] 175.8 MB/205.9 MB
0de3b0993fca: Retrying in 1 second
2253b7eea5e6: Retrying in 1 second
05b90b2afe02: Retrying in 1 second
ec96d31f8eaf: Retrying in 1 second
e3ac10d00c60: Retrying in 1 second
2c2153fbd032: Retrying in 1 second
d9a069c1d0fc: Retrying in 1 second
a5eb0fc1decb: Retrying in 1 second
b2ac5371e0f2: Retrying in 1 second
142a601d9793: Retrying in 1 second
net/http: TLS handshake timeout
+ logger Docker push stopped -- Oct 20 15:57:54 govern peterbrooks: Docker push stopped
```
These are the tcp parameters I tried modifying in /etc/sysctl.conf -- but, as I said, it didn't seem to make any difference. If you've any suggestions for a good setting, that'd be great!
```
net.ipv4.tcp_mtu_probing=1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.xfrm_aevent_etime = 50
net.ipv4.conf.docker0.accept_local = 0
net.ipv4.conf.docker0.accept_redirects = 1
net.ipv4.conf.docker0.accept_source_route = 1
net.ipv4.conf.docker0.arp_accept = 0
net.ipv4.conf.docker0.arp_announce = 0
net.ipv4.conf.docker0.arp_filter = 0
net.ipv4.conf.docker0.arp_ignore = 0
net.ipv4.conf.docker0.arp_notify = 0
net.ipv4.conf.docker0.bootp_relay = 0
net.ipv4.conf.docker0.disable_policy = 0
net.ipv4.conf.docker0.disable_xfrm = 0
net.ipv4.conf.docker0.force_igmp_version = 0
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.docker0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.docker0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.docker0.ignore_routes_with_linkdown = 0
net.ipv4.conf.docker0.log_martians = 0
net.ipv4.conf.docker0.mc_forwarding = 0
net.ipv4.conf.docker0.medium_id = 0
net.ipv4.conf.docker0.promote_secondaries = 0
net.ipv4.conf.docker0.proxy_arp = 0
net.ipv4.conf.docker0.proxy_arp_pvlan = 0
net.ipv4.conf.docker0.route_localnet = 0
net.ipv4.conf.docker0.rp_filter = 1
net.ipv4.conf.docker0.secure_redirects = 1
net.ipv4.conf.docker0.send_redirects = 1
net.ipv4.conf.docker0.shared_media = 1
net.ipv4.conf.docker0.src_valid_mark = 0
net.ipv4.conf.docker0.tag = 0
net.ipv4.ipfrag_time = 50
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_stale_time = 600
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.retrans_time_ms = 3000
net.ipv4.neigh.docker0.anycast_delay = 400
net.ipv4.neigh.docker0.app_solicit = 0
net.ipv4.neigh.docker0.base_reachable_time_ms = 90000
net.ipv4.neigh.docker0.delay_first_probe_time = 1
net.ipv4.neigh.docker0.delay_first_probe_time = 5
net.ipv4.neigh.docker0.gc_stale_time = 600
net.ipv4.neigh.docker0.gc_stale_time = 600S
net.ipv4.neigh.docker0.locktime = 100
net.ipv4.neigh.docker0.locktime = 1000
net.ipv4.neigh.docker0.mcast_resolicit = 0
net.ipv4.neigh.docker0.mcast_solicit = 3
net.ipv4.neigh.docker0.proxy_delay = 80
net.ipv4.neigh.docker0.proxy_qlen = 64
net.ipv4.neigh.docker0.retrans_time_ms = 9000
net.ipv4.neigh.docker0.ucast_solicit = 3
net.ipv4.neigh.docker0.unres_qlen = 31
net.ipv4.neigh.docker0.unres_qlen_bytes = 65536
net.ipv4.neigh.enp0s3.base_reachable_time_ms = 30000
net.ipv4.neigh.enp0s3.delay_first_probe_time = 5
net.ipv4.neigh.enp0s3.gc_stale_time = 60
net.ipv4.neigh.enp0s3.locktime = 100
net.ipv4.neigh.enp0s3.retrans_time_ms = 1000
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.route.gc_timeout = 300
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_sack = 1
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv6.conf.docker0.accept_dad = 1
net.ipv6.conf.docker0.accept_ra = 1
net.ipv6.conf.docker0.accept_ra_defrtr = 1
net.ipv6.conf.docker0.accept_ra_from_local = 0
net.ipv6.conf.docker0.accept_ra_min_hop_limit = 1
net.ipv6.conf.docker0.accept_ra_mtu = 1
net.ipv6.conf.docker0.accept_ra_pinfo = 1
net.ipv6.conf.docker0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.docker0.accept_ra_rtr_pref = 1
net.ipv6.conf.docker0.accept_redirects = 1
net.ipv6.conf.docker0.accept_source_route = 0
net.ipv6.conf.docker0.autoconf = 1
net.ipv6.conf.docker0.dad_transmits = 1
net.ipv6.conf.docker0.disable_ipv6 = 0
net.ipv6.conf.docker0.force_mld_version = 0
net.ipv6.conf.docker0.force_tllao = 0
net.ipv6.conf.docker0.forwarding = 0
net.ipv6.conf.docker0.hop_limit = 64
net.ipv6.conf.docker0.ignore_routes_with_linkdown = 0
net.ipv6.conf.docker0.max_addresses = 16
net.ipv6.conf.docker0.max_desync_factor = 600
net.ipv6.conf.docker0.mc_forwarding = 0
net.ipv6.conf.docker0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.docker0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.docker0.mtu = 4096
net.ipv6.conf.docker0.ndisc_notify = 0
net.ipv6.conf.docker0.proxy_ndp = 0
net.ipv6.conf.docker0.regen_max_retry = 9
net.ipv6.conf.docker0.router_probe_interval = 600
net.ipv6.conf.docker0.router_solicitation_delay = 1
net.ipv6.conf.docker0.router_solicitation_interval = 4
net.ipv6.conf.docker0.router_solicitations = 3
net.ipv6.conf.docker0.suppress_frag_ndisc = 1
net.ipv6.conf.docker0.temp_prefered_lft = 86400
net.ipv6.conf.docker0.temp_valid_lft = 604800
net.ipv6.conf.docker0.use_oif_addrs_only = 0
net.ipv6.conf.docker0.use_tempaddr = 2
net.ipv6.neigh.docker0.anycast_delay = 100
net.ipv6.neigh.docker0.app_solicit = 0
net.ipv6.neigh.docker0.base_reachable_time_ms = 90000
net.ipv6.neigh.docker0.delay_first_probe_time = 5
net.ipv6.neigh.docker0.gc_stale_time = 600
net.ipv6.neigh.docker0.locktime = 0
net.ipv6.neigh.docker0.mcast_resolicit = 0
net.ipv6.neigh.docker0.mcast_solicit = 3
net.ipv6.neigh.docker0.proxy_delay = 80
net.ipv6.neigh.docker0.proxy_qlen = 128
net.ipv6.neigh.docker0.retrans_time_ms = 9000
net.ipv6.neigh.docker0.ucast_solicit = 3
net.ipv6.neigh.docker0.unres_qlen = 31
net.ipv6.neigh.docker0.unres_qlen_bytes = 65536
``` Peter:
I apologize for my misleading comment: The address hub.docker.com is used in your web browser but not with the Docker client. With Docker when you **omit** the registry address you are referring to DockerHub by default:
```
[~] docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (brucehoff): brucehoff
Password:
Login Succeeded
[~]
```
You name the image with your DockerHub user name followed by the repository name, e.g.
```
docker build -t fustbariclation/your-inspired-model .
```
Then push to DockerHub
```
docker push fustbariclation/your-inspired-model
```
Would you please let me know if this works for you?
I've created a repository there. Unfortunately I get a different problem - I can't find a solution on-line. This is how it goes.
As you can see, everything seems to work, it even looks as if it's trying to push, but, after a couple of seconds it gives the error:
```
Index response didn't contain any endpoints
```
```
root@govern:/media/sf_mammogram/blue# docker login hub.docker.com
Username (fustbariclation): fustbariclation
Password:
Login Succeeded
root@govern:/media/sf_mammogram/blue# docker build -t hub.docker.com/fustbariclation/fustbariclation .
Sending build context to Docker daemon 110.1 kB
Step 1 : FROM python
---> 1d0326469b55
Step 2 : RUN date
---> Using cache
---> 2c4f8b9e35be
Step 3 : RUN pip install pydicom
---> Using cache
---> 1b14765d4688
Step 4 : RUN pip install Image
---> Using cache
---> 010062ae6c6d
Step 5 : RUN pip install cm
---> Using cache
---> ba770065980c
Step 6 : RUN pip install scipy
---> Using cache
---> 4ee14dc8f763
Step 7 : RUN pip install numpy
---> Using cache
---> 22a73112174a
Step 8 : RUN pip install setuptools
---> Using cache
---> 8aff5e4216a0
Step 9 : RUN apt-get update --fix-missing && apt-get install -y --fix-missing r-base
---> Using cache
---> 3a07386fba61
Step 10 : COPY train.py /train.py
---> Using cache
---> 5cc4c88f89e0
Step 11 : COPY train.sh /train.sh
---> Using cache
---> 7e2843466f0d
Step 12 : COPY test.sh /test.sh
---> Using cache
---> c67abe918ab7
Step 13 : COPY preprocess.sh /preprocess.sh
---> Using cache
---> e1833a96b9d8
Step 14 : RUN /bin/df -H
---> Using cache
---> e6985005a2bb
Step 15 : RUN ls -lt
---> Using cache
---> 2382a1ab1087
Step 16 : RUN echo "Completed initialise phase - now running preprocess.sh"
---> Using cache
---> f99dd0f01931
Step 17 : RUN date
---> Using cache
---> 95a88fbfbe86
Step 18 : RUN /preprocess.sh
---> Using cache
---> c81bf9f489f9
Step 19 : RUN date
---> Using cache
---> 4810f9fcaf0e
Step 20 : RUN echo "End of Dockerfile"
---> Using cache
---> 33778bef622c
Successfully built 33778bef622c
root@govern:/media/sf_mammogram/blue# docker push hub.docker.com/fustbariclation/fustbariclation:latest
The push refers to a repository [hub.docker.com/fustbariclation/fustbariclation]
93d8703dde2a: Preparing
934396e52c9d: Preparing
8efaddf4ca67: Preparing
ffd427b1c79a: Preparing
92313c584362: Preparing
4a4f58055766: Waiting
adb25476d2c1: Waiting
655a6c01124a: Waiting
0de3b0993fca: Waiting
2253b7eea5e6: Waiting
05b90b2afe02: Waiting
ec96d31f8eaf: Waiting
e3ac10d00c60: Waiting
2c2153fbd032: Waiting
d9a069c1d0fc: Waiting
a5eb0fc1decb: Waiting
b2ac5371e0f2: Waiting
142a601d9793: Waiting
Index response didn't contain any endpoints
``` Peter:
Do you have the same or different experience when you push to DockerHub (https://hub.docker.com/) rather than Synapse? The answer could help determine whether the problem lies with our Docker registry, with your client, or with the network connectivity.
Thank you.