I'm just wondering if there's another way to submit things to docker. The problems I'm getting seem to be related to timeouts, and that seems to be connected to my location and infrastructure. I've submitted a report of the problem to docker, and they don't seem to know what is causing it. I'm going to try submitting it to them, but it also takes a long time, and, during that time, I'm not pushing the files onto your system.
Is there a server that I could, perhaps, use, that's nearer to the synapse network, that I could, perhaps, have an account on? I could then copy over the few set up files, a job of minutes, and do the docker push from there.
It's quite frustrating to have to watch it pushing the images - each image is taking several days. So it's a week or ten days to get it onto your system, before it can run.
Alternatively, could I send the files to you, so that you could do the push on my behalf?
I don't know if anybody else is having this problem.
Created by Peter Brooks fustbariclation Thank you!
I've now setup an account with DigitalOcian, and pushed everything up - all in a few minutes.
All working perfectly!
I look forward to getting the results.... Hi: I'm sorry for these difficulties. I'll try to make a few helpful comments.
> It's quite frustrating to have to watch it pushing the images - each image is taking several days.
Pushing an image should be very quick, a few minutes at the most. If it takes several days something is seriously wrong.
> Is there a server that I could, perhaps, use, that's nearer to the synapse network
Yes, that's a fine idea. At the current time our Docker registry runs on machines hosted by Amazon Web Services, in its eastern US region. If you were to 'spin up' an Amazon machine, log in and do your work there, then pushing to our registry would likely go fast. You can even stop the machine when not in use to help control your budgetary expenses, without losing data.
> the Docker Tab on Synapse has a button "Add External Repositories". How does that feature work with external private repositories?
That's a good question. In a separate discussion thread we established that Peter's link is as slow to DockerHub (the most popular Docker registry) as to the Synapse Docker registry. So I don't think it would solve the problem. However to answer your question the procedure is:
1) Push to DockerHub and note the tag name and digest (SHA 256) of the commit;
2) In Synapse create an 'external' repository, entering the DockerHub repository name and also the commit (tag and digest);
3) In DockerHub share the repository with the DockerHub user 'dreamchall'.
You can now submit to Synapse and our system will pull your image from DockerHub. Note: What's going on here is that Synapse records your external repository information as a convenience. Synapse does not actually communicate in any way with DockerHub or any other external Docker registry.
> I've asked Bruce to see if we can change the TLS handshake timeout but this may take a few days.
I'm not exactly sure what "TLS handshake timeout" refers to, but timeouts usually are imposed by *clients* not *servers* so I would expect this to be something on Peter's end (e.g. in his Docker client.).
> Peter. Get a cheap DigitalOcean box in NYC. Problem solved.
Certainly AWS is not the only cloud provider. Any commodity cloud machine you can ssh into, having the Docker client installed and having good bandwidth to the US East should help get past this problem.
Thank you for being tenacious and please keep us posted!
Peter. Get a cheap DigitalOcean box in NYC. Problem solved. No, I don't. I have access to a server in the UK, but it runs under cpanel, and docker, and git, need an interactive shell.
I agree the link seems very unreliable, with the timeouts. I've tried to make it as clear as possible - I've stopped everything like google drive, dropbox, apple icloud, and so forth, but the timeouts still continue.
I think it would be a great help if it was possible to increase the timeouts. If you knew of a machine I could access temporarily, for this, that would be extremely helpful too. Thanks Peter, I've just add a look at the log you sent me. It seems to me that your connection is not very reliable (TLS handshake takes too long). I've asked Bruce to see if we can change the TLS handshake timeout but this may take a few days.
Do you have access to a remote machine which you can connect to and `docker pull`from there? I've rebuilt
docker.synapse.org/syn7364819/blue_training
I've changed the DNS servers.
I've tried uploading to google hub - but I get the same error 'Index response didn't contain any endpoints'
Usually I don't halt the push - it terminates with a timeout error. This used to be after exactly 19 minutes, but now it varies between longer intervals. I've kept the intervals:
I'll upload the log and share it with you, along with the speed test, and the log of the intervals and reason for push failure. Do you mind trying using Google DNS 8.8.8.8 and 8.8.4.4 (https://developers.google.com/speed/public-dns/)?
Can you share privately with me the output of the upload speed test from http://testmy.net/ after setting up Google DNS? OK. I've rebuilt it, and the docker hub version.
I've retried the synapse version and the errors are the same:
```
time="2016-10-25T20:52:21.075430842Z" level=debug msg="Pushing layer: sha256:a0dc2eaec3c374bcdd40b18e4020374bb392224cdf090ab912ebd1cbb2945333"
time="2016-10-25T20:52:21.075531535Z" level=error msg="Upload failed, retrying: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2016-10-25T20:52:26.079930658Z" level=debug msg="Pushing layer: sha256:ca20b906b87c8498d6e9759ea4fa0f3283ac83006e2db468761831706966543f"
time="2016-10-25T20:52:36.077231769Z" level=error msg="Upload failed, retrying: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
```
With the docker hub version, I'm getting this error, at once:
```
sh-3.2# docker logout
Remove login credentials for https://index.docker.io/v1/
sh-3.2# docker login hub.docker.com
Username (fustbariclation): fustbariclation
Password:
Login Succeeded
sh-3.2# docker push hub.docker.com/fustbariclation/blue_training
The push refers to a repository [hub.docker.com/fustbariclation/blue_training]
d46afcd68ca7: Preparing
a2eb9966230c: Preparing
c2ade7efa02d: Preparing
c998354a0a8a: Preparing
67aca2b232ea: Preparing
b0afbd3c7b31: Waiting
38ece981f0d9: Waiting
3ac0d1046245: Waiting
4ce5da20b321: Waiting
a0dc2eaec3c3: Waiting
ca20b906b87c: Waiting
30596d6a4f4c: Waiting
ec96d31f8eaf: Waiting
e3ac10d00c60: Waiting
2c2153fbd032: Waiting
d9a069c1d0fc: Waiting
a5eb0fc1decb: Waiting
b2ac5371e0f2: Waiting
142a601d9793: Waiting
Index response didn't contain any endpoints
```
I'll leave the push running for the next few hours, in case it succeeds.
Here is the error:
```
time="2016-10-25T18:45:41.735402052Z" level=debug msg="Pushing layer: sha256:30596d6a4f4c7b351708a9e233c9becbb1000801de46aa3d0df5ce31f0fd7ba3"
time="2016-10-25T18:45:56.740957585Z" level=error msg="Upload failed, retrying: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2016-10-25T18:46:01.743254671Z" level=debug msg="Pushing layer: sha256:30596d6a4f4c7b351708a9e233c9becbb1000801de46aa3d0df5ce31f0fd7ba3"
time="2016-10-25T18:46:16.748723050Z" level=error msg="Upload failed, retrying: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time="2016-10-25T18:46:26.750990627Z" level=debug msg="Pushing layer: sha256:30596d6a4f4c7b351708a9e233c9becbb1000801de46aa3d0df5ce31f0fd7ba3"
time="2016-10-25T18:46:40.974658336Z" level=error msg="Upload failed, retrying: net/http: TLS handshake timeout"
```
Before going deeper, can you test a few things?
- `docker rmi docker.synapse.org/syn7364819/blue_training`, rebuild it and retry
- send the same image to DockerHub (as a private repository, 1 private repository included in the free tier) Thank you for looking at it - I've shared it with you.
I'm on a Mac, so it's in a different place, but it's the same log, I believe.
I did, by the way, try using linux instead, for a while. I had the same problems, and thought it might be easier to work out what was going wrong with just the one machine - not one machine hosted on another.
I've also added the other log files for you, to save time, if you need a bigger picture.
Can you try the following command to get additional information from Docker, upload the log file to Synapse and share it with me?
```
journalctl -u docker --since today > docker.log
``` That's an interesting idea. I had a look at that, but I couldn't see quite how it worked.
The push looks as if it might complete some time tomorrow. Mostly, what slows it down is, for some reason, the segment '67aca2b232ea', it pushes it until it gets to 177MB - then goes on to about 180MB, and then aborts and goes back to 'Retrying in 1 second', only to re-load it all over again. It does the same with some others, but that seems to be the one it dislikes most.
The output of the push, at this moment, is:
```
Tue 25 Oct 2016 18:59:38 SAST
The push refers to a repository [docker.synapse.org/syn7364819/blue_training]
d46afcd68ca7: Pushed
a2eb9966230c: Pushed
c2ade7efa02d: Pushed
c998354a0a8a: Retrying in 1 second
67aca2b232ea: Pushing [====> ] 16.31 MB/177 MB
b0afbd3c7b31: Pushing [=> ] 2.214 MB/103.6 MB
38ece981f0d9: Retrying in 1 second
3ac0d1046245: Retrying in 1 second
4ce5da20b321: Waiting
a0dc2eaec3c3: Waiting
ca20b906b87c: Waiting
30596d6a4f4c: Waiting
ec96d31f8eaf: Waiting
e3ac10d00c60: Waiting
2c2153fbd032: Waiting
d9a069c1d0fc: Waiting
a5eb0fc1decb: Waiting
b2ac5371e0f2: Waiting
142a601d9793: Waiting
```
Hi Peter,
@brucehoff, the Docker Tab on Synapse has a button "Add External Repositories". How does that feature work with external private repositories? Peter could upload his Docker images to DockerHub as private repositories and then "softly" imported them to his private Synapse project.
Peter, can you show the output of the `docker push`command?