trying to download on the CLI
`synapse get -r syn21560412`
It stalls after 7-8GBs. Same command, same study, smaller folder downloaded with no problems.
Created by ghhenry Hmm, to close the loop here I was unable to reproduce this even using the conda environment from above (see below).
I did notice than in a **conda list** in a previous reply you had a "synapse" package installed from pypi which is a separate unrelated piece of software and thought that it might be causing some namespace collision issues or similar, but installing that doesn't reproduce the issue either.
```
(base) ubuntu@linux1:/# echo "name: synapse
> channels:
> - bioconda
> - r
> - conda-forge
> - defaults
> dependencies:
> - _libgcc_mutex=0.1=conda_forge
> - _openmp_mutex=4.5=1_gnu
> - ca-certificates=2020.11.8=ha878542_0
> - certifi=2020.11.8=py36h5fab9bb_0
> - ld_impl_linux-64=2.35.1=hed1e6ac_0
> - libffi=3.2.1=he1b5a44_1007
> - libgcc-ng=9.3.0=h5dbcf3e_17
> - libgomp=9.3.0=h5dbcf3e_17
> - libstdcxx-ng=9.3.0=h2ae2ef3_17
> - ncurses=6.2=h58526e2_4
> - openssl=1.1.1h=h516909a_0
> - pip=20.2.4=py_0
> - python=3.6.11=h4d41432_2_cpython
> - python_abi=3.6=1_cp36m
> - readline=8.0=he28a2e2_2
> - setuptools=49.6.0=py36h9880bd3_2
> - sqlite=3.33.0=h4cf870e_1
> - tk=8.6.10=hed695b0_1
> - wheel=0.35.1=pyh9f0ad1d_0
> - xz=5.2.5=h516909a_1
> - zlib=1.2.11=h516909a_1010" > environment.yml
(base) ubuntu@linux1:/# conda env create -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate synapse
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) ubuntu@linux1:/# conda activate synapse
(synapse) ubuntu@linux1:/# synapse --debug get -r syn21560412
Synapse username: jordank
b'Password or api key for jordank: '
Welcome, Jordan Kiang!
2020-12-09 23:43:29,557 [client:409 - INFO]: Welcome, Jordan Kiang!
Downloading [####################]100.00% 1.4GB/1.4GB (32.0MB/s) facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20200616.RC4.Robj.synapse_download_54557964 Done...
Downloading [####################]100.00% 1.6GB/1.6GB (33.8MB/s) droplet_normal_lung_blood_seurat_ntiss10x.P1.anno.20191002.RC4.Robj.synapse_download_49506559 Done...
Downloading [####################]100.00% 617.0MB/617.0MB (32.6MB/s) facs_normal_lung_blood_seurat_ntiss.P3.anno.gencode.20200616.RC4.Robj.synapse_download_54557870 Done...
Downloading [####################]100.00% 933.6MB/933.6MB (34.6MB/s) facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937 Done...
Downloading [####################]100.00% 1001.1MB/1001.1MB (31.4MB/s) mouse_facs_combined_tiss.mouse.anno.ncbi.20200121.RC4.Robj.synapse_download_52271626 Done...
Downloading [####################]100.00% 4.5GB/4.5GB (34.5MB/s) droplet_normal_lung_blood_seurat_ntiss10x.P3.anno.20191002.RC4.Robj.synapse_download_49506244 Done...
Downloading [####################]100.00% 4.7GB/4.7GB (35.3MB/s) droplet_normal_lung_seurat_ntiss10x.P2.anno.20191002.RC4.Robj.synapse_download_49506293 Done...
Downloading [####################]100.00% 1.4GB/1.4GB (28.7MB/s) facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20191002.RC4.Robj.synapse_download_48601559 Done...
Downloading [####################]100.00% 3.4GB/3.4GB (25.1MB/s) mouse_droplet_TMS_tiss10x.mouse.anno.ncbi.20200222.RC4.Robj.synapse_download_49506717 Done...
Downloading [####################]100.00% 951.2MB/951.2MB (18.1MB/s) facs_normal_lung_seurat_ntiss.P2.anno.gencode.20191002.RC4.Robj.synapse_download_48601561 Done...
Downloading [####################]100.00% 634.6MB/634.6MB (34.3MB/s) facs_normal_lung_seurat_ntiss.P3.anno.gencode.20191002.RC4.Robj.synapse_download_49506012 Done...
- Downloaded 21.0GB (95.5MB/s)(synapse) ubuntu@linux1:/#
```
Sure...
PC:
```
name: synapse
channels:
- defaults
dependencies:
- _libgcc_mutex=0.1=main
- ca-certificates=2020.10.14=0
- certifi=2020.11.8=py37h06a4308_0
- ld_impl_linux-64=2.33.1=h53a641e_7
- libedit=3.1.20191231=h14c3975_1
- libffi=3.3=he6710b0_2
- libgcc-ng=9.1.0=hdf63c60_0
- libstdcxx-ng=9.1.0=hdf63c60_0
- ncurses=6.2=he6710b0_1
- openssl=1.1.1h=h7b6447c_0
- pip=20.3=py37h06a4308_0
- python=3.7.9=h7579374_0
- readline=8.0=h7b6447c_0
- setuptools=50.3.1=py37h06a4308_1
- sqlite=3.33.0=h62c20be_0
- tk=8.6.10=hbc83047_0
- wheel=0.35.1=pyhd3eb1b0_0
- xz=5.2.5=h7b6447c_0
- zlib=1.2.11=h7b6447c_3
- pip:
- bcrypt==3.2.0
- cffi==1.14.4
- chardet==3.0.4
- cryptography==3.2.1
- deprecated==1.2.10
- entrypoints==0.3
- idna==2.10
- keyring==12.0.2
- keyrings.alt==3.1
- numpy==1.19.4
- pandas==1.1.4
- paramiko==2.7.2
- pycparser==2.20
- pynacl==1.4.0
- pysftp==0.2.9
- python-dateutil==2.8.1
- pytz==2020.4
- requests==2.25.0
- secretstorage==2.3.1
- six==1.15.0
- synapseclient==2.2.2
- urllib3==1.26.2
- wrapt==1.12.1
prefix: /home/ghenry/anaconda3/envs/synapse
```
HPC:
```
name: synapse
channels:
- bioconda
- r
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_gnu
- ca-certificates=2020.11.8=ha878542_0
- certifi=2020.11.8=py36h5fab9bb_0
- ld_impl_linux-64=2.35.1=hed1e6ac_0
- libffi=3.2.1=he1b5a44_1007
- libgcc-ng=9.3.0=h5dbcf3e_17
- libgomp=9.3.0=h5dbcf3e_17
- libstdcxx-ng=9.3.0=h2ae2ef3_17
- ncurses=6.2=h58526e2_4
- openssl=1.1.1h=h516909a_0
- pip=20.2.4=py_0
- python=3.6.11=h4d41432_2_cpython
- python_abi=3.6=1_cp36m
- readline=8.0=he28a2e2_2
- setuptools=49.6.0=py36h9880bd3_2
- sqlite=3.33.0=h4cf870e_1
- tk=8.6.10=hed695b0_1
- wheel=0.35.1=pyh9f0ad1d_0
- xz=5.2.5=h516909a_1
- zlib=1.2.11=h516909a_1010
prefix: /home2/ghenry/.conda/envs/synapse
``` Ok great. If you still feel like doing so feel free to post a **conda env export** and I'll see if it reproduces with a more exactly replicated conda environment. Oh, good to know re-connection resets... I was surprised by them, but good to know they are recoverable... I guess I stopped the recursive get for no reason then. Interestingly the second time around isn't resulting in any.
It is still downloading now... but it has gotten super far... with the fear of jinxing myself, I think I'm not having md5 error doing it on EC2.
I think that means that there is probably some weird environment specific library which is causing the problem... that makes it even weirder that I'm having the same problem on my local system also! Ahh well, at least I found a way to make the download work!
Thank you so much!!
Gervaise @ghhenry
with the --debug option a ConnectionReset showing doesn't necessarily mean the download failed, it's among the recoverable errors that the Synapse client will retry on but it being output to the console because debug was enabled although it's a bit surprising to see it on EC2 since EC2 to S3 should be a very stable environment.
The MD5 errors on the other hand are not going to recover properly if they reproduce. Now on EC2 getting connection resets...
```
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 753, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib64/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
```
the only thing on the EC2 instance is python3, pip, synapseclient... (gave it a 100GB drive)
...trying again No need to apologize, I have a knack for finding edge case errors... for better or worse.
I'll try the Docker container... though I will run it with Singularity (my HPC only allows us to run Docker containers through Singularity.
I'm also goin to try and download it using an EC2 machine to see if I can get it that way.
Gervaise @ghhenry
I tried downloading from a conda environment and replicating most of the packages (I realize didn't ask for a **conda env export** but I installed the packages manually), and still wasn't able to reproduce the issue from within EC2 or locally on my own Mac device.
Sorry for so much back and forth but it's difficult to address the issue until I can reproduce it.
Can you see if it reproduces within a docker container so we know we are running the exact same environment?
I used the following Dockerfile to create an otherwise empty conda environment:
```
FROM continuumio/miniconda3
RUN pip install synapseclient
CMD /bin/bash
```
Full steps were as follows. I repeated this multiple times to see if there was some variability.
From the console:
```
# generate Dockerfile
printf "FROM continuumio/miniconda3\nRUN pip install synapseclient\nCMD /bin/bash\n" > Dockerfile
# build the image
docker build .
# start the image (last image built)
docker run -ti $(docker images --format='{{.ID}}' | head -1)
```
My output:
```
ubuntu:/tmp/docker$ printf "FROM continuumio/miniconda3\nRUN pip install synapseclient\nCMD /bin/bash\n" > Dockerfile
ubuntu:/tmp/docker$ docker build .
Sending build context to Docker daemon 4.096kB
Step 1/3 : FROM continuumio/miniconda3
---> edd40e0b2f30
Step 2/3 : RUN pip install synapseclient
---> Using cache
---> f02250bac0cd
Step 3/3 : CMD /bin/bash
---> Using cache
---> e00bfca65200
Successfully built e00bfca65200
ubuntu:/tmp/docker$ docker run -ti $(docker images --format='{{.ID}}' | head -1)
```
From within the container:
```
(base) root@5e7583e83c95:/#
(base) root@5e7583e83c95:/# synapse --debug get -r syn21560412
Synapse username: jordank
b'Password or api key for jordank: '
Welcome, Jordan Kiang!
2020-12-02 23:20:07,745 [client:409 - INFO]: Welcome, Jordan Kiang!
Downloading [####################]100.00% 1.4GB/1.4GB (29.6MB/s) facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20200616.RC4.Robj.synapse_download_54557964 Done...
Downloading [####################]100.00% 1.6GB/1.6GB (30.3MB/s) droplet_normal_lung_blood_seurat_ntiss10x.P1.anno.20191002.RC4.Robj.synapse_download_49506559 Done...
Downloading [####################]100.00% 933.6MB/933.6MB (32.7MB/s) facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937 Done...
Downloading [####################]100.00% 617.0MB/617.0MB (31.2MB/s) facs_normal_lung_blood_seurat_ntiss.P3.anno.gencode.20200616.RC4.Robj.synapse_download_54557870 Done...
Downloading [####################]100.00% 1001.1MB/1001.1MB (30.7MB/s) mouse_facs_combined_tiss.mouse.anno.ncbi.20200121.RC4.Robj.synapse_download_52271626 Done...
Downloading [####################]100.00% 4.5GB/4.5GB (33.0MB/s) droplet_normal_lung_blood_seurat_ntiss10x.P3.anno.20191002.RC4.Robj.synapse_download_49506244 Done...
Downloading [####################]100.00% 4.7GB/4.7GB (33.2MB/s) droplet_normal_lung_seurat_ntiss10x.P2.anno.20191002.RC4.Robj.synapse_download_49506293 Done...
Downloading [####################]100.00% 1.4GB/1.4GB (40.3MB/s) facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20191002.RC4.Robj.synapse_download_48601559 Done...
Downloading [####################]100.00% 3.4GB/3.4GB (26.6MB/s) mouse_droplet_TMS_tiss10x.mouse.anno.ncbi.20200222.RC4.Robj.synapse_download_49506717 Done...
Downloading [####################]100.00% 951.2MB/951.2MB (15.9MB/s) facs_normal_lung_seurat_ntiss.P2.anno.gencode.20191002.RC4.Robj.synapse_download_48601561 Done...
Downloading [####################]100.00% 634.6MB/634.6MB (34.1MB/s) facs_normal_lung_seurat_ntiss.P3.anno.gencode.20191002.RC4.Robj.synapse_download_49506012 Done...
- Downloaded 21.0GB (96.2MB/s)(base) root@5e7583e83c95:/#
```
If the above successfully downloads, then can you conda env export one of your conda environments from which it fails, I will add that to the docker image and see if it then reproduces. And just for confirmation, I just did a single download on my local PC (yesterday I only tried recursive) and it happened again. Doesn't seem to happen on small files.
`synapse --debug get syn21560469`
```
File "/home/ghenry/anaconda3/envs/synapse/bin/synapse", line 8, in
sys.exit(main())
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/__main__.py", line 902, in main
perform_main(args, syn)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/__main__.py", line 852, in perform_main
args.func(args, syn)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/__main__.py", line 94, in get
downloadLocation=args.downloadLocation)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 665, in get
return self._getWithEntityBundle(entityBundle=bundle, entity=entity, **kwargs)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 781, in _getWithEntityBundle
self._download_file_entity(downloadLocation, entity, ifcollision, submission)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 843, in _download_file_entity
downloadPath = self._downloadFileHandle(entity.dataFileHandleId, objectId, objectType, downloadPath)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1847, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /mnt/d/download/assets/Analysis/r objects/droplet_normal_lung_blood_seurat_ntiss10x.P1.anno.20191002.RC4.Robj.synapse_download_49506559's md5 359748a24cae6b6e5680d28fb913d44f does not match expected MD5 of 5d4b0b24732004c626fff356b68559da
``` Done in conda environment with nothing really extra in there:
PC with Ubuntu WSL2:
```
(synapse) ghenry@SW529051:/mnt/c/Users/ghenry$ conda list
# packages in environment at /home/ghenry/anaconda3/envs/synapse:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
bcrypt 3.2.0
ca-certificates 2020.10.14 0
certifi 2020.11.8 py37h06a4308_0
cffi 1.14.4
chardet 3.0.4
cryptography 3.2.1
Deprecated 1.2.10
entrypoints 0.3
idna 2.10
keyring 12.0.2
keyrings.alt 3.1
ld_impl_linux-64 2.33.1 h53a641e_7
libedit 3.1.20191231 h14c3975_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libstdcxx-ng 9.1.0 hdf63c60_0
ncurses 6.2 he6710b0_1
numpy 1.19.4
openssl 1.1.1h h7b6447c_0
pandas 1.1.4
paramiko 2.7.2
pip 20.3 py37h06a4308_0
pycparser 2.20
PyNaCl 1.4.0
pysftp 0.2.9
python 3.7.9 h7579374_0
python-dateutil 2.8.1
pytz 2020.4
readline 8.0 h7b6447c_0
requests 2.25.0
SecretStorage 2.3.1
setuptools 50.3.1 py37h06a4308_1
six 1.15.0
sqlite 3.33.0 h62c20be_0
synapseclient 2.2.2
tk 8.6.10 hbc83047_0
urllib3 1.26.2
wheel 0.35.1 pyhd3eb1b0_0
wrapt 1.12.1
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
(synapse) ghenry@SW529051:/mnt/c/Users/ghenry$ conda list --explicit
# This file may be used to create an environment using:
# $ conda create --name --file
# platform: linux-64
@EXPLICIT
https://repo.anaconda.com/pkgs/main/linux-64/_libgcc_mutex-0.1-main.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2020.10.14-0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ld_impl_linux-64-2.33.1-h53a641e_7.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-9.1.0-hdf63c60_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-9.1.0-hdf63c60_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.3-he6710b0_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.2-he6710b0_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/openssl-1.1.1h-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.5-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.11-h7b6447c_3.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20191231-h14c3975_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/readline-8.0-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.10-hbc83047_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.33.0-h62c20be_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/python-3.7.9-h7579374_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/certifi-2020.11.8-py37h06a4308_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/wheel-0.35.1-pyhd3eb1b0_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/setuptools-50.3.1-py37h06a4308_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pip-20.3-py37h06a4308_0.tar.bz2
```
Linux HPC
```
(synapse) [ghenry@Nucleus009 ref]$ conda list
# packages in environment at /home2/ghenry/.conda/envs/synapse:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
aiohttp 3.7.3
async-timeout 3.0.1
ca-certificates 2020.11.8 ha878542_0 conda-forge
certifi 2020.11.8 py36h5fab9bb_0 conda-forge
Deprecated 1.2.10
entrypoints 0.3
fastjsonschema 2.14.5
idna-ssl 1.1.0
keyring 12.0.2
keyrings.alt 3.1
lark-parser 0.8.9
ld_impl_linux-64 2.35.1 hed1e6ac_0 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge
libgomp 9.3.0 h5dbcf3e_17 conda-forge
libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge
lmdb 0.99
msgpack 0.6.2
multidict 5.0.2
ncurses 6.2 h58526e2_4 conda-forge
openssl 1.1.1h h516909a_0 conda-forge
pip 20.2.4 py_0 conda-forge
prompt-toolkit 3.0.8
Pygments 2.6.1
pyOpenSSL 19.1.0
python 3.6.11 h4d41432_2_cpython conda-forge
python_abi 3.6 1_cp36m conda-forge
readline 8.0 he28a2e2_2 conda-forge
regex 2020.11.13
SecretStorage 2.3.1
setuptools 49.6.0 py36h9880bd3_2 conda-forge
sqlite 3.33.0 h4cf870e_1 conda-forge
synapse 2.2.0
synapseclient 2.2.0
tk 8.6.10 hed695b0_1 conda-forge
tornado 6.1
typing-extensions 3.7.4.3
wcwidth 0.2.5
wheel 0.35.1 pyh9f0ad1d_0 conda-forge
wrapt 1.12.1
xxhash 1.4.4
xz 5.2.5 h516909a_1 conda-forge
yarl 1.6.3
zlib 1.2.11 h516909a_1010 conda-forge
(synapse) [ghenry@Nucleus009 ref]$ conda list --explicit
# This file may be used to create an environment using:
# $ conda create --name --file
# platform: linux-64
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2020.11.8-ha878542_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-9.3.0-h2ae2ef3_17.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.35.1-hed1e6ac_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libgomp-9.3.0-h5dbcf3e_17.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-1_gnu.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-9.3.0-h5dbcf3e_17.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libffi-3.2.1-he1b5a44_1007.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.2-h58526e2_4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/openssl-1.1.1h-h516909a_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.5-h516909a_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-h516909a_1010.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/readline-8.0-he28a2e2_2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.10-hed695b0_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.33.0-h4cf870e_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/python-3.6.11-h4d41432_2_cpython.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.6-1_cp36m.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/wheel-0.35.1-pyh9f0ad1d_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/certifi-2020.11.8-py36h5fab9bb_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/setuptools-49.6.0-py36h9880bd3_2.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pip-20.2.4-py_0.tar.bz2
``` @ghhenry
Hmm, that you have reproduced it in two separate evironments while we don't have other reports of this and I haven't been able to reproduce it is perplexing. While there have been some changes to the recursive download implementation recently, the single object download hasn't changed much in several versions and you are reproducing with the single object download.
Is it possible you have some other library in your conda environments that could be affecting things in an unexpected way? It looks like you are using conda in both the Linux and Windows environments. Can you show the output of the following in your Linux environment and I will try to reproduce it with the same package versions?
```
conda list
conda list --explicit
```
Thanks. Just tried it on local Windows PC with WSL2 Ubuntu python 3.7 and some worked but then the same problem...
```
(synapse) ghenry@SW529051:/mnt/d/download$ synapse --debug get -r syn21041850
Synapse username: ghhenry
b'Password or api key for ghhenry: '
Welcome, ghhenry!
2020-12-01 19:20:07,976 [client:410 - INFO]: Welcome, ghhenry!
Downloading [####################]100.00% 816.0kB/816.0kB (2.7MB/s) mouse_facs_combined_metadata.csv Done...
Downloading [####################]100.00% 3.7MB/3.7MB (7.2MB/s) mouse_droplet_TMS_metadata.csv Done...
Downloading [####################]100.00% 6.1MB/6.1MB (7.9MB/s) krasnow_hlca_facs_metadata.csv Done...
Downloading [####################]100.00% 12.2MB/12.2MB (2.3MB/s) weissman_blood_bulk_FPKM.csv.synapse_download_48601671 Done...
Downloading [####################]100.00% 16.6MB/16.6MB (1.2MB/s) krasnow_hlca_10x_metadata.csv.synapse_download_48601552 Done...
Downloading [####################]100.00% 301.0MB/301.0MB (1.8MB/s) mouse_facs_combined_counts.csv.synapse_download_52271661 Done...
Downloading [####################]100.00% 1.6GB/1.6GB (1.5MB/s) droplet_normal_lung_blood_seurat_ntiss10x.P1.anno.20191002.RC4.Robj.synapse_download_49506559 Done...
Downloading [####################]100.00% 3.2GB/3.2GB (1.6MB/s) krasnow_hlca_10x_UMIs.csv.synapse_download_48601633 Done...
Downloading [####################]100.00% 4.1GB/4.1GB (1.5MB/s) facs_normal_lung_blood_scanpy.20200619.RC4.h5ad.synapse_download_54621147 Done...
Downloading [####################]100.00% 1.4GB/1.4GB (1.3MB/s) facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20200616.RC4.Robj.synapse_download_54557964 Done...
Downloading [####################]100.00% 617.0MB/617.0MB (1.7MB/s) facs_normal_lung_blood_seurat_ntiss.P3.anno.gencode.20200616.RC4.Robj.synapse_download_54557870 Done...
| Downloaded 30.5GB (8.8MB/s)2020-12-01 20:19:31,197 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:32,072 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
/ Downloaded 30.5GB (8.8MB/s)2020-12-01 20:19:32,910 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:33,193 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:33,451 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:33,871 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:34,242 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:34,480 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:34,792 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:19:35,350 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
\ Downloaded 30.8GB (8.8MB/s)2020-12-01 20:20:06,330 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
| Downloaded 30.8GB (8.8MB/s)2020-12-01 20:20:07,065 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
/ Downloaded 30.8GB (8.8MB/s)2020-12-01 20:20:07,598 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
\ Downloaded 30.8GB (8.8MB/s)2020-12-01 20:20:08,317 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:08,811 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
\ Downloaded 30.8GB (8.8MB/s)2020-12-01 20:20:12,461 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:12,467 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:13,166 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:13,201 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
| Downloaded 30.9GB (8.8MB/s)2020-12-01 20:20:13,710 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:13,734 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:14,179 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:14,184 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
/ Downloaded 30.9GB (8.8MB/s)2020-12-01 20:20:14,673 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-12-01 20:20:14,720 [client:1809 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Downloading [####################]100.00% 933.6MB/933.6MB (1.6MB/s) facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937 Done...
- Downloaded 30.9GB (8.8MB/s)Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseutils/sync.py", line 259, in _sync_file
followLink=followLink,
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 665, in get
return self._getWithEntityBundle(entityBundle=bundle, entity=entity, **kwargs)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 781, in _getWithEntityBundle
self._download_file_entity(downloadLocation, entity, ifcollision, submission)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 843, in _download_file_entity
downloadPath = self._downloadFileHandle(entity.dataFileHandleId, objectId, objectType, downloadPath)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1792, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/client.py", line 1847, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /mnt/d/download/assets/Analysis/scanpy objects/facs_normal_lung_blood_scanpy.20200205.RC4.h5ad.synapse_download_49507012's md5 9081445341ea23868e14ee8f228e9c8a does not match expected MD5 of a4625e72d748456f53d62e9237ada2d2
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ghenry/anaconda3/envs/synapse/bin/synapse", line 8, in
sys.exit(main())
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/__main__.py", line 902, in main
perform_main(args, syn)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/__main__.py", line 852, in perform_main
args.func(args, syn)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseclient/__main__.py", line 77, in get
synapseutils.syncFromSynapse(syn, args.id, args.downloadLocation, followLink=args.followLink)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseutils/sync.py", line 96, in syncFromSynapse
files = sync_from_synapse.sync(entity, path, ifcollision, followLink)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseutils/sync.py", line 228, in sync
root_folder_sync = self._sync_root(entity, path, ifcollision, followLink, progress)
File "/home/ghenry/anaconda3/envs/synapse/lib/python3.7/site-packages/synapseutils/sync.py", line 305, in _sync_root
raise ValueError("File download failed during sync") from exception
ValueError: File download failed during sync
``` even the XFS gave the same MD5 not matching error Lustre fs
```
(synapse) [ghenry@Nucleus009 ref]$ stat -f -c %T .
lustre
(synapse) [ghenry@Nucleus009 ref]$ df .
Filesystem 1K-blocks Used Available Use% Mounted on
10.100.164.11@o2ib:10.100.164.12@o2ib:/project 3797470430136 3011078988468 748037739456 81% /project
```
couldn't mount because I don't have root access (HPC)
I am trying to download into /tmp non the less which is a XFS @ghhenry
Hmmm. What operating system is this (I'm guessing a Linux distribution based on some the file paths), and what is the filesystem that it is being downloaded to?
On Linux info about the current disk/file system e.g. (from the intended download path):
```
stat -f -c %T .
df .
```
On Linux if the failed download for syn21560469 is reproducible and you have 2GB of memory to allocate we could test mounting a tmpfs volume in memory and downloading to that instead. If we can reproduce the error there then I think we can rule out the disk being related altogether, conversely if it's not reproducible there then it might still be disk related somehow:
e.g.
```
# create 2 GB tmpfs volume at /tmp/ramdisk, mount may need sudo
mkdir -p /tmp/ramdisk
mount -t tmpfs -o size=2048mb myramdisk /tmp/ramdisk
# download the file e.g., perhaps a few times to verify/reproduce
synapse --debug get syn21560469 --downloadLocation /tmp/ramdisk
# when done can remove the tmpfs partition
umount /tmp/ramdisk
```
So... update... some single file download worked... It's not just recursive that failed... but some single downloads are failing too:
`synapse --debug get syn21560469 --downloadLocation .`
```
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /project/urology/Strand_lab/ghenry/ref/droplet_normal_lung_blood_seurat_ntiss10x.P1.anno.20191002.RC4.Robj.synapse_download_49506559's md5 496111935d301b3d4a35
```
ps... this is to a stable filesystem @ghhenry So I am clear the single file download worked with and without the downloadLocation option, but the recursive option fails using the option (to a directory mounted to a stable filesystem)? The downloadLocation option should in effect be the same as invoking **synapse get** from the directory specified (i.e. it allows you to specify a location other than the current working directory) but I wasn't sure if you were invoking it from a directory mounted to the filesystem you mentioned previously. So, just `get`ting the single object (and also using `--downloadLocation .` which I didn't use before), it worked... @ghhenry
Hmm, that's odd the stack suggests that an md5 checksum of some of these files after downloading didn't match the expected checksum from the Synapse meta data. The expected MD5s shown in the stack traces match the md5s of the files themselves in Synapse object storage, and I'm not able to reproduce this myself after using the same command to download the full 21gb syn21560412 folder (the files download intact and match the expected md5). You mentioned an unstable file system which could certainly cause an MD5 mismatch in the resultant download but it sounds like you are now also reproducing this with a different file system as well.
Can you try downloading one of these files by itself and seeing if it successfully downloads?
```
synapse --debug get syn22168625
```
This will download _facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Rob_ by itself as a single download in isolation of the rest of the folder.
You can use the **--downloadLocation** parameter in either a recursive folder or single file download to specify a different download path to the more stable file system (perhaps you are already doing that). same error with debug on the more stable filesystem output with debug flag
```
(synapse) [ghenry@Nucleus009 Travaglini_Nature2020]$ synapse --debug get -r syn21560412
Synapse username: ghhenry
b'Password or api key for ghhenry: '
Welcome, ghhenry!
2020-11-25 11:57:45,896 [client:409 - INFO]: Welcome, ghhenry!
Downloading [####################]100.00% 634.6MB/634.6MB (2.9MB/s) facs_normal_lung_seurat_ntiss.P3.anno.gencode.20191002.RC4.Robj.synapse_download_49506012 Done...
\ Downloaded 8.3GB (26.9MB/s)2020-11-25 12:03:01,007 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937's md5 3705adcc4315c45fae2580c627bf91c8 does not match expected MD5 of 1e3dbcb91f8fefbf83119a37ba222df4
| Downloaded 10.1GB (22.1MB/s)2020-11-25 12:05:36,495 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20200616.RC4.Robj.synapse_download_54557964's md5 1d8c032160d5f61025afe85e694a6f95 does not match expected MD5 of fc03dcf472cd07db61ec672f3f1cbe81
- Downloaded 10.2GB (22.1MB/s)2020-11-25 12:05:38,684 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/previous objects/facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20191002.RC4.Robj.synapse_download_48601559's md5 b19405784ca2f81728f451b3e42583ce does not match expected MD5 of 3d102ee730dacbc66775f525009ec76a
\ Downloaded 9.1GB (16.5MB/s)2020-11-25 12:07:13,586 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/droplet_normal_lung_blood_seurat_ntiss10x.P1.anno.20191002.RC4.Robj.synapse_download_49506559's md5 b54b4fe723ae521c8e36f28012ad6293 does not match expected MD5 of 5d4b0b24732004c626fff356b68559da
\ Downloaded 8.4GB (13.6MB/s)2020-11-25 12:08:14,693 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937's md5 e7252f227253339573086c4ab853bc55 does not match expected MD5 of 1e3dbcb91f8fefbf83119a37ba222df4
\ Downloaded 12.3GB (13.8MB/s)2020-11-25 12:13:04,604 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20200616.RC4.Robj.synapse_download_54557964's md5 b6d89692b5c9ed097a4a2855038234d8 does not match expected MD5 of fc03dcf472cd07db61ec672f3f1cbe81
| Downloaded 11.2GB (12.4MB/s)2020-11-25 12:13:12,926 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/previous objects/facs_normal_lung_blood_seurat_ntiss.P1.anno.gencode.20191002.RC4.Robj.synapse_download_48601559's md5 86bb883b01a636b62b766807021df102 does not match expected MD5 of 3d102ee730dacbc66775f525009ec76a
/ Downloaded 10.0GB (11.0MB/s)2020-11-25 12:13:18,699 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937's md5 30c88b9a7e91d94ae9de6bda50a164a9 does not match expected MD5 of 1e3dbcb91f8fefbf83119a37ba222df4
- Downloaded 11.2GB (10.3MB/s)2020-11-25 12:16:18,184 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/mouse_droplet_TMS_tiss10x.mouse.anno.ncbi.20200222.RC4.Robj.synapse_download_49506717's md5 3fad8b1ef2cfcd6e688b1bd8d83b3db5 does not match expected MD5 of 953fb1bfcb0c8344104ca7bd8db8b9d3
\ Downloaded 9.3GB (7.6MB/s)2020-11-25 12:18:31,959 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937's md5 e2e5a8dfcf0b8f3ff7fb35db8626baea does not match expected MD5 of 1e3dbcb91f8fefbf83119a37ba222df4
/ Downloaded 10.3GB (7.2MB/s)2020-11-25 12:22:13,908 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/droplet_normal_lung_blood_seurat_ntiss10x.P3.anno.20191002.RC4.Robj.synapse_download_49506244's md5 1a79d9da2d32432c7bac94e5f841a941 does not match expected MD5 of c4c68a2d0b3b1c188fa511c4899bb826
| Downloaded 6.9GB (4.5MB/s)2020-11-25 12:23:38,828 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1833, in _download_from_url_multi_threaded
filename=temp_destination, md5=actual_md5, expected_md5=expected_md5
synapseclient.core.exceptions.SynapseMd5MismatchError: Downloaded file /endosome/work/urology/ghenry/RNA-Seq/SingleCell/ANALYSIS/REF/Travaglini_Nature2020/facs_normal_lung_blood_seurat_ntiss.P2.anno.gencode.20200616.RC4.Robj.synapse_download_54557937's md5 e353c8290da689cd92699567d5b7fede does not match expected MD5 of 1e3dbcb91f8fefbf83119a37ba222df4
| Downloaded 7.0GB (4.5MB/s)2020-11-25 12:24:07,494 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-11-25 12:24:08,240 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-11-25 12:24:08,662 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-11-25 12:24:09,080 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
/ Downloaded 7.2GB (4.2MB/s)2020-11-25 12:26:35,566 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-11-25 12:26:36,401 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-11-25 12:26:36,845 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2020-11-25 12:26:37,329 [client:1795 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1782, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/client.py", line 1819, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 294, in download_file
pending_futures,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/site-packages/synapseclient/core/multithread_download/download_threads.py", line 357, in _submit_chunks
end,
File "/home2/ghenry/.conda/envs/synapse/lib/python3.6/concurrent/futures/thread.py", line 117, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
``` FYI, 2.2.0 also stalled.
when it stalls, it show: ` \ Downloaded 6.9GB (4.5MB/s)` with the spinning indicator... though the it isn't spinning anymore... even left it overnight just to give it more time, and the next morning was still in the same spot.
I am running it now with the debug flag just to see.
I am also trying something else... the filesystem that I am trying to download it to is fairly new and has had some issues (that I don't completely understand). I don't think that is the problem, but I am going to try and download it to a more stable filesystem also just to be sure. There is not much a change in how downloads are treated between 2.2.0 and 2.2.2 other than some exception handing changes that I think would be unlikely to cause any stalling.
If any errors are encountered (network connectivity etc) during the download the Synapse client will enter into a retry backoff during which it can appear to be making no additional progress, that is one possibility.
@ghhenry when the download stalls can you describe what the CLI is displaying? Is the progress indicator spinning?
If you continue to encounter this can you add the debug option to the download command? This will cause any exceptions encountered during a download that result in a retry to be printed to the console.
e.g.
```
synapse --debug get -r syn21560412
```
@jordank Could you please review this thread and see if you have any insights to share? Not done yet, but it seems to be working...
the latest version in pypi is 2.2.2 and it stalls... but downgraded to 2.2.0 seems to work... I downgraded to client 2.2.0 and it has gotten further than it has before... stay tuned... Synapse Client version 2.2.2
It is being downloaded on a petabyte HPC, the file system that I am downloading it to , I have at least 5Tb's of quota. For the cache, my home has 25Gb left before reaching it's soft quota and the hard quota being 50Gb's away. Follow up: I was able to run the same command without any problem. Please make sure you have the latest version of the Synapse CLI, version 2.2.0:
```
synapse --version
Synapse Client 2.2.0
```
It looks like the overall content size is 21 GB. Have you checked whether your disk is full? What version of the CLI are you using?
Drop files to upload
Stall on recursive download with CLI page is loading…