Hi, When I download with the command 'synapse get-download-list' it keep failing after downloaded some GB data. The messages are as follows:
(base) [psychencode@hpc-login01 RNAseq]$ synapse get-download-list
Exception in version check: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /Sage-Bionetworks/synapsePythonClient/master/synapseclient/synapsePythonClient (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))
[####################]100.00% 1/1 Done...
Downloading [####################]100.00% 281.5kB/281.5kB (393.8kB/s) manifest.csv Done...
Downloading [#-------------------]3.29% 80.0MB/2.4GB (2.2MB/s) 01-052914_AN19442_BA7.bam.synapse_download_19114817 [ERROR] Unable to download file
Downloading [#-------------------]3.04% 64.0MB/2.1GB (1.7MB/s) 01-053014_AN16641_BA17.bam.synapse_download_19114891
How can I figure it out?
I am looking forward for your help
Created by Jinchen Li JCLi Hi @JCLi ,
This is beyond the scope of what I am able to answer. Please review the Wiki of the dataset you are using. If you are unable to find the answer please post a new Thread under the Discussion tab of the dataset:
https://www.synapse.org/#!Synapse:syn4921369/wiki/
https://www.synapse.org/#!Synapse:syn4921369/discussion/default Hi @bfauble ?
I want to ask you that how to know which of the RNAseq data from the UCLA-ASD study are from disease samples and which are from control samples?
I did not find these information in the metadata.
I?m looking for your reply.
Sincerely,
Lee Hi @JCLi Great observations.
When a file is fully downloaded from Synapse a calculation of the MD5 of the file is performed to verify that all chunks of the file were successfully downloaded. That code exists here: https://github.com/Sage-Bionetworks/synapsePythonClient/blob/v4.1.1/synapseclient/client.py#L3098C26-L3098C62 .
Larger GB+ files can take a large amount of time to run this calculation which is why you are seeing it on your low resource ec2 instance. Hi @bfauble ?
$ synapse --debug get syn11286054.2
2024-04-04 01:06:59,317 [client:5827 - DEBUG]: Sending get request to /userProfile
2024-04-04 01:06:59,356 [client:5827 - DEBUG]: Sending post request to /entity/syn11286054/version/2/bundle2
2024-04-04 01:06:59,450 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
2024-04-04 01:06:59,500 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [#-------------------]4.69% 136.0MB/2.8GB (5.9MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:07:24,875 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [##------------------]11.87% 344.0MB/2.8GB (6.9MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:07:50,501 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [####----------------]18.22% 528.0MB/2.8GB (7.2MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:08:15,109 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [#####---------------]24.84% 720.0MB/2.8GB (7.2MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:08:40,249 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [######--------------]31.47% 912.0MB/2.8GB (7.3MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:09:05,992 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [########------------]38.37% 1.1GB/2.8GB (7.4MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:09:32,098 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [#########-----------]45.54% 1.3GB/2.8GB (7.5MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:09:58,703 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [###############-----]77.01% 2.2GB/2.8GB (11.0MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:10:23,054 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [#################---]83.63% 2.4GB/2.8GB (10.7MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:10:49,203 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [##################--]87.77% 2.5GB/2.8GB (10.1MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:11:14,831 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [###################-]94.40% 2.7GB/2.8GB (9.9MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 2024-04-04 01:11:39,784 [client:5827 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [####################]100.00% 2.8GB/2.8GB (9.7MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 Done...
Downloaded file: PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz
2024-04-04 01:15:08,622 [__main__:164 - INFO]: Downloaded file: PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz
Creating /home/ubuntu/PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz
2024-04-04 01:15:08,622 [__main__:171 - INFO]: Creating /home/ubuntu/PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz
you see, after this line ?Downloading [####################]100.00% 2.8GB/2.8GB (9.7MB/s) PEC_HSB_Yale-UCSF_mRNA_RiboZero_HSB500-R2.fastq.gz.synapse_download_133381074 Done...?
The debug?s result is here?so is the slow running problem because of the ec2 low configuration?
@JCLi With the later version of the Synapse Python Client you are able to use the `--debug` flag to see exactly what calls the client is doing: https://python-docs.synapse.org/tutorials/command_line_client/?h=debug#options
`synapse --debug get syn7113690`
Hi @bfauble ?
After I run the code below each time, the command already prompts for the download to complete, but each time it pauses for a long time before it prompts for the ability to continue entering line commands, what's going on?
How can I figure it out?
$ synapse get syn7113690
Downloading [####################]100.00% 6.9GB/6.9GB (10.9MB/s) PEC_HSB_Yale-UCSF_V1C_mRNA_RiboZero_HiSeq2500_HSB373.bam.synapse_download_133419458 Done...
@JCLi When you use the `synapse get` CLI command you need to use the `--recursive` flag
https://python-docs.synapse.org/tutorials/command_line_client/?h=recursive#get
`synapse get --recursive syn17096299` Hi @thomas.yu @bfauble ,
The new problem comes out:
ubuntu@ip-172-31-90-92:~$ synapse get syn17096299
Synapse username (Optional):
Auth token:
Welcome, Jinchen Li!
WARNING: No files associated with entity syn17096299
Folder: RNAseq (syn17096299)
properties:
concreteType=org.sagebionetworks.repo.model.Folder
createdBy=273995
createdOn=2018-12-13T11:05:48.141Z
etag=e2a7b402-7ab2-11e9-98fa-026b0a0ad230
id=syn17096299
modifiedBy=273995
modifiedOn=2018-12-13T11:05:56.075Z
name=RNAseq
parentId=syn5613805
annotations:
what are these code mean?
Thank you very much! Hi @bfauble ?
Why I come up to this problem?how can I figure it out?
:~$ synapse get syn4566330
[INFO 04:45:54.875515] [synapse-main:267] Starting up...
(synapse:3078): Gtk-WARNING **: 04:45:54.876: cannot open display:
@JCLi As Anthony shared - See the access requirements at: https://www.synapse.org/#!AccessRequirement:AR_ID=9606304 - After accepting the terms of use see the WIKI page for information on how to download the data: https://www.synapse.org/#!Synapse:syn4921369/wiki/626346
There were additional restrictions put in place that you'll need to complete. Hi @thomas.yu ,
I noticed that the psychENCODE data were moved to NDA, could you please tell me where I can find the PsychENCODE data exactly?
Thank you very much for your help. Hello,
Please note that just a couple of hours ago, in consultation with the NIH, we implemented an emergency change to data egress. Please see the access restriction placed on PsychENCODE at the following link introducing this topic. There are additional links provided as resources, including instructions for resuming downloads through the month of March.
https://www.synapse.org/#!AccessRequirement:AR_ID=9606304
Thank you for your understanding and cooperation.
Anthony
on behalf of the Synapse Access & Compliance Team (ACT) I am looping in @anthony.pena for assistance on this issue Hi @thomas.yu ,
my command line is just below:
(base) [psychencode@hpc-login01 RNAseq]s synapse get -r syn4588490Connection refused'))
Exception in version check: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded w
ith url: /Sage-Bionetworks/synapserythonClient/master/synapseclient/synapserythonClient Caused by ewlonnectionri
ori': Failed to establish a new connection: IErrno 111
**Downloaded 2.8GB (1.2MB/s )/s)s)**
The information means these data are being download, but the thing is: the total data size is always around 2.8GB, then it won't increase, why such thing happened?And how can I figure it out?
Thank you very much!! Hi @JCLi ,
Could you try re-running the command? There's been data migrations for this project, so it may be causing some instability in downloads.
Best,
Tom Dear @thomas.yu ,
How can I download the data in my download list?
the debug report is here, can you help me find how can I figure it out?
(base) [psychencode@hpc-login01 RNAseq]$ synapse --debug get-download-list
Exception in version check: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /Sage-Bionetworks/synapsePythonClient/master/synapseclient/synapsePythonClient (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))
2024-02-26 23:38:02,823 [client:5826 - DEBUG]: Sending get request to /userProfile
2024-02-26 23:38:03,133 [client:5826 - DEBUG]: Sending post request to /download/list/manifest/async/start
2024-02-26 23:38:03,462 [client:5826 - DEBUG]: Sending get request to /download/list/manifest/async/get/33435723
2024-02-26 23:38:05,950 [client:5826 - DEBUG]: Sending get request to /download/list/manifest/async/get/33435723
[####################]100.00% 1/1 Done...
2024-02-26 23:38:06,244 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:06,545 [client:3164 - DEBUG]: Downloading from https://data.prod.sagebase.org/3461275/aa5e921e-f1fd-486e-903a-7b5d5a491cda/manifest.csv?response-content-disposition=attachment%3B%20filename%3D%22manifest.csv%22%3B%20filename%2A%3Dutf-8%27%27manifest.csv&response-content-type=text%2Fcsv&X-Amz-Date=20240226T154007Z&X-Amz-Expires=30&Expires=1708962037&Signature=kO5JwEYbcX5IJPQF9ES~fOVUo-lTMAQh6dEyaZTbmN9-Bc3MVzK-zPE6K1NuJGdzpo7a5CQcUQkStzkIZsUsBX25qlNctxha184K1S2JjMgMuBiY79FPKF0q3mA4F53~yt7iqVgEMPh-TV1DzG7dC2A82ydm0vkpRp5oyJwetPKrduQzekqagA3ZL~fZRiu1~k6mvjYBJ00c8EfGUsiR6FdpJdrTbEIXyPg1y1cUXg8epu3V5K-z69CcCUXn7TGQSY6qHinkocya9AaM1rNo1bTMxDJJl-v7YRIAhCIG-~72ITFXvePUUnSlSjqms866OqcwjgPt2-BEHapNEKwexA__&Key-Pair-Id=K1ODM3BLJ5L7YV to /gpfs/hpc/home/public/psychencode/ASD/UCLA_ASD/RNAseq
Downloading [####################]100.00% 229.0kB/229.0kB (295.4kB/s) manifest.csv Done...
2024-02-26 23:38:08,645 [client:5826 - DEBUG]: Sending post request to /entity/syn11255735/bundle2
2024-02-26 23:38:08,985 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:09,291 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:10,839 [client:3044 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 304, in download_file
file_size = _get_file_size(url_info.url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 222, in _get_file_size
return int(res_get.headers["Content-Length"])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-length'
2024-02-26 23:38:10,841 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:11,152 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:12,678 [client:3044 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 304, in download_file
file_size = _get_file_size(url_info.url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 222, in _get_file_size
return int(res_get.headers["Content-Length"])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-length'
2024-02-26 23:38:12,679 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:12,978 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:14,450 [client:3044 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 304, in download_file
file_size = _get_file_size(url_info.url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 222, in _get_file_size
return int(res_get.headers["Content-Length"])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-length'
2024-02-26 23:38:14,451 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:14,769 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:16,325 [client:3044 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 304, in download_file
file_size = _get_file_size(url_info.url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 222, in _get_file_size
return int(res_get.headers["Content-Length"])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-length'
2024-02-26 23:38:16,326 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:16,647 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:18,186 [client:3044 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 304, in download_file
file_size = _get_file_size(url_info.url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 222, in _get_file_size
return int(res_get.headers["Content-Length"])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-length'
2024-02-26 23:38:18,187 [client:2207 - ERROR]: Unable to download file
2024-02-26 23:38:18,187 [client:5826 - DEBUG]: Sending post request to /entity/syn11255938/bundle2
2024-02-26 23:38:18,555 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:18,857 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:20,351 [client:3044 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 304, in download_file
file_size = _get_file_size(url_info.url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 222, in _get_file_size
return int(res_get.headers["Content-Length"])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-length'
2024-02-26 23:38:20,352 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-26 23:38:20,673 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
Hi @JCLi
The synapse client will cache all previous downloads so if you just run the command again, it will only download the files that haven't been downloaded.
We will look into the issue on Monday when we are back during our work hours - it would be very helpful to add the --debug parameter so that we can help with errors like this. Hi @thomas.yu ,
Downloading the data failed again.
The message is?
ValueError: File download failed during sync
caused by ValueError: Failed downloading syn52059507 to /gpfs/hpc/home/public/psychencode/ASD/UCLA_ASD/RNAseq/Manuscript-Related Analysisles/Loupe,J., et al. Public Share/Loupe,J., et al. Public Share - Data/ZNF384_FP_Output_30M.GSLv5-8_i7_63.merge.nodup_30M.GSLv5-8_i7_67.mernodup.tagAlign_x_30M_dedup_trim_FP_1224_control.nodup_30M_dedup_trim_FP_1230_control.nodup.tagAlign.pval.signal.bw.synapse_download_1269792
caused by SynapseError: Could not download the file: ZNF384_FP_Output_30M.GSLv5-8_i7_63.merge.nodup_30M.GSLv5-8_i7_67.merge.nodup.tagAl_x_30M_dedup_trim_FP_1224_control.nodup_30M_dedup_trim_FP_1230_control.nodup.tagAlign.pval.signal.bw, please try again.
How can I figure it out?
Thank you!
Hi @thomas.yu ,
I'm sorry I just fogot to enter the letter "get",
It just start to download the data, but it hasn't been downloading those GBs of data yet, it's been downloading small data, I'll watch it for a while longer
Thank you very much!
Hi @thomas.yu ,
I have enter the Project SynID:syn4921369, and apply the command line, but it also cannot success:
(base) [psychencode@hpc-login01 RNAseq]$ synapse -r syn4921369 usage: synapse [-h] [--version] [-u SYNAPSEUSER] [-p SYNAPSE_AUTH_TOKEN] [-c CONFIGPATH] [--debug] [--silent] [-s]
[--otel {console,otlp}]
{get,manifest,sync,store,add,mv,cp,get-download-list,associate,delete,query,submit,show,cat,list,config,set-provenance,get-provenance,set-annotations,get-annotations,create,store-table,onweb,login,test-encoding,get-sts-token,migrate}
...
synapse: error: argument subparser: invalid choice: 'syn4921369' (choose from 'get', 'manifest', 'sync', 'store', 'add', 'mv', 'cp', 'get-download-list', 'associate', 'delete', 'query', 'submit', 'show', 'cat', 'list', 'config', 'set-provenance', 'get-provenance', 'set-annotations', 'get-annotations', 'create', 'store-table', 'onweb', 'login', 'test-encoding', 'get-sts-token', 'migrate')
hi @bfauble ,
I have run synapse --debug get-download-list and the debug messages here:
(base) [psychencode@hpc-login01 RNAseq]$** synapse --debug get-download-list**
Exception in version check: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /Sage-Bionetworks/synapsePythonClient/master/synapseclient/synapsePythonClient (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))
2024-02-24 09:30:01,478 [client:5826 - DEBUG]: Sending get request to /userProfile
2024-02-24 09:30:01,788 [client:5826 - DEBUG]: Sending post request to /download/list/manifest/async/start
2024-02-24 09:30:02,108 [client:5826 - DEBUG]: Sending get request to /download/list/manifest/async/get/33401076
2024-02-24 09:30:04,588 [client:5826 - DEBUG]: Sending get request to /download/list/manifest/async/get/33401076
[####################]100.00% 1/1 Done...
2024-02-24 09:30:04,881 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-24 09:30:05,189 [client:3164 - DEBUG]: Downloading from https://data.prod.sagebase.org/3461275/585b12ae-620e-49be-9006-68a7154c6153/manifest.csv?response-content-disposition=attachment%3B%20filename%3D%22manifest.csv%22%3B%20filename%2A%3Dutf-8%27%27manifest.csv&response-content-type=text%2Fcsv&X-Amz-Date=20240224T013204Z&X-Amz-Expires=30&Expires=1708738354&Signature=DZlmye1wCFTdcdXyspR6eca7xWoH91i0XXTBr5b7bGdn2qbK0cI3CXYlu3Cp0X6~VU8WRoEeJjJ2KmLVkeAr~iTpM8z~QkCB8M5l~TBGhbwu4kmmQDC39zmfId4Q5HMynrLmg02ZdbvolEAvbnTpEq0y6PY2XOoiTpB26CWXrWfwN0XVgFzeMfrYaJbxBGLwO0q-oUPJ0MUAFxx56x5x7lOs5wr~s3djSRZW76l3VP9xganLm~H46uqhW~RXzxlrtV6wJiz~rwRaXgz1ZY6DCJSwGQI2IJJ6OwesOQ46xrdfMVAP2INwMPKbstlYlrmWTU9GVuqzFBfUGHKIE-F3~w__&Key-Pair-Id=K1ODM3BLJ5L7YV to /gpfs/hpc/home/public/psychencode/ASD/UCLA_ASD/RNAseq
Downloading [####################]100.00% 231.1kB/231.1kB (340.1kB/s) manifest.csv Done...
2024-02-24 09:30:07,135 [client:5826 - DEBUG]: Sending post request to /entity/syn11255735/bundle2
2024-02-24 09:30:07,485 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-24 09:30:07,801 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [--------------------]1.97% 48.0MB/2.4GB (1.3MB/s) 01-052914_AN19442_BA7.bam.synapse_download_19114817 2024-02-24 09:32:22,938 [client:3044 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 379, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: 01-052914_AN19442_BA7.bam, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3020, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/client.py", line 3092, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 336, in download_file
self._check_for_errors(request, completed_futures)
File "/gpfs/hpc/home/public/psychencode/miniconda3/lib/python3.11/site-packages/synapseclient/core/multithread_download/download_threads.py", line 439, in _check_for_errors
raise ValueError(
ValueError: Failed downloading syn11255735 to /gpfs/hpc/home/public/psychencode/ASD/UCLA_ASD/RNAseq/01-052914_AN19442_BA7.bam.synapse_download_19114817
2024-02-24 09:32:22,950 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
2024-02-24 09:32:24,090 [client:5826 - DEBUG]: Sending post request to /fileHandle/batch
Downloading [#-------------------]3.62% 88.0MB/2.4GB (2.2MB/s) 01-052914_AN19442_BA7.bam.synapse_download_19114817
(base) [psychencode@hpc-login01 RNAseq]$ **synapse --version**
**Synapse Client 4.1.0**
and my python's version is 3.11.5 Hi @JCLi ,
Something that will give me some more troubleshooting information: If you run `synapse --debug get-download-list` can you post the logs from the debug messages that show up please?
Could you also give me the results of:
`synapse --version`
and
`python --version` or `python3 --version` Hi @JCLi ,
I understand the urgency. Everything file, folder, project in Synapse has a unique identifier. Lets take this: https://www.synapse.org/#!Synapse:syn7222066 as an example.
1. Top left corner with "Project SynID: syn7222066"
2. Click on a folder: https://www.synapse.org/#!Synapse:syn7844529 -> middle left : "SynID syn7844529"
If you search for those strings, you should find it. alternatively, you will see that the synapse ids are in the URLS.
Separately the `-r` command recursively downloads an entire project or folder. So you aren't looping through the files yourself. I'm sorry @thomas.yu , but it's really a urgent problem for our team, after all, it's too much trouble to download them one by one Hi, @thomas.yu
These data are from UCLA-ASD, but other data I download also failed.
?So how can I download the files greater than 100MB ?
?How to know what is the project or folder id?
Thank you very much! Hi @JCLi,
Could you let us know the synapse Id of 1 01-053014_AN16641_BA17.bam or the project you are trying to download from? --Files that are greater than 100MB cannot be downloaded via that download cart function.--
If you are downloading files from a specific project, I would leverage the `synapse get -r ` command.
Drop files to upload
Synapse client keep failing download page is loading…