I am a beginner in programming. I have selected several bulk RNA-seq data and added them to my Download Cart list. Now I want to download all the data in the list at once. Following the instructions for Programmatic Download, I used the command line "synapse get-download-list",but it returned an error. How can I resolve this issue?
My Synapse Client is 4.0.0,
Created by Jiaqi Liang liangjiaqi Hi @sjobe ,
It does not work after I upgrading the synapseclient. When I changed the network, it can download the whole folder files even though the download speed is very slow. So I think the problem is from my network.
Best,
Liang Hi @liangjiaqi,
I am curious to hear if upgrading the synapseclient resolved your error?
Thanks,
Sophia Hi @liangjiaqi,
I tried to reproduce the error using Python 3.12.2 and synapseclient 4.0.0 but it downloaded the file successfully on my end. Based on the debug logs, it seems the actual error is not printing out. This issue can probably be resolved by upgrading the synapseclient to version 4.1.1. Could you try to upgrade synapseclient and re-do the download? Here is the command to upgrade the synapseclient.
```
pip3 install --upgrade synapseclient
```
Thanks,
Dan Hi @bfauble
My synapse -- version is Synapse Client 4.0.0
My python --version is Python 3.12.2
When I run the command line: synapse --debug get syn4897168, it shows the long debug logs as following:
Downloading [--------------------]2.01% 160.0MB/7.8GB (1.2MB/s) 11263_TCX.snap.bam.synapse_download_75495528 2024-02-28 09:49:13,064 [client:2955 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 379, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: 11263_TCX.snap.bam, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 2931, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 3003, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 336, in download_file
self._check_for_errors(request, completed_futures)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 439, in _check_for_errors
raise ValueError(
ValueError: Failed downloading syn4897168 to /cpfs01/zhuhai/labdata/Mayo_Clinic_RNAseq_AD/control/11263_TCX.snap.bam.synapse_download_75495528
Downloading [--------------------]1.50% 120.0MB/7.8GB (1.5MB/s) 11263_TCX.snap.bam.synapse_download_75495528 2024-02-28 09:53:35,721 [client:2955 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 379, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: 11263_TCX.snap.bam, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 2931, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 3003, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 336, in download_file
self._check_for_errors(request, completed_futures)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 439, in _check_for_errors
raise ValueError(
ValueError: Failed downloading syn4897168 to /cpfs01/zhuhai/labdata/Mayo_Clinic_RNAseq_AD/control/11263_TCX.snap.bam.synapse_download_75495528
Downloading [--------------------]0.10% 8.0MB/7.8GB (266.4kB/s) 11263_TCX.snap.bam.synapse_download_75495528 2024-02-28 09:55:05,638 [client:2955 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 379, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: 11263_TCX.snap.bam, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 2931, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 3003, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 336, in download_file
self._check_for_errors(request, completed_futures)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 439, in _check_for_errors
raise ValueError(
ValueError: Failed downloading syn4897168 to /cpfs01/zhuhai/labdata/Mayo_Clinic_RNAseq_AD/control/11263_TCX.snap.bam.synapse_download_75495528
Downloading [--------------------]0.80% 64.0MB/7.8GB (767.5kB/s) 11263_TCX.snap.bam.synapse_download_75495528 2024-02-28 09:57:41,686 [client:2955 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 379, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: 11263_TCX.snap.bam, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 2931, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 3003, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 336, in download_file
self._check_for_errors(request, completed_futures)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 439, in _check_for_errors
raise ValueError(
ValueError: Failed downloading syn4897168 to /cpfs01/zhuhai/labdata/Mayo_Clinic_RNAseq_AD/control/11263_TCX.snap.bam.synapse_download_75495528
Downloading [#-------------------]5.72% 456.0MB/7.8GB (1.7MB/s) 11263_TCX.snap.bam.synapse_download_75495528 2024-02-28 10:03:11,336 [client:2955 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 379, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: 11263_TCX.snap.bam, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 2931, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 3003, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 336, in download_file
self._check_for_errors(request, completed_futures)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 439, in _check_for_errors
raise ValueError(
ValueError: Failed downloading syn4897168 to /cpfs01/zhuhai/labdata/Mayo_Clinic_RNAseq_AD/control/11263_TCX.snap.bam.synapse_download_75495528
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 379, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: 11263_TCX.snap.bam, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zhanglab01/.conda/envs/synapse4/bin/synapse", line 10, in
sys.exit(main())
^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/__main__.py", line 1897, in main
perform_main(args, syn)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/__main__.py", line 1780, in perform_main
args.func(args, syn)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/__main__.py", line 152, in get
entity = syn.get(
^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 1052, in get
return_data = self._getWithEntityBundle(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 1240, in _getWithEntityBundle
self._download_file_entity(
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 1363, in _download_file_entity
downloadPath = self._downloadFileHandle(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 2931, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/client.py", line 3003, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 255, in download_file
downloader.download_file(download_request)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 336, in download_file
self._check_for_errors(request, completed_futures)
File "/home/zhanglab01/.conda/envs/synapse4/lib/python3.12/site-packages/synapseclient/core/multithread_download/download_threads.py", line 439, in _check_for_errors
raise ValueError(
ValueError: Failed downloading syn4897168 to /cpfs01/zhuhai/labdata/Mayo_Clinic_RNAseq_AD/control/11263_TCX.snap.bam.synapse_download_75495528 Hi @liangjiaqi ,
For one of these downloads that failed - For example `1045_TCX.snap.bam.synapse_download_75497388`, could you please run:
`synapse --debug get syn123`
replace syn123 with the ID of the file in question.
When you run the command you'll get back a large number of debug logs that will tell me what went wrong - If you could post those logs to this thread it'll help me diagnose the problem and suggest a solution.
Could you also give me the results of:
`synapse --version`
and
`python --version or python3 --version` Hi @thomas.yu
There are more than 200 bam files in the folder of syn4894912, but I want to download about 100 bam files (about 1Tb size) from this folder. Currently, I add the 100 bam files into my download cart. How could I use the command to download the download cart files efficiently?
Best regards! Hi @thomas.yu ,
The ID of 1045_TCX.snap.bam is syn4898149, which is in the folder of syn4894912. I have selected some interesting bam. files in my download list from the folder of syn4894912. In this case, how should I use the command line synapse get-download-list to download my insterested bam files? Thanks a lot ! Hi @liangjiaqi,
Could you let us know the synapse Id of `1045_TCX.snap.bam` or the project you are trying to download from?
If you are downloading files from a specific project, I would leverage the `synapse get -r ` command. Hi @bfauble ,
I have created a new token with selecting view, download and modify permission. When I use the command line synapse get-download-list, it shows the error following:
[####################]100.00% 1/1 Done...
Downloading [####################]100.00% 30.2kB/30.2kB (599.5kB/s) manifest.csv Done...
[ERROR] Unable to download file
Downloading [--------------------]1.73% 256.0MB/14.5GB (2.2MB/s) 1045_TCX.snap.bam.synapse_download_75497388 [ERROR] Unable to download file
How could I solve this error? thanks a lot! Get it, thank you for your kind reminder. Hi @liangjiaqi ,
As Bruce mentions please follow his steps to invalidate your token and create a new one.
When you create your new token please select `view`, `download`, and `modify` permissions for the scope of the token. All are required to use the command line `synapse get-download-list`. You must never share access tokens, passwords, or other credentials, especially in a public forum. Please **invalidate** the token (click the red trash icon next to the token on this page, https://www.synapse.org/#!PersonalAccessTokens:), create a new one, and keep it secure.
Drop files to upload
SynapseHTTPError: 403 Client Error: insufficient_scope. Request lacks scope(s) required by this service: modify page is loading…