Hi, when I download RNA-seq fastq files via command using synapse, some files fail to download with errors. And along with that, I can't even get the metadata.tsv.
Created by Misato Kaishima K-Misato Hello @K-Misato,
Apologies for the delay - I have been trying to reproduce this error, but I haven't been able to. I've been able to create AWS ecs instances and download the data locally from my computer:
```
(synapse) ? test synapse --debug get syn33073353
Downloading [####################]100.00% 29.8GB/29.8GB (38.3MB/s) MEM00018_R1.fastq.synapse_download_99084107 Done...
Downloaded file: MEM00018_R1.fastq
2023-05-04 09:11:31,995 [__main__:119 - INFO]: Downloaded file: MEM00018_R1.fastq
Creating /.../MEM00018_R1.fastq
2023-05-04 09:11:31,998 [__main__:124 - INFO]: Creating /.../MEM00018_R1.fastq
````
The only thing I can think of now is this: https://help.synapse.org/docs/Synapse-Infrastructure.2835382273.html
If you are on an institution's server, can you double check that all of those IP addresses are whitelisted by your IT team? One way to check is to try to run synapse command on your home network.
Hello Will,
My disk space is 50T 4.2T 46T 9% /mnt/PRAKP.
I ran the same command on someone else's PC and server and got the same error.
Can you please confirm the error once you also type this command?
synapse --debug get syn33073353
I would like to share my server information.
CPU: Xeon Gold 6142 (16core, 2.60GHz) x 2
Memory: 384GB (32GB x 12)
Best,
Misato Hi @K-Misato,
Thanks for sending that information, but unfortunately nothing obvious is coming out of that error either. Can you share the details of the server/computer/instance that you are using? (RAM/memory and CPUs). In addition, can you confirm that you have enough disk space available to download these files? You can run the 'df -h' command to view disk space.
Best,
Will Hi Will,
Thank you so much for your helping.
I tried this command
synapse --debug get syn33073353
and results is that
Downloading [#-------------------]2.96% 904.0MB/29.8GB (26.2MB/s) MEM00018_R1.fastq.synapse_download_99084107 2023-04-28 19:29:42,907 [client:2018 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 351, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: MEM00018_R1.fastq, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 1998, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(fileHandleId,
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 2043, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 235, in download_file
downloader.download_file(download_request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 312, in download_file
self._check_for_errors(request, completed_futures)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 408, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn33073353 to /mnt/PRAKP/data/MEM00018_R1.fastq.synapse_download_99084107
Downloading [#-------------------]3.12% 952.0MB/29.8GB (26.6MB/s) MEM00018_R1.fastq.synapse_download_99084107 2023-04-28 19:30:29,955 [client:2018 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 351, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: MEM00018_R1.fastq, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 1998, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(fileHandleId,
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 2043, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 235, in download_file
downloader.download_file(download_request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 312, in download_file
self._check_for_errors(request, completed_futures)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 408, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn33073353 to /mnt/PRAKP/data/MEM00018_R1.fastq.synapse_download_99084107
Downloading [#-------------------]3.09% 944.0MB/29.8GB (27.3MB/s) MEM00018_R1.fastq.synapse_download_99084107 2023-04-28 19:31:14,714 [client:2018 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 351, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: MEM00018_R1.fastq, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 1998, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(fileHandleId,
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 2043, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 235, in download_file
downloader.download_file(download_request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 312, in download_file
self._check_for_errors(request, completed_futures)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 408, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn33073353 to /mnt/PRAKP/data/MEM00018_R1.fastq.synapse_download_99084107
Downloading [#-------------------]2.83% 864.0MB/29.8GB (25.5MB/s) MEM00018_R1.fastq.synapse_download_99084107 2023-04-28 19:31:58,990 [client:2018 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 351, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: MEM00018_R1.fastq, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 1998, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(fileHandleId,
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 2043, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 235, in download_file
downloader.download_file(download_request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 312, in download_file
self._check_for_errors(request, completed_futures)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 408, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn33073353 to /mnt/PRAKP/data/MEM00018_R1.fastq.synapse_download_99084107
Downloading [#-------------------]3.23% 984.0MB/29.8GB (27.7MB/s) MEM00018_R1.fastq.synapse_download_99084107 2023-04-28 19:32:45,586 [client:2018 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 351, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: MEM00018_R1.fastq, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 1998, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(fileHandleId,
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 2043, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 235, in download_file
downloader.download_file(download_request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 312, in download_file
self._check_for_errors(request, completed_futures)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 408, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn33073353 to /mnt/PRAKP/data/MEM00018_R1.fastq.synapse_download_99084107
Traceback (most recent call last):
File "/home/kid20959/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 351, in _get_response_with_retry
raise SynapseError(
synapseclient.core.exceptions.SynapseError: Could not download the file: MEM00018_R1.fastq, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/kid20959/anaconda3/bin/synapse", line 8, in
sys.exit(main())
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/__main__.py", line 1212, in main
perform_main(args, syn)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/__main__.py", line 1116, in perform_main
args.func(args, syn)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/__main__.py", line 114, in get
entity = syn.get(args.id, version=args.version, # limitSearch=args.limitSearch,
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 731, in get
return self._getWithEntityBundle(entityBundle=bundle, entity=entity, **kwargs)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 851, in _getWithEntityBundle
self._download_file_entity(downloadLocation, entity, ifcollision, submission)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 913, in _download_file_entity
downloadPath = self._downloadFileHandle(entity.dataFileHandleId, objectId, objectType, downloadPath)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 1998, in _downloadFileHandle
downloaded_path = self._download_from_url_multi_threaded(fileHandleId,
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/client.py", line 2043, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 235, in download_file
downloader.download_file(download_request)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 312, in download_file
self._check_for_errors(request, completed_futures)
File "/home/kid20959/anaconda3/lib/python3.8/site-packages/synapseclient/core/multithread_download/download_threads.py", line 408, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn33073353 to /mnt/PRAKP/data/MEM00018_R1.fastq.synapse_download_99084107
Hi @K-Misato,
Can you try running the following command and share the full output?
```
synapse --debug get syn21597100
```
Thanks,
Will Hello Will,
I am getting the same error as before, it seems to be for files over 1GB.
I have access to files that are control restricted. I have already submitted the documents and the key is green and open.
Only some large size files are unable d unable to download.
Best,
Misato Hi @K-Misato,
Sorry for the late reply and the bad suggestion, I forgot that the Synapse client doesn't offer a verbose option. If you run the same command without the v option:
```
synapse get syn21597100
```
Does this result in a different error, or the same one as before? The Synapse client can be used to download large files so that shouldn't be an issue. My other question would be, since this is a controlled dataset, have you gained access to the files by completing the appropriate Data Use Certificate (DUC)? If so, you should see a green lock icon next to "Access" above the dataset. If you were able to download some, but not all of the files then this is unlikely to be the issue however.
Best,
Will
Hi,
Are there any restrictions, such as files over 5 GB cannot be downloaded from S3 using the API?
Thanks again,
Misato
Hi will,
I could not use -v option. I think it is command for version check.
Best,
Misato Hi @K-Misato,
Thanks for that information. Unfortunately I'm not immediately clear what the cause of that error is. Can you try downloading a single one of those files, with the verbose flag added to produce a more specific error? For example:
```
synapse get syn21597100 -v
```
That should hopefully help us narrow down the cause of the error.
Best,
Will Hi Will,
Thank you for your helping.
When I used this command " synapse get -r syn21589959 ", it was happened.
I could not download about 100 samples, for example,
"RISK_184_rerun_S128_R1_001.fastq.gz"
"RISK_184_rerun_S128_R2_001.fastq.gz"
"RISK_187_rerun_S129_R1_001.fastq.gz"
"RISK_188_rerun_S130_R1_001.fastq.gz"
"RISK_188_rerun_S130_R2_001.fastq.gz"
"RISK_189_rerun_S131_R2_001.fastq.gz"
"RISK_207_redo_S44_R2_001.fastq.gz"
"RISK_227_redo_S56_R2_001.fastq.gz"
"RISK_229_redo_S64_R2_001.fastq.gz"
and, error massage was that
SynapseHTTPError: 400 Client Error:
Batch must contain at least one item
Best,
Misato Hi @K-Misato,
I'm happy to help you troubleshoot this issue. Could you share an example synapseid for a file that is failing, a specific error message that you are getting, and the command that you are trying to run?
Best,
Will
Drop files to upload
Some RNA-seq fastq files fail to download via API. page is loading…