Hello, I am trying to download some fastq files with synapse get command ``` synapse get syn18838198 ``` however during download it return randomly the following error ``` Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 ValueError: Failed downloading syn18838198 to /path/to/dir/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 caused by SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. ``` it happens also with other files ( I tried with 30 different samples) Am I doing something wrong or is there a workaround for this ? Thanks

Created by gianluca mattei ginlucks
@ginlucks I do not think it is anything specific to the file being downloaded, I do not encounter any issues downloading it, e.g. ``` ~$ synapse --debug get syn18838198 Downloading [####################]100.00% 9.2GB/9.2GB (151.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 Done... Downloaded file: swn_patient_41_blood_1.fastq.gz Creating /home/ubuntu/swn_patient_41_blood_1.fastq.gz ``` Are you sure the stack from the command run from the test version installed was the same? It may have looked similar but it should have had some added debugging information that may shed some light on the underlying cause. If it is different at all could you post it here? Could you also post the output of the following commands? They may help me reproduce the environment you are running on. ``` python3 --version synapse --version # if you are running in a conda environment conda env export pip list openssl version # from the directory you would download into/run the synapse get command df -k . cat /proc/cpuinfo | grep processor | wc -l ```
nothing...I get almost the same error. "Almost" because it stucks at ~ 25%. I also tried to download the file through Pyhton but it always stacks at ~25%. It is strange since I can download the syn3158111 (the tutorial file)
@ginlucks Unfortunately the stack trace doesn't show as much information as I was hoping for. If possible, could you try installing an alternate test version of the Synapse python client? This version has some enhanced retry logic and some additional logging that is being evaluated for inclusion in a future version. This test version can be installed from test.pypi.org as follows: ``` pip3 install --upgrade --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple synapseclient==2.3.1.326 ``` Once installed if you could reattempt the download using the debug option and if any further stack traces occur please provide them here. ``` synapse --debug get syn18838198 ``` Thank you!
@jordank this is the error I get ...can't understand it very well.... maybe something related to multithread ? ``` synapse --debug get syn18838198 Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:25:45,244 [client:1857 - DEBUG]: Retrying download on error: [] after progressing 0 bytes Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry f'Could not download the file: {presigned_url_provider.get_info().file_name},' synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle expected_md5=fileHandle.get('contentMd5')) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded multithread_download.download_file(self, request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file downloader.download_file(download_request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file self._check_for_errors(request, completed_futures) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 ``` EDIT: It returned the same error several times for just one command (synapse --debug get syn18838198), thus I reported it just once Here the full error message:
Full error message ``` (python35) gianlucam@neo:/data2-new/ShwannomatosisPapi/wes/fastq> synapse --debug get syn18838198 Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:25:45,244 [client:1857 - DEBUG]: Retrying download on error: [] after progressing 0 bytes Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry f'Could not download the file: {presigned_url_provider.get_info().file_name},' synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle expected_md5=fileHandle.get('contentMd5')) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded multithread_download.download_file(self, request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file downloader.download_file(download_request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file self._check_for_errors(request, completed_futures) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:28:50,424 [client:1857 - DEBUG]: Retrying download on error: [] after progressing 0 bytes Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry f'Could not download the file: {presigned_url_provider.get_info().file_name},' synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle expected_md5=fileHandle.get('contentMd5')) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded multithread_download.download_file(self, request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file downloader.download_file(download_request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file self._check_for_errors(request, completed_futures) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 Downloading [##------------------]8.95% 840.0MB/9.2GB (4.3MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:32:09,151 [client:1857 - DEBUG]: Retrying download on error: [] after progressing 0 bytes Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry f'Could not download the file: {presigned_url_provider.get_info().file_name},' synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle expected_md5=fileHandle.get('contentMd5')) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded multithread_download.download_file(self, request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file downloader.download_file(download_request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file self._check_for_errors(request, completed_futures) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 Downloading [##------------------]8.95% 840.0MB/9.2GB (4.2MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:35:28,942 [client:1857 - DEBUG]: Retrying download on error: [] after progressing 0 bytes Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry f'Could not download the file: {presigned_url_provider.get_info().file_name},' synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle expected_md5=fileHandle.get('contentMd5')) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded multithread_download.download_file(self, request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file downloader.download_file(download_request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file self._check_for_errors(request, completed_futures) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 Downloading [##------------------]8.95% 840.0MB/9.2GB (4.5MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:38:38,774 [client:1857 - DEBUG]: Retrying download on error: [] after progressing 0 bytes Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry f'Could not download the file: {presigned_url_provider.get_info().file_name},' synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle expected_md5=fileHandle.get('contentMd5')) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded multithread_download.download_file(self, request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file downloader.download_file(download_request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file self._check_for_errors(request, completed_futures) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry f'Could not download the file: {presigned_url_provider.get_info().file_name},' synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/tools/anaconda2/envs/python35/bin/synapse", line 8, in sys.exit(main()) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/__main__.py", line 1059, in main perform_main(args, syn) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/__main__.py", line 972, in perform_main args.func(args, syn) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/__main__.py", line 112, in get downloadLocation=args.downloadLocation) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 713, in get return self._getWithEntityBundle(entityBundle=bundle, entity=entity, **kwargs) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 829, in _getWithEntityBundle self._download_file_entity(downloadLocation, entity, ifcollision, submission) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 891, in _download_file_entity downloadPath = self._downloadFileHandle(entity.dataFileHandleId, objectId, objectType, downloadPath) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle expected_md5=fileHandle.get('contentMd5')) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded multithread_download.download_file(self, request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file downloader.download_file(download_request) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file self._check_for_errors(request, completed_futures) File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 ```

@ginlucks If you are able to reliably reproduce this (sounds like you are) could you try this download from the command line using the --debug option and include the stack trace that is displayed (the debug option will cause the full error trace to be displayed in the console). e.g. ``` synapse --debug get syn18838198 ``` Thanks!
"/path/to/dir" is just an alias. Dont ask me why but I dont feel comfortable to write the real paths of my server usually
@ginlucks I notice in the error message that you have chosen to download to a location on your system called, "/path/to/dir". Is that intentional? Does that location exist?

Cant download files using synapse get page is loading…