How to use synapse get to resume from break-out?and how to speed up synapse get?do we have any other method to download data?
Thanks!
Created by zhangliandong Hi @jordank
Thanks for your reply!
Limiting taske into single_threaded mode works.
Thanks very much! @zhangliandong
Can you try adding the debug option to the Synapse client to see if there is an additional error log output displayed if an additional file does not download.
It can be added to the Synapse constructor as follows, e.g.
```
syn = synapseclient.Synapse(debug=True)
```
or by using the --debug option from the command line, .e.g
```
synapse --debug get -r syn3388564
```
Could the system you are running on be restricting the number of network requests your user is allowed to initiate? If so one possibility for limiting the client to one request at a time would be to put it into single_threaded mode (this would also have the affect of slowing the client down). You could do so by adding the following to any Synapse client Python code:
```
import synapseclient.core.config
synapseclient.core.config.single_threaded = True
```
Hi @jordank
Thanks very much!
I want to restrict the download speed because I find fast download speed may let download job stop.
I find synapse can use 4 minutes to download a 5Gb file,and the synapse will not download the next file.
Thanks! Hi @zhangliandong
Synapse does not have a built in feature for limiting its own bandwidth consumption, it will download from its network connections allow.
A possibility would be to use a separate network bandwidth shaping tool such as [wondershaper](https://github.com/magnific0/wondershaper/)wondershaper. It is a script that allows one to artificially constrain the network bandwidth of a specified network interface. Depending on the privileges of your user, it may require elevating to sudo permissions however, I don't now if you have privileges on your system to do so.
For example if the wondershaper script exsts in the current directory the following command will restrict the eth0 interface to 4Kbps download and 8Kbps upload. Any commands while wondershaper is in affect are restricted accordingly.
```
# limit eth0 interface to 4k download and 8k upload
sudo ./wondershaper -a etho -d 4096 -d 8192
```
```
# reset limits
sudo ./wondershaper -ac eth0
```
Can you say why you want to restrict the download speed? Are you on a shared system and need to limit your own bandwidth usage?
Thanks,
Jordan
Hi! @nicole.kauer
I have no idea about how Synapse check this files are download.I moved part files of the folder to another files,and I move this files back to same folders.Synapse also redownload the files.
How to download the rest files in this folders?
And I also have another question.how to restrict the python2 Synapse Client 1.9.4 download speed?
Thanks very much for your reply! Hi @jordank
Thanks for your reply! And sorry to reply you late,because I missed your reply message.
Truncated files are files which names have "synapse_download".
Maybe the internet speed caused this situation.The disk volume is enough for these files.When very small files are downloaded, same situation happened.
Recently my schools' internet speed was improved,python2 Synapse Client 1.9.4 can download files very fast,therefor the download jobs are shut down.
Do you know how to restrict the python2 Synapse Client 1.9.4 download speed?
Thanks again! Hi @zhangliandong
Can you describe what you mean by truncated files? Are the file names complete or do some have "synapse_download_" in them (indicating an in progress download).
Can you try the following command from the command line and show the output it generates? This is the same basic command as your script above but using the command line which will have some progress bars that will indicate some output.
```
synapse get -r syn3388564 --downloadLocation=/picb/lilab4/datasets/restricted/ROSMAP
```
Also can you ensure that you have adequate disk space on the disk volume that includes the /picb/lilab4/datasets/restricted/ROSMAP directory to store these files (this is a very large project to download).
You can check the disk space with the following command:
```
df -h /picb/lilab4/datasets/restricted/ROSMAP
```
I found when I use python2 Synapse Client 1.9.4, synapseutils.syncFromSynapse function can download bulk data,some completed files can be downloaded.
but the speed is slow,and maybe after 3 day this script will sleep,and no file will download.
when I use python3 Synapse Client 2.2.0,same python script.several truncated files are download,and no complete file will be downloaded.
my Linux version (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) That's interesting. Could you please let me know some information about your computing environment, such as the operating system and the versions of Python, synapseclient, and synapseutils?
I download bulk data,when I use Linux command line ,no file is download.And I use python to download data,I use my username and password,but only some file is download and then the program sleeps,no error messages.About 1 week,some truncated files are downloaded.
import synapseclient
import synapseutils
syn = synapseclient.Synapse()
syn.login('synapse_username','password')
files = synapseutils.syncFromSynapse(syn, ' syn3388564 ',,path="/picb/lilab4/datasets/restricted/ROSMAP") No problem.
I see that the response was edited. For clarification, if downloading to the same folder that some files have already been downloaded to, then the Synapse client should check whether the files exist there, yet. If they do (and the files haven't changed), then the client does not re-download the data. Thanks! I am not sure I understand what is meant by "How to...resume from break-out?" The speed of download is dependent on multiple factors, but I highly recommend keeping your version of Synapse Client up to date, if you are downloading outside of the web interface.
The Synapse Documentation has information on various ways to download. You can find the main [download guide here](https://docs.synapse.org/articles/downloading_data.html) and [how to download in bulk here](https://docs.synapse.org/articles/uploading_in_bulk.html).
Drop files to upload
Some question about data download page is loading…