The connection is getting reset/lost when I utilize the python script to batch download the bam files in the folder. The connection is affected when downloading the files for more than 12 hours. Is there a way to resume downloading from the file where it was paused ?

Created by S Canchi scanchi
Thanks! I tried to reproduce using 1.6.1, setting the location directory, and with 'keep.local' set (though that shouldn't come into play). On my machine it worked as expected! However, I'm running Fedora Linux, not Windows 10. I might need to track down a Windows machine to try this on!
I have the synapse 1.6.1 version installed. I have specified a certain location for download and that directory already has around 50 files from this folder downloaded. Here are the steps that follow once I execute the script: 1. The location is set to the directory I mentioned 2. The download begin at the first file even though the file name already exists in the directory. 3. I can see the progress bar and the partial download files has this syntax - xxxxxxxx.bam.synapse_download_xxxxxxx 4. After complete download, the downloaded file replaces the previous file with the same name. Unless the files version in the folder changed in last two weeks, I am not sure why the download is overruling the local files.
@kdaily and @scanchi the ifcollision parameter only comes into effect if either the locally (already downloaded) file or the version of the file in Synapse has changed since the last download. @scanchi can you confirm that the files are actually being downloaded, i.e. do you see the progress bar and do you get a new file of the file with a different suffix. For example if you downloaded file_1.bam then restart the recursive download do you end up with a new file called file_1.bam(1)? Even when you restart a download the client will check every file already downloaded to make sure they haven't changed or that Synapse has a newer version of the file. This means that if you have 100 files already downloaded the command line client will report on every file and spend about .25-1 second communicating with Synapse to verify that the files are the latest. Then proceed to the rest of the files.
Specifically, is that version 1.6.1 or 1.6.2? (`synapse --version` will tell you). I was using `1.6.2`, but with the default value for `ifcollision='overwrite.local'` (what the command line `synapse get` does). Possible that it was a recent change in 1.6.2 ( @larssono, @ziming )?
Thank you Kenneth. I am currently using the synapseclient version 1.6 and Python v 3.5.1 on windows 10. Based on the source code, I set the collision parameter to keep the local files. Even though the bam files are present the folder, the download inevitably begins with the first file.
Hmmm. I could not reproduce your issue.   I tried using the most recent released version of the client (from `pip install synapseclient`) and tested the recursive download from the command line (`synapse get -r`). I used `Ctrl-C` to kill it midway - both in the middle of a file, and after some files had completed. Each time, it restarted as expected - partway through a file, or at the first file that wasn't downloaded.   Can you provide some more details on what you're doing - maybe which operating system as well? Looping in @ziming and @larssono, two of our other Python developers.
How would I address the recursive download when split into sessions? When I re-run the download script, It tends to start from the very first file all over again. How do I ensure the download continues from the last active point?
Thank you Kenneth for the information. Re-running the script helped start the process without re-downloading the files already downloaded. Since I have a previous version of the python client the download process overwrote on the partially downloaded files. I will upgrade to newer version before the next run.
Can you share the Python script are you using, or explain further how you are doing the batch download?   If you are asking if you can resume exactly at the point where it broke (assuming that your program exited because of a connection issue), the answer is sort of. Rerunning the program will not download files that were successfully downloaded again (though it will take a few seconds per file to ascertain that you already have the file).   If you are asking about resuming a partially downloaded file, the most recent version of the Python client (1.6.1) supports this, though there have been some cases where a file might retry to download from the beginning if an interruption occurred.

Problem with batch download due to connection reset (Folder ID = syn5958425) page is loading…