Hi,
I needed to download a whole folder containing all the IDAT files on methylation data (syn7357283) (https://www.synapse.org/#!Synapse:syn7357283). However, I noticed that the files within the IDAT folder can only be downloaded one by one (there are thousands of them). Is there any way that I can download the whole folder together? Appreciate any guidance to do this either through the website or through R.
Thank you!
regards,
TM Liew
Created by TM Liew tmliew There have been changes to the Entity query service (the thing you're trying to use). They are summarized in this post: https://www.synapse.org/#!SynapseForum:threadId=2004
For now, the best replacement is to use the R function `synRestPOST("/entity/children", ...)`, as a utility function has not been written for the R client:
```
# Create an entity children request object to get all files in the folder syn2775243
ecr <- list(includeTypes=list('file'), parentId='syn2775243')
# Get a list of children
results <- synRestPOST("/entity/children", ecr)
# Convert to data frame, similar to original queryResults example
queryResults <- data.frame(Reduce(rbind, results$page))
```
Unless file types were added as annotations to the files, there isn't a way to use a query to limit by file type. In your case, a regular expression package (I like [`stringr`](https://cran.r-project.org/web/packages/stringr/)) would help you limit using the file name column. Hi Kenneth,
Follow up on your previous post, I have additional questions after trying the R code you proposed.
I noticed that synQuery only returns a maximum of 1000 results.
-Is this normal?
-What do I do if I need to get beyond the 1000 results (for example, when there are more than 1000 files)?
Among the files in the folder-of-interest, there are various type of files (for example, .jpeg, .txt and .idat). If I am only interested to download .idat files and not the others, how do I modify the synQuery code? I looked through the annotations but did not find a way to do this by querying the annotations.
Thank you
regards,
TM Liew That depends on what operating system you are using. If you are using Windows, this approach may not be the best (or, I am not the best person to tell you how to do that!). If you are using a Mac or Linux, the 'command line' is commonly known as the 'Terminal'. On a Mac, see [here](http://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line) for some documentation on how to use this. You might ask (if you are at an academic institution) for some assistance from your IT staff.
I think your case is easily solved using R though. As noted on the [downloading data](http://docs.synapse.org/articles/downloading_data.html#finding-and-downloading-files) help page, you can use properties of Synapse files to download in batch. Every file has a property that says which folder it is in (called the `parentId`). You can use this with a query to find all files in a specific folder, and use R to download them all.
Since you want files from folder syn7357283, this would be the `parentId` of interest. So, you can do this:
```r
library(synapseClient)
synapseLogin()
# Run the query to find the Synapse IDs of all the files in Folder syn7357283
queryResults <- synQuery('select id from file where parentId=="syn7357283"')
# loop over the results and download the files
entityList <- lapply(queryResults$file.id, synGet)
# Get a list of the paths to the files downloaded, as they will be in a special
# cache directory for Synapse files
filePathList <- lapply(entityList, getFileLocation)
```
If you want to download the files to a specific location, you can do this instead:
```r
entityList <- lapply(queryResults$file.id, synGet, downloadLocation='/path/to/my/directory')
```
The result of this code will be the same as if you were using the command line or Terminal.
Thank you for the reply. From which "programme" or environment do I run the "command line"? (For example, for R code, I will run the codes within the R programme but what about the "command line"?)
Please forgive my ignorance. Hello,
Go to the docs page, and install the command line client here: http://docs.synapse.org/articles/getting_started.html#installing-synapse-clients Hi,
Thank you for the reply.
It seems from the references you provided, it is easiest to do large downloading through "command line". Can I clarify how do I get access to the "command line" (as opposed to say R program)?
Thank you
regards,
TM Liew See this [Wiki page](https://www.synapse.org/#!Synapse:syn2580853/wiki/409846) in the AMP AD Knowledge Portal for examples.
Also, see this [forum post](https://www.synapse.org/#!SynapseForum:threadId=544) for more examples on downloading in batch focused on using the command line with files that have annotations *(this feature will change in the future).*
Lastly, more general information about query and download can be found in our documentation site: http://docs.synapse.org/articles/downloading_data.html#recursive-downloads
Let us know if you are still having difficulties. Thanks!