I'm seeing two errors when I try to run getRestFeatures.R 1. Line 35 > rest_json_files <- synDownloadTableColumns(actv_rest_syntable, "deviceMotion_walking_rest.json.items") 34270 total files requested: At least 10 are downloaded; 10000 will be retrieved in the next batch. Completed 0.7%. Completed 1.3%. Completed 2.0%. . . . Completed 100.0%........ This repeats a few times, each time reporting that more files have been downloaded. The last time, however, it reports: Completed 99.9%......... . . . filehandle3034892 failed: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: FDD208FF20F6ABDB) The whole process took about 6 hours, and there are 40+ GB of new data in my synapse cache. Since it did pull down a lot of data, I continued running the code line-by-line. Everything seems OK until line 62: 2. Line 62 > getRestFeatures(row$rest_json_file) There is no function getRestFeatures() defined. I thought that the sample code was intended to provide a basic example of feature extraction. Is that not true - is it just pulling out the data and we insert our own feature extraction function? Thank you.

Created by David Merberg dmerberg
mpowertools install is failing on a windows platform because of "doMC" ... multicore dependency ... log file below. ``` > library(devtools); devtools::install_github("Sage-Bionetworks/mpowertools"); Downloading GitHub repo Sage-Bionetworks/mpowertools@master from URL https://api.github.com/repos/Sage-Bionetworks/mpowertools/zipball/master Installing mpowertools Installing 1 package: changepoint Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/changepoint_2.2.2.zip' Content type 'application/zip' length 528222 bytes (515 KB) downloaded 515 KB package ?changepoint? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: dplyr Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) There is a binary version available (and will be installed) but the source version is later: binary source dplyr 0.5.0 0.7.2 trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/dplyr_0.5.0.zip' Content type 'application/zip' length 2520396 bytes (2.4 MB) downloaded 2.4 MB package ?dplyr? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: fractal Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/fractal_2.0-1.zip' Content type 'application/zip' length 528956 bytes (516 KB) downloaded 516 KB package ?fractal? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: ggplot2 Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) Warning: package ?ggplot2? is in use and will not be installed Installing 1 package: lomb Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/lomb_1.0.zip' Content type 'application/zip' length 45701 bytes (44 KB) downloaded 44 KB package ?lomb? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: plyr Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/plyr_1.8.4.zip' Content type 'application/zip' length 1121290 bytes (1.1 MB) downloaded 1.1 MB package ?plyr? successfully unpacked and MD5 sums checked Warning: cannot remove prior installation of package ?plyr? The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: pracma Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) There is a binary version available (and will be installed) but the source version is later: binary source pracma 2.0.4 2.0.7 trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/pracma_2.0.4.zip' Content type 'application/zip' length 1054259 bytes (1.0 MB) downloaded 1.0 MB package ?pracma? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: qpcR Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/qpcR_1.4-0.zip' Content type 'application/zip' length 4555290 bytes (4.3 MB) downloaded 4.3 MB package ?qpcR? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: tester Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/tester_0.1.7.zip' Content type 'application/zip' length 185218 bytes (180 KB) downloaded 180 KB package ?tester? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages Installing 1 package: tidyr Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) There is a binary version available (and will be installed) but the source version is later: binary source tidyr 0.6.1 0.7.1 trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/tidyr_0.6.1.zip' Content type 'application/zip' length 802821 bytes (784 KB) downloaded 784 KB package ?tidyr? successfully unpacked and MD5 sums checked The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages "C:/PROGRA~1/R/R-32~1.3/bin/x64/R" --no-site-file --no-environ --no-save --no-restore --quiet CMD INSTALL \ "P:/__temp__/RtmpGqRjLh/devtools361456033a4/Sage-Bionetworks-mpowertools-8bccc2b" \ --library="P:/__SYSTEM/Documents/R/win-library/3.2" --install-tests ERROR: dependencies 'plyr', 'doMC' are not available for package 'mpowertools' * removing 'P:/__SYSTEM/Documents/R/win-library/3.2/mpowertools' Error: Command failed (1) > library(mpowertools); Error in library(mpowertools) : there is no package called ?mpowertools? > install.packages("plyr",dependencies=T); Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) also installing the dependencies ?abind?, ?foreach?, ?doParallel?, ?itertools? trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/abind_1.4-5.zip' Content type 'application/zip' length 40002 bytes (39 KB) downloaded 39 KB trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/foreach_1.4.3.zip' Content type 'application/zip' length 389142 bytes (380 KB) downloaded 380 KB trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/doParallel_1.0.10.zip' Content type 'application/zip' length 184498 bytes (180 KB) downloaded 180 KB trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/itertools_0.1-3.zip' Content type 'application/zip' length 79805 bytes (77 KB) downloaded 77 KB trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/plyr_1.8.4.zip' Content type 'application/zip' length 1121290 bytes (1.1 MB) downloaded 1.1 MB package ?abind? successfully unpacked and MD5 sums checked package ?foreach? successfully unpacked and MD5 sums checked package ?doParallel? successfully unpacked and MD5 sums checked package ?itertools? successfully unpacked and MD5 sums checked package ?plyr? successfully unpacked and MD5 sums checked Warning: cannot remove prior installation of package ?plyr? The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages > install.packages("doMC",dependencies=T); Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) Warning message: package ?doMC? is not available (for R version 3.2.3) > library(devtools); devtools::install_github("Sage-Bionetworks/mpowertools"); Downloading GitHub repo Sage-Bionetworks/mpowertools@master from URL https://api.github.com/repos/Sage-Bionetworks/mpowertools/zipball/master Installing mpowertools Installing 1 package: ggplot2 Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) Warning: package ?ggplot2? is in use and will not be installed Installing 1 package: plyr Installing package into ?P:/__SYSTEM/Documents/R/win-library/3.2? (as ?lib? is unspecified) trying URL 'https://cloud.r-project.org/bin/windows/contrib/3.2/plyr_1.8.4.zip' Content type 'application/zip' length 1121290 bytes (1.1 MB) downloaded 1.1 MB package ?plyr? successfully unpacked and MD5 sums checked Warning: cannot remove prior installation of package ?plyr? The downloaded binary packages are in P:\__temp__\RtmpGqRjLh\downloaded_packages "C:/PROGRA~1/R/R-32~1.3/bin/x64/R" --no-site-file --no-environ --no-save --no-restore --quiet CMD INSTALL \ "P:/__temp__/RtmpGqRjLh/devtools361434875b0c/Sage-Bionetworks-mpowertools-8bccc2b" \ --library="P:/__SYSTEM/Documents/R/win-library/3.2" --install-tests ERROR: dependencies 'plyr', 'doMC' are not available for package 'mpowertools' * removing 'P:/__SYSTEM/Documents/R/win-library/3.2/mpowertools' Error: Command failed (1) > > version _ platform x86_64-w64-mingw32 arch x86_64 os mingw32 system x86_64, mingw32 status major 3 minor 2.3 year 2015 month 12 day 10 svn rev 69752 language R version.string R version 3.2.3 (2015-12-10) nickname Wooden Christmas-Tree ```
Thank you, yes I did install mpowertools. I just checked and I can find getRestFeatures at the prompt if I preface it with "mpowertools::". I'll add that to the line that invokes it in the sample code - then it should work.
@dmerberg - Did you install the `mpowertools` library? I find the function by calling `mpowertools::getRestFeatures`.
@sieberts I see two problems. The first (as you suggested) seems to be an intermittent outage in AWS. The `synDownloadTableColumns` function is designed to retry or (if you have to rerun it) to download just the missing content. I don't have any insight into the issue, "There is no function getRestFeatures()". If the script's author can't resolve the issue I can try to help but I would need to be pointed to the script.
Thank you. I'm trying to rerun it now. What do you think about the issue I mentioned at line 62? Is that function supposed to be defined in the sample code, or is it code that we supply ourselves?
This looks to me to be an AWS failure. Have you tried to re-run it? @brucehoff, do you have any insight?

Errors running getRestFeatures.R (sample code) page is loading…