Synapse Team,
Unable to download longitudinal files. For a strange reason, i am unable to view any download options.
In the past, was able to download 20,000 & now do not see that option.
The programmatic option also does not work either. Am I missing something? Help appreciated.
Thanks,
Rahul
Created by rahul pathri rpathri @sieberts
Acknowledged with thanks. Will figure out a way to mask the non-ML preprocessing code
Models must be submitted for evaluation on the Holdout data in Docker format. See [the instructions](https://www.synapse.org/Synapse:syn31472953/wiki/625268) for further details. Your Docker submission does not need to be shared publicly if your method is proprietary. @sieberts
Thanks for reiterating that.
1) We plan to submit a ML model during this month that involves non-ML based pre-processing on the holdout data set that is proprietary in nature. Since we have access to 10,008 holdout StudyIds provisioned by UCSF, we can run the predictions & submit the Python ML code, binary results & probability scores (per-individual or per-cough which are identical) as done in the past. This may include Montreal data sets, but I am not sure at this time. Please let us know if this is acceptable
2) A DL model is planned for submission during next month that does not include any proprietary pre/post processing against the holdout sets
Please advise on #1. @rpathri
Please see my previous message. The holdout set is not available for download. You may submit models to be evaluated against the holdout set, however. The instructions are linked in my previous message. @sieberts
Unable to access the holdout set from the link you shared. Please see screenshot below:
https://drive.google.com/file/d/1g-Ga2Kf_I4oKW4VMQF9OX3_Z6sKX558h/view?usp=drive_link
Thanks for the response around scoring set & instructions to submit the subchallenges. We shall follow them accordingly. @rpathri
1. The holdout set is not available for download. You're welcome to submit models for evaluation against the holdout set using the [instructions here](https://www.synapse.org/Synapse:syn31472953/wiki/625268). It contains data from both the UCSF (R2D2) and University of Montreal cohorts.
2. Per [the instructions](https://www.synapse.org/Synapse:syn31472953/wiki/625268) submissions are limited to one per month per subchallenge.
3. You can submit for evaluation in either or both subchallenges at your discretion. @sieberts ,
Downloaded the entire training set (Longitudinal and Solicited), it took a while given our internal cloud migration. A few follow up questions:
1) Could I use the same holdout set received from UCSF folks for generating the results & probability scores or it is better to request the data on the synapse platform to avoid variance
2) We plan to submit two sets of results (Explainable ML using ensemble and DL models). Could we do that in one go or there are any restrictions as I remember seeing limitations around the number of submissions in a given month
3) Lastly, based on our earlier learnings, we would lean towards SC2 (demographic/clinical & WAV) & totally skip the SC1 approach (minimize submissions)
Please advise. Thanks & greatly appreciated. The authentication works fine and successfully ran a test download for 1k files. Please allow me some time to confirm the complete download. @rpathri -
I think I fixed the problem. Can you give it a try now? @Hayley.Sanchez can you verify that @rpathri has access to the CODA TB data? Very helpful & greatly appreciated. Been able to login successfully but the code results in an error message - synapseclient.core.exceptions.SynapseHTTPError: 403 Client Error:
You lack READ access to the requested entity.
The tokens have all permissions explicitly granted as seen below & does not work for either solicited or longitudinal.
view and download
Permissions: ViewDownload
Last used 3 minutes ago | Created 3 minutes ago
ok this is not working for some reason
Permissions: ViewDownloadModify
Last used 7 minutes ago | Created 10 minutes ago @rpathri -
You'll need to set up an Auth Token for login: https://python-docs.synapse.org/tutorials/authentication/
I believe password-based authentication has been deprecated in the R and Python clients. @sieberts
Thanks for the link & while running the code, the credentials are not working & currently locked out for multiple failed attempts. I am currently using the MS-Authenticator to logon to the portal & wondering if there is a dependency. @rpathri -
I would suggest downloading the data programmatically, as download will be extremely tedious and inefficient to do through the web interface. Python code to do this is available at the bottom of [this page](syn31472953/wiki/619711). The code traverses all the nested subfolders, which contain the data and downloads it. In continuation to my previous query:
The total number of files I am able to package are - "19,996 Files eligible for packaging" from both folders longitudnal_1 and longitudnal_2. How to access the remaining files
Is there a way to get to all the 700k files.
Thanks,
Rahul