Synapse Team, Unable to download longitudinal files. For a strange reason, i am unable to view any download options. In the past, was able to download 20,000 & now do not see that option.  The programmatic option also does not work either. Am I missing something? Help appreciated. Thanks, Rahul

Created by rahul pathri rpathri
@rchai Synapse Docker registry still has permissions issue when we try to push the image based on your instructions around providing userId. However, we preferred the last option of uploading all the files onto the project syn63900418 & have provided you with Admin access. From an operations standpoint, please refer to the TimBre_readMe.docx for the preprocessing, model execution and results thereon. Thanks for your help.
Hi @rpathri, We recently fixed a bug related to pushing images to our Docker registry. Could you please try pushing your image to the Synapse Docker registry again? If the issue persists, you can push the image to the GitHub Container Registry by following the [instructions](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry): - Authenticate with GitHub Container Registry using your own [GitHub Personal Access Token (PAT)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic): ``` echo "" | docker login ghcr.io -u --password-stdin ``` - Build your image locally and tag it using the format below: ``` docker tag : ghcr.io///: ``` - Push the tagged image to the GitHub Container Registry: ``` docker push ghcr.io///: ``` - Share the image in the Package Settings on GitHub Alternatively, you can upload all the necessary Docker files to your project on [syn63900418](https://www.synapse.org/Synapse:syn63900418) and grant me (@rchai) write access so I can assist further. Let me know if this works or if additional details are needed!
@rchai I am not sure we are able to push image onto the GitHub. However, when we tried to upload the files, the following error appears: https://drive.google.com/file/d/1SAA34GZuWO1WPFpzxY38LhduoqTSOGPN/view?usp=sharing Is there a better way to share the files. Upload elsewhere on Synapse platform
@rchai Please make sure you provide permissions for your gitHub account for our team to push the docker image
@rchai Ok great, we will push it to ghcr followed by the syn63900418 write access Thanks, Rahul
Hi @rpathri, Sorry for the late response. Based on the error, it seems you're having trouble connecting to the Synapse Docker registry, even though you have the necessary project permissions and credentials for pushing your image. Alternatively, could I assist you in pushing your image to your project through one of the following approaches? 1. Push your image to another registry, such as [GitHub Container Registry (ghcr)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry). - Update the visibility permissions as needed (note that all ghcr images are private by default). - My GitHub username is _rrchai_. 2. Share the necessary Docker files with me so I can build the image on my end. Afterward, please grant me (@rchai) write access to your project (syn63900418) so that I can push the image to your project. Thank you
This is a test reply. It will be deleted. This reply does not tag any users. It should be sent to all users that subscribe to this thread.
@rchai 1) Project_Sharing_Settings is in the following screenshot : https://drive.google.com/file/d/1m0X7pYk9iT0Il-1FD8OVA4ZWkPyZ8Q6Q/view?usp=sharing 2) The docker registry did not prompt for any credentials when used as follows : $ docker login docker.synapse.org (am I missing something here?) 3) However, we did use rpathri and an auth code (that I used to download longitudinal datasets) & tried to push the docker image which resulted in the access issue as well Thanks!
Thanks @rpathri. Could you please also share the current permission settings (Project Tools > Project Sharing Settings) for the project (syn63900418) you?d like to push to? I see you mentioned you created the project, but I just want to ensure that you have the appropriate permissions for it. Could you also confirm you're using the same account (@rpathri) to login to the Synapse Docker Registry? Thank you!
@rchai Yes, tried with the suggested login approach as opposed to the AuthCode please see the screenshot. https://drive.google.com/file/d/1KBeItMhO620YVyOAuidZiDBt-LXLgrRj/view?usp=sharing thanks!
Yes, tried with the suggested login approach as opposed to the AuthCode please see the screenshot. https://drive.google.com/file/d/1KBeItMhO620YVyOAuidZiDBt-LXLgrRj/view?usp=sharing thanks!
Hi @rpathri, Have you tried to login to the Synapse Docker Registry before your push the image? ``` $ docker login docker.synapse.org ```
Absolutely yes. The logging was successful via AuthToken used earlier for downloading the longitudinal datasets. The docker push command was used post this step. Please refer to the original screenshot just in case. Thanks!
@rpathri - Thank you. Just to confirm, you successfully logged in before this step?
Here is the command: $ docker push docker.synapse.org/syn63900418/dt_tb_model:version1 Here is a clearer image just in case: https://drive.google.com/file/d/1ZDZqxHa7ggsxoDATxM6N8d-Hm2PqfxCQ/view?usp=sharing Thanks!
@rpathri - It's almost impossible to read the images attached. Can you post the command you used that caused the failure? This will help us to better troubleshoot. Thanks!
@sieberts Quick update from earlier messages: 1) We have masked all of the pre-processing activity & created docker images 2) Successfully able to login with the same credentials used for downloading the longitudinal data 3) The docker push command however returns an access denied (image in the link below) 4) Project id is attached just in case we are doing something wrong (image in the link below) https://drive.google.com/drive/folders/1OxEX_IrL4U_XVUII9FQ0RQJRd7pxwSfZ?usp=sharing
@sieberts Acknowledged with thanks. Will figure out a way to mask the non-ML preprocessing code
Models must be submitted for evaluation on the Holdout data in Docker format. See [the instructions](https://www.synapse.org/Synapse:syn31472953/wiki/625268) for further details. Your Docker submission does not need to be shared publicly if your method is proprietary.
@sieberts Thanks for reiterating that. 1) We plan to submit a ML model during this month that involves non-ML based pre-processing on the holdout data set that is proprietary in nature. Since we have access to 10,008 holdout StudyIds provisioned by UCSF, we can run the predictions & submit the Python ML code, binary results & probability scores (per-individual or per-cough which are identical) as done in the past. This may include Montreal data sets, but I am not sure at this time. Please let us know if this is acceptable 2) A DL model is planned for submission during next month that does not include any proprietary pre/post processing against the holdout sets Please advise on #1.
@rpathri Please see my previous message. The holdout set is not available for download. You may submit models to be evaluated against the holdout set, however. The instructions are linked in my previous message.
@sieberts Unable to access the holdout set from the link you shared. Please see screenshot below: https://drive.google.com/file/d/1g-Ga2Kf_I4oKW4VMQF9OX3_Z6sKX558h/view?usp=drive_link
Thanks for the response around scoring set & instructions to submit the subchallenges. We shall follow them accordingly.
@rpathri 1. The holdout set is not available for download. You're welcome to submit models for evaluation against the holdout set using the [instructions here](https://www.synapse.org/Synapse:syn31472953/wiki/625268). It contains data from both the UCSF (R2D2) and University of Montreal cohorts. 2. Per [the instructions](https://www.synapse.org/Synapse:syn31472953/wiki/625268) submissions are limited to one per month per subchallenge. 3. You can submit for evaluation in either or both subchallenges at your discretion.
@sieberts , Downloaded the entire training set (Longitudinal and Solicited), it took a while given our internal cloud migration. A few follow up questions: 1) Could I use the same holdout set received from UCSF folks for generating the results & probability scores or it is better to request the data on the synapse platform to avoid variance 2) We plan to submit two sets of results (Explainable ML using ensemble and DL models). Could we do that in one go or there are any restrictions as I remember seeing limitations around the number of submissions in a given month 3) Lastly, based on our earlier learnings, we would lean towards SC2 (demographic/clinical & WAV) & totally skip the SC1 approach (minimize submissions) Please advise.
Thanks & greatly appreciated. The authentication works fine and successfully ran a test download for 1k files. Please allow me some time to confirm the complete download.
@rpathri - I think I fixed the problem. Can you give it a try now?
@Hayley.Sanchez can you verify that @rpathri has access to the CODA TB data?
Very helpful & greatly appreciated. Been able to login successfully but the code results in an error message - synapseclient.core.exceptions.SynapseHTTPError: 403 Client Error: You lack READ access to the requested entity. The tokens have all permissions explicitly granted as seen below & does not work for either solicited or longitudinal. view and download Permissions: ViewDownload Last used 3 minutes ago | Created 3 minutes ago ok this is not working for some reason Permissions: ViewDownloadModify Last used 7 minutes ago | Created 10 minutes ago
@rpathri - You'll need to set up an Auth Token for login: https://python-docs.synapse.org/tutorials/authentication/ I believe password-based authentication has been deprecated in the R and Python clients.
@sieberts Thanks for the link & while running the code, the credentials are not working & currently locked out for multiple failed attempts. I am currently using the MS-Authenticator to logon to the portal & wondering if there is a dependency.
@rpathri - I would suggest downloading the data programmatically, as download will be extremely tedious and inefficient to do through the web interface. Python code to do this is available at the bottom of [this page](syn31472953/wiki/619711). The code traverses all the nested subfolders, which contain the data and downloads it.
In continuation to my previous query: The total number of files I am able to package are - "19,996 Files eligible for packaging" from both folders longitudnal_1 and longitudnal_2. How to access the remaining files Is there a way to get to all the 700k files. Thanks, Rahul

Longitudinal data page is loading…