The submision does not find the directories
```
STDERR: find: `/preprocessedData': No such file or directory
STDERR: ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device gpu failed:
STDERR: initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1
```
As you can see also the GPU initialization find fails of memory.
Is there something we can do to check what is going wrong
Created by Kiko Albiol kikoalbiol Kiko:
> WILL be available as a temporary large directory, and should be inmutable during the contest. Is this true?
If your submission includes a preprocessing step then /preprocessingData is available as a writable directory during the preprocessing step and as a readable directory during the training step. The volume is 10TB in size. If you submit a series of submissions, each including a preprocessing step then, if the preprocessing step is the same in all submissions the system will note this and avoid repeating the preprocessing step. If you change the preprocessing step from one submission to the next then the system will clear the 10TB volume and perform the new preprocessing step.
> Still not working...
The system works as designed, but we may need to clarify the instructions. You are to visit the page for your project, syn7349799, and 'share' with the challenge infrastructure account, 'dmchallengeit'. We will modify the notification to include these instructions. Still not working...
The submission provides this information:
```
Dear 42 Is the answer:
Your Submission to the Digital Mammography challenge, syn7435757 (ID 7466949), was invalid. The message is:
The repository for image docker.synapse.org/syn7349799/vanilla_convolucional@sha256:ec32bfcda96dfd8a7ad802fa4e4d7d921e86866cf184cf67e4650507b0e883b6 does not exist or is not accessible by the submission processing infrastructure.
Please direct any questions to the challenge forum, https://www.synapse.org/#!Synapse:syn4224222/discussion .
Sincerely,
Challenge Administration
```
So the design is a bit confusing. If you don't provide the exact same directories in every working stage you can provide false or logic problems
In this case what I understand is that you need to separate training from preprocesing, What I understand is that:
```
/preprocessingData
```
WILL be available as a temporary large directory, and should be inmutable during the contest. Is this true? Update: Resubmitting yields an error condition with the following log:
```
STDOUT: REMOVIG DATA, PLEASE NEXT TIME REMOVE OR COMMENT THIS LINE
STDOUT: REMOVIG DATA, PLEASE NEXT TIME REMOVE OR COMMENT THIS LINE
STDOUT: REMOVIG DATA, PLEASE NEXT TIME REMOVE OR COMMENT THIS LINE
STDOUT: REMOVIG DATA, PLEASE NEXT TIME REMOVE OR COMMENT THIS LINE
STDOUT: REMOVIG DATA, PLEASE NEXT TIME REMOVE OR COMMENT THIS LINE
STDERR: find: `/preprocessedData': No such file or directory
STDERR: Using gpu device 0: Tesla K80 (CNMeM is enabled with initial size: 10.0% of memory, cuDNN 5103)
STDERR: Traceback (most recent call last):
STDERR: File "/root/train_programs/vanilla_convolutional.py", line 44, in
STDERR: from DreamChallengeDataset import DreamChallengeDataset
STDERR: File "/root/train_programs/DreamChallengeDataset.py", line 5, in
STDERR: import image_contest_generator_api
STDERR: ImportError: No module named image_contest_generator_api
```
**The system is working as designed:** As explained under "Training Submissions" on the page: https://www.synapse.org/#!Synapse:syn4224222/wiki/401759
when you do **not** include a preprocessing step then the input folder mounted to run your model is `/trainingData`. You will not find a preprocessedData folder. If you wish to have preprocessing as well as training, then prepare a submission as described under "Training with Preprocessing".
I hope this explanation helps. If you feel the documentation is unclear, that is something we can and should address. If you can point to something specific it would help us revise the doc's.
I understand there is the separate issue of GPU memory. Let us track that issue under @snaggle 's thread https://www.synapse.org/#!Synapse:syn4224222/discussion/threadId=1122
It looks like you submitted this Docker image twice:
https://www.synapse.org/#!Synapse:syn7435757
${leaderboard?queryTableResults=true&path=%2Fevaluation%2Fsubmission%2Fquery%3Fquery%3Dselect%2B%2A%2Bfrom%2Bevaluation%5F7213944%2Bwhere%2BUSER%5FID%253D%253D%25223343783%2522&paging=false&pageSize=100&showRowNumber=false&columnConfig0=none%2CSubmission ID%2CobjectId%3B%2CNONE&columnConfig1=none%2CStatus%2Cstatus%3B%2CNONE&columnConfig2=none%2CStatus Detail%2CSTATUS%5FDESCRIPTION%3B%2CNONE&columnConfig3=epochdate%2CLast Updated%2CmodifiedOn%3B%2CNONE&columnConfig4=synapseid%2CSubmitted Entity%2CentityId%3B%2CNONE&columnConfig5=none%2CSubmitting User or Team%2CSUBMITTER%3B%2CNONE&columnConfig6=synapseid%2CSaved Model State%2CMODEL%5FSTATE%5FENTITY%5FID%3B%2CNONE}
We will try to reproduce the error with this submission. Please post your Reason=CNMEM_STATUS_OUT_OF_MEMORY log on the thread below, to add some weight.
I provided full details there, but seemingly nobody has responded.
https://www.synapse.org/#!Synapse:syn4224222/discussion/threadId=1122
Drop files to upload
Not mounting expected directories page is loading…