Dear organizers, We submitted a docker image that we tested locally. In local everything runs smoothly, and after pushing the image we received a mail stating "Your submission (syn23633765) is valid!". However, 7min later we received another mail stating "Your workflow job, (submission ID 9710549), has failed to complete." Could you tell us what is the difference between a valid submission and a valid workflow, and how we can have the former and not the latter? That would help us debugging the docker image and the scripts within it. Apparently the prediction.csv file is not found, but in local it is actually generated and saved. Also, in a situation where the submission is valid but the workflow fails, does it count in the limit of the 5 valid submissions per challenge? Finally, the synthetic validation phase intended to debug such issues is over, but if other teams have such issues would it be possible to have the logs of the docker image to debug the pipeline within it? Thank you very much for your help, Best, Alexis Coullomb

Created by Alexis Coullomb Alexis_C
Dear @Michael.Mason, Alright, indeed we don't train at all on test data since patient outcomes are not available. We only perform a relatively simple normalization, and after simplifying our model our workflow runs also on test data. Thank you very much for all your explanations. Best, Alexis
Dear @Alexis_C , It is not that training is forbidden but that training is not possible. Participants do not have access to patient outcomes other than the synthetic data which lacks any true signal. However, normalization *is* possible here but we do not provide compute resources that allow for complex/time consuming procedures. Hope this helps, Mike
Dear @Michael.Mason, It is indeed what we are doing. The idea was to correct batch effects, as we noticed they had an important influence on predictions. We solved our issue by selecting important variables in local before pushing the image, so the model is much more lightweight. However we hope we didn't break rules by training online, is it forbidden? Also, why would training online not be ideal in this challenge, is it an issue in terms of mathematics, or hardware, or something else? Thank you for your advice, Regards, Alexis
Dear @Alexis_C , Is it possible your docker image is trying to do some training or normalization? Some Challenges are designed to have training done as part of the submission, this is not one of those Challenges. We find that some participants who have participated in cloud based training Challenges default to doing it in many challenges even those where it is not ideal. Kind Regards, Mike
Dear @Michael.Mason, Thank you for the information. We suspect our model takes too much resources, possibly memory, we are trying to simplify it to overcome this limitation, hoping it is indeed the reason it breaks on real data and not on mock data. Best, Alexis
Dear @Alexis_C , Correct there should be the same variable names between the synthetic and real data. Regards, Mike
Dear @vchung , Thank you very much for your answer, now we know we can try multiple things to debug the image for the real data without running out of valid submissions. Just to be sure: there is no difference in variables' names between mock data and real data right? Even for gene names, only values (and number of samples) are different right? Because I really wonder how the workflow runs on mock data and not on real data. Thanks again! Best, Alexis Coullomb
Thank you! In our case there is no error on the mock data. We'll try to debug.
Dear @Alexis_C , Thank you for participating in the Challenge! **after pushing the image we received a mail stating "Your submission (syn23633765) is valid!". However, 7min later we received another mail stating "Your workflow job, (submission ID 9710549), has failed to complete."**
The first email you received was to notify you that your model ran successfully against the synthetic data -- it is essentially the same validation email you received during the Synthetic Validation phase. The second email, on the other hand, was to notify you that it did not run successfully against the validation data. As you mentioned, the workflow did not complete because it was expecting a predictions.csv file that did not generate with the validation data. **in a situation where the submission is valid but the workflow fails, does it count in the limit of the 5 valid submissions per challenge?**
Great question. No, it will not count towards your submission limit, as the overall status of the submission is not ACCEPTED. If you check the [Submission Dashboard](#!Synapse:syn18404605/wiki/607232) (assuming you are the submitter for your team), the final status for your submission is INVALID. **the synthetic validation phase intended to debug such issues is over, but if other teams have such issues would it be possible to have the logs of the docker image to debug the pipeline within it?**
During this phase, your model continues to run against the synthetic data, so you should still have access to those Docker log files. I hope these helped answer your questions!

Valid submission but workflow failed page is loading…