Dear organizers,
We just submitted our first submission (a .zip file with executable singularity .sif file inside) for trial.
I have read from here: https://www.synapse.org/#!Synapse:syn27130803/wiki/619289, which says we will get feedback about our submission state via email. However, we haven't received any feedback in any form yet, including the leaderboard. I have some questions as follows:
1. How much time would it be before we get any feedback?
2. There are two ways of submission (.sif or .zip). Will this impact the way of validation? Is it automatic or carried out by researchers from the other side? Should we specify the usage in any documents (i.e. the wiki writeup)?
2. Will there be logs for the submission results, such as the errors the programs may encounter during invalid submissions? This may be important for debugging and adjusting the code. Even though we have made sure the code can be run through the simulated data, the real data may also lead to some unexpected errors.
Best,
Hanrui
Created by Hanrui Zhang rayezh Dear @rayezh,
Thank you for your submission, we confirmed that we received it.
Regarding the e-mail upon receiving, we, unfortunately, haven't managed to do so yet, but we will try to incorporate this in our workflow.
1. It really depends on the queue of submissions, you should expect to receive the feedback no later than 1 week after submission. If not, we will try our best to communicate the reason to your e-mail or in this forum if we received an extra load nearing the deadline.
We can process up to 5 submissions/day based on our quota to run this project on our server (can be adjusted if needed). And one it is evaluated, we will e-mail you whether the submission is SCORED or INVALID.
2. Thanks for this question, so far we managed to run the automatic process if the submission is submitted in a readily .sif file. We still have to tweak some of the steps with the .zip file submission. We will update you soon if we need any more assistance with the usage, but I think it is always good idea to provide us with 1 file that contain the 1 line command to run the submission without the argument (i.e file named Run), just in case there is a hiccup in our workflow so we can trace manually.
3. In case of INVALID submission, we will return the error log to your e-mail. In case of SCORED submission, we will update the leaderboard and send you an e-mail that your submission has been scored.
Best regards,
Pande