Hi,
in the last 2 days it seems that all my submissions have been queued up for an extended time in particular, the submission with ID 7813861, has been in the VALIDATED state for more than 10 hours.
on this thread : https://www.synapse.org/#!Synapse:syn4224222/discussion/threadId=1288
I saw someone else raise the same concern and in the moderator Mr Bruce Hoff (brucehoff) had this answer.
"I wouldn't use the word "stuck". The submission was queued on the server where your reprocessed data resides while the server ran other submissions. You will now find that 7812767 is in progress."
if I understood correctly, submissions will be assigned to the serve where the reprocessed data is stored, and will not be executed until other queued tasks are complete. Does this mean that we cannot use other servers computational resources and mount the storage of our prepossessed data? this seems to create congestion on the servers with prepossessed data, and longer wait times.
is it possible to use other idle servers and mount the preprocessed data to them?
Created by wissam Baddar wbaddar Thank you for your question, actually every submission we do has to wait for up to a day, but so far our submissions seem to be executing correctly . @wbaddar Thank you for these questions, which are very pertinent. We have considered how best to provision available resources so that they are useful to the participant community. As explained [here](https://www.synapse.org/#!Synapse:syn4224222/wiki/401759) (under 'Time Quota') our goal is to provide 14 days of server time to you for each five week round.
> 2- is it possible to know an estimate of the number of userers/machine to estimate the waiting time? the last submission was executed almost a day after the original submission time.
Estimating waiting time is difficult because we cannot tell how long a single submission will run. The run time can range from one minute to two weeks (at which time we would cut off the job). In the absolute worst case you would have to wait three weeks for your job to run, at which time you would have two full weeks of server access.
> is it possible to use other idle servers and mount the preprocessed data to them?
As @tschaffter explains, your preprocessed data is stored directly on the local disk of the GPU-enhanced, bare metal machine, to maximize data access speed. As a result your submission is queued for the server on which the preprocessed data is stored, not for the server pool in general. (Queueing theory tells us that wait time is minimized by having all the servers pull from a single, common submission queue, but the caching of preprocessed data precludes our using this strategy.)
> 1- If i submit that submission file provided, wouldn't that mean loosing my stored preprocessed data?
Yes. Please don't submit an alternate preprocessing job if you intend to reuse your current preprocessing result for model training. If you choose to do so, then the current result would have to be recomputed when needed at a later time.
> so I was wondering if the server is operating well?
If you provide a Submission ID we can investigate the submission's current disposition.
1- If i submit that submission file provided, wouldn't that mean loosing my stored preprocessed data?
2- is it possible to know an estimate of the number of userers/machine to estimate the waiting time? the last submission was executed almost a day after the original submission time.
> Does this mean that we cannot use other servers computational resources and mount the storage of our prepossessed data?
This is correct. This allow fast read/write access to your pre-processed data, which is extremely important considering the size of the dataset (about 7 TB).
> this seems to create congestion on the servers with prepossessed data, and longer wait times.
Ultimately you would still be able to start using your 2 weeks of allotted computational time when a slot becomes free, either because the job of other participants have stopped or because they have used all their allotted time.
Can you try submitting this example [submission file](https://www.synapse.org/#!Synapse:syn7501865). Do you receive the email that says that the submission successfully completed?
Thanks! An update: I also tried to submit submit the docker into the express lane (twice), however the submission was not shown in the express lane submission list, so I was wondering if the server is operating well?