I have a job sent to the training lane, which is "VALIDATED" hence waiting to be executed. But it has been in this state for almost 30 hours. In my experience this too much time, but since I can't see my place in the queue I can really just guess...
Could someone have a look at submission 8014454?
ps- it would be a nice-to-have feature if we could see the job's position in the queue
Created by Jose Costa Pereira josecp @josecp Thanks to @brucehoff, you are now able to use the tip suggested earlier, that is, the system will redirect your submission to the first available machine if your submission specifies a new pre-processing container and if you don't have any submissions running that use the original pre-processed data (you can only have access to one `/preprocessedData` directory at a time).
Thanks for your feedback! Hi Jose,
> When submitting this submission, the system will detect that the pre-processed data must be recomputed because the pre-processing container is different and your job will be sent to the first machine available.
This tip is actually not yet available. It is currently being tested and should be deployed soon. Thank you @brucehoff, I can see that too.
When I posted previously in this thread both submissions were in "VALIDATED" status. My point was to see whether submission 8028966 (using a brand new pre-processing, and submitted Jan-16) would be executed **before** submission 8014454 (which used a pre-existing preprocessing, and submitted much earlier Jan-12). Submission 8014454 has now completed successfully and your submission 8028966 is running. I see no other submissions which are waiting to run.
I did something similar... tried to submit a new job with new preprocessing container that could be assigned to a machine with lower load.
For submission 8028966, I created another docker container "preprocess_alt" (different name, hence different digest signature) and submitted a new dual-container text file.
The first submission (8014454) is still in "VALIDATED" state (>6 days); and the last submission (8028966) also rests in "VALIDATED" state since Jan-16 (~2days). This is frustrating...
${imageLink?synapseId=syn8039955&align=None&responsive=true} Hi Jose,
> I know I'm sharing the hardware with others, but I've never witnessed a waiting time of this magnitude for the whole round 1.
The participation has increased in Round 2 compared to Round 1.
> I'd rather recompute the whole preprocess; this has been stuck for over 5 days (and counting!)
I've started a discussion with Bruce to see if we can allow you to notify the system of this preference. If your submission doesn't start shortly, you can rebuild your pre-processing container with a little modification to ensure that its digest changes. When submitting this submission, the system will detect that the pre-processed data must be recomputed because the pre-processing container is different and your job will be sent to the first machine available.
Thanks! Hi @tschaffter,
I know I'm sharing the hardware with others, but I've never witnessed a waiting time of this magnitude for the whole round 1.
Apparently because one of my submissions (8014454) is re-using some preprocessing, it was assigned to the machine where that data resided (irrespective whether another machine was free, I read that [here](https://www.synapse.org/#!Synapse:syn4224222/discussion/threadId=1488)). I'd rather recompute the whole preprocess; this has been stuck for over 5 days (and counting!)
Speaking of which, my other "Validated" submission (8028966) is using a **new preprocessing** ... and is waiting to be executed for 24hrs.
Could you double check that nothing is going on with the machines where these are queued?
Thanks Hi Jose,
You are sharing the hardware with other participants and jobs in this Challenge can typically take days to complete. Because participants have 2 weeks of computation time per Round, in the worst case scenario you may have to wait for two weeks if all the participants in the queue before you decide to use all their time quota at once. However in practice this is not the case, so hopefully a slot will become available for you shortly.
Thanks!
submission is stuck for 54hours in the "VALIDATED" state...
submission [8014454](https://www.dropbox.com/s/dvaleqr0cqf0666/r2_img1.png?dl=0)