Hi, when I tried making a very basic submission last week my submission started running almost instantly. Today however it's been almost 4 hours since I made my submission and it still hasn't started. Is this because there's a queue of people that's trying to submit today, some problem with the server or with my submission? Asking mainly because I want to make my submission before this round ends later today. If there is some sort of queue is it possible to see how long it is? Thanks in advance!
Created by Dennis Svedberg sds Glad to hear it! It works now, thanks!
--(It seems the quota is still occupied by them. I got this message "Hongyang Li and Yuanfang Guan has reached the submission quota." when I tried to resubmit. On the dashboard, the "Workflow Status" cells are empty for them.)-- Hi all,
We should be back up and running now. All submissions will be processed in the order they were received.
The one exception are the two stalled submissions from @arcanum449 - you will need to resubmit, we invalidated those on our end so they do not count against your quota.
After we looked into the issue more, I think it was just bad luck that we hit a disk size limit when running your image. That should be resolved now, but we'll keep an eye on it.
Thanks! Our method does not require too much space. Actually, when I tested our docker yesterday, it worked fine without the disk quota error. I thought there were some new requirement about disk usage since this morning.
I checked our two submissions (ID 9701292 and 9701293) and they were still freezing. Is there a way we can stop them? So that they will not occupy the submission quota.
Thanks! It looks like we did run out of the space allocated to us by UAB - got filled up with tempfiles. @arcanum449 - without necessarily revealing what your approach is, do you expect it to require many GB of disk space? Just trying to figure out how best to accommodate this.
Thanks! It's not clear to me yet whether the error caused the crash or vice versa.
There is no disk space limit that we have _intentionally_ put into place but we'll look into whether there is something enforced by default that we missed. Between the workflow orchestrator, Singularity, and Slurm, there are a lot of places where this error could be happening :)
Thank you for the quick response, @allawayr !
We also received an error saying"OSError: [Errno 122] Disk quota exceeded". Is there a disk quota limit for the docker submission? Or this error was caused by the crash? No problem! I think we should have the server running again in an hour or so; it looks like something crashed and we'll need to restart it. Apologies for the inconvenience.
cc @HongyangLiandYuanfangGuan @allawayr Ok, thank you for the quick update! Hi @sds - Another team just reached out to us separately asking the same thing. It looks like there is a backlog of submissions - we're still trying to nail down the cause, but your submission is still in the RECEIVED state, so that is not the problem. Will follow up as soon as we figure this out!