Hello, Anybody experiencing a delay in getting leaderboard submission scored? Any idea how long it usually takes? I have submitted my predictions since last 2 hours and still waiting for them to be scored. Thanks.

Created by Muhammad Ali muhammad.ali
Yeah. Nathan and I are both on flights at the moment. So dont have access to the system right now. We expect to be able to fix these things in the next 5-6 hours. Will send a message when we think we've solved the problem. Then try again and let us know if it still is an issue. -Anshul.
Validation still happening ``` Hello maximus, Sorry, but we were unable to process your submission to the ENCODE-DREAM Transcription Factor Binding Site Prediction Challenge (LEADERBOARD). Please refer to the challenge instructions which can be found at https://www.synapse.org/#!Synapse:syn6131484/wiki/402044 and to the error message below: submission name: L.RFX5.HepG2.tab.gz submission ID: 7327206 The submitted filename (L.RFX5.HepG2.tab(2).gz) does not match expected naming pattern '[0-9\.]*([LFB])\.(.+?)\.(.+?).tab.gz$' If you have questions, please ask on the forums at https://www.synapse.org/#!Synapse:syn6131484/discussion/. Sincerely, the scoring script ```
My last submissions too hit the spurious (1) problem. But I got Anshul's mail asking us to hold off for a few hours, and will do so!
@nboley, @akundaje et al, thanks for your efforts! It looks like Nathan wouldn't have got much sleep at night. Have submitted a few, which hopefully will go through...
Folks. We are extending the deadline to Sunday mightnight Pacific Time (official email coming in a bit) to account for all these issues. We believe we have fixed the issues that were causing problems. So do try again. Note that scoring on the leaderboard in not instantaneous, it can take upto an hour depending on how many submissions were made. For the final round submission, after verification, the system should show your submissions on the final round board again ~within an hour or so. If you do not see this within an hour or 2 of submission, please let us know. Sorry for the trouble and thanks for your patience. These things tend to break at the most critical times! But we are on it and fixing issues as soon as we can. Thanks again, Anshul.
@rsidd120, @fschmidt, @simonvh : I've fixed another bug in the code related to re-submissions, and further increased the amount of disk space. Submissions *should* be working now -- although I realize that I've said that before. Best, Nathan
What is the current status? Is there an estimation when the evaluation works again?
Same issue with validation of the file name.. With regards to uploading an identical file, I had that before, I don't think it's related. The synapseclient indeed seems to check md5sum, modification date or something.
My second submission came back with the same "(1)" error. Even more bizarre -- I tried re-uploading both files and normally it takes 4-5 minutes but this time it claimed to have uploaded instantaneously. Either it is inferring from the file size that it's the same file and not bothering with the upload, or there's some other issue?
I experienced the same issue.
My submission of half an hour ago came back with the error ``` The submitted filename (L.ATF7.MCF-7.tab(1).gz) does not match expected naming pattern '[0-9\.]*([LFB])\.(.+?)\.(.+?).tab.gz$' ``` The filename I submitted did not have the "(1)" in it. Presumably there was a stray unscored file with the same name left over in the scoring area, so the (1) got tacked on?
Nathan, I had made one submission just before posting the previous message, half an hour ago, no scoring or error yet. Just made another.
@sidd120 I think so. Can you please try again? I'm very sorry about this.
It did not work for me 30 minutes ago.
Nathan, all submissions for the past 6 hours were rejected with the error message I posted (disk full). Are things normal now? I just tried submitting again.
As far as I can tell, the queue is currently up to date (and currently properly scoring new submissions). Do things look good from everyone else's end?
Emotions run wild in the heat of the battle! :D I guess it's not so easy to fix, otherwise it would have been fixed, no? That being said, it would be awesome for us to be able to check which final round submissions have been correctly added to the queue. Right now we don't have any way of knowing if a file has been submitted to the final queue or not.
Three submissions I made last night (about 8 hours ago) got scored eventually. But one (on YY1) was apparently scored according to an old version of the file, though the files page shows the new version! I resubmitted without re-uploading, 45 minutes ago, and then it got scored correctly. Some more submissions I made this morning (over half an hour ago) have not been scored, I imagine because there is a large queue. I have been seeing since yesterday that, though I felt sure I have uploaded a file, the timestamp sometimes doesn't reflect it and I need to re-upload. [edit] Now well over an hour since my last submissions. Is the queue stalled again? [edit2] The submissions mentioned above did get scored, but later ones haven't. The turnaround time seems to be well over an hour now, versus 10-15 minutes earlier. I know it is nighttime in the USA, but it does make it hard to evaluate our methods... [edit3] Disk full error? ``` Traceback (most recent call last): File "challenge.py", line 213, in score submission = syn.getSubmission(submission, downloadLocation=conf.SUBMISSION_DIR) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/synapseclient/client.py", line 2463, in getSubmission submission=submission_id, **kwargs) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/synapseclient/client.py", line 868, in _getWithEntityBundle self.cache.add(file_handle_id=entityBundle['entity']['dataFileHandleId'], path=downloadPath) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/synapseclient/cache.py", line 203, in add with Lock(self.cache_map_file_name, dir=cache_dir): File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/synapseclient/lock.py", line 100, in __enter__ self.blocking_acquire() File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/synapseclient/lock.py", line 80, in blocking_acquire lock_acquired = self.acquire(break_old_locks) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/synapseclient/lock.py", line 53, in acquire os.makedirs(self.lock_dir_path) File "/home/ubuntu/anaconda2/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/home/ubuntu/anaconda2/lib/python2.7/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/ubuntu/.synapseCache/410/13299410' ```
The problem is still not fixed. I really want to see whether my final submission is valid or not. Thank you, Yichao
Have waited for more than 6 hours for the leaderboard submission to be scored, but still nothing happens. This is really a great inconvenience since we need to determine which model to submit based on the leaderboard results.
Yup. Agreed. It is a problem and we are on it.
I understand people need to sleep and I can definitely identify with that. It would be good if the organisers could also acknowledge that this queue failure is a real problem for participants. This is the last day I had to do any checking of my models and because the leaderboard has been down I have not had the chance. In fact I still don't know if my submissions to the final queue will be accepted. Tomorrow I'll be up a mountain in the Dolomites so there's not much I can do if they are not. Best regards, Epi.
Ya, I'm checking this out right now.
Leaderboard rankings only get updated daily (not immediately after you submit). Leaderboard scoring speed depends on how many submissions are in the queue. Having said that, we will look into the queues if they are in fact stalled once we wake up!
Folks it was 3 am in the wee hours in the morning when u sent your email. We do need to sleep. Please remember that :)
Hi, i am also facing the same problem.
This is quite difficult for me as well. I now only have a couple of hours in which I can make my final submissions and I don't know if it is working.
The recent update: I have sent an email to organizers (Anshul, Nathan and James) and in a reply Anshul has asked Nathan to take a look (2 hours ago) and still nobody else has get back to me. This is really a great inconvenience at a very inappropriate time..!
I am waiting as well for both final round submissions and leaderboard submissions. Today is the last day it is possible for me to make any submissions so I really need to make sure they have worked correctly (and receive feedback from the leaderboard submissions). Thanks, Epi

Delay in Scoring Leaderboard Submissions page is loading…