Dear EHR DREAM Challenge Participants,
The NCATS server used by the following submissions queues is down:
- EHR DREAM Challenge Fast Lane (9614311)
- EHR DREAM Challenge Submission (9614309)
@thomas.yu will post an update here once we have resolved the issue.
Thanks!
Thomas
Created by Thomas Schaffter tschaffter Dear @ivanbrugere,
That is very strange indeed. When you submit, I can see that your container running, but when I manually extract logs, I don't actually see the model writing anything. I submitted some test models and I can successfully obtain logs from these models.
Best,
Tom @thomas.yu it is the same pipeline as 9697641 (which outputs logs) except I reduce the number of models to fit within time allotted. that model was before the server issue. I can try flushing but I've never had any special handling of it before. Dear @ivanbrugere ,
I was looking at your submission and stopped it manually. Are you writing to stdout or stderr? If you are in python, are you making sure to `flush` the print statements? I don't see any logs from your container. That is most likely why you aren't getting any log files for your training step. I submitted a test submission on my end and things are working as expected.
Best,
Tom ok. I just resubmitted. ID: 9697772 Dear @ivanbrugere and @maria_p ,
Apologies for the inconveniences. There are models that are completing on the NCATS server. Please submit again so that I can attempt to debug the situation.
Best,
Tom @ivanbrugere same here there may still be a problem with the server. the jobs on Fast Lane I submit do not create a training log at all, and eventually will time out. they should be completing in well under allotted time so I don't know. without the logs We will not be restarting any of the submissions that failed within that time period. Hi, I emailed you for cancelling the main submissions 9697685, 9697686. However, 9697686 is now restarted, and 9697685 is not yet. Could you check 9697685 is cancelled ? I don't want to suffer again by server issue and restarting the submission after mailing to notice fail without any other mail to notice restarting. Hello all,
Apologies for the inconveniences caused. The queues should be back up and running - please try re-submitting. None of your "invalid" submissions will count towards your submission quota.
Best,
Tom My submission failed and workflow status is also blank. If you could please delete my three (apologies!) fast lane submissions 9697661, 9697666, 9697681, I'll just start over when it's resolved.
Thank you Hi, my submissions(ID:**9697673**,**9697674**) are blocked yesterday for this problem. And today I received **Failed Email** but the workflow status is also **BLANK**.
And new submission(ID:**9697689**) is failed immediately for Error:packages/toil/leader.py", line 246, in run STDERR: 2020-01-16T04:11:36.281998251Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore). Of cource without workflow status.
Can you fix the problem for me? Thank you. It's still not working for me The server is now back up. Thanks for your patience.