Hi,
I am confused that I submitted the docker image that passed the local test in my environment, but I received the message of 'workflow failed' exactly 12 hours after submission. No errors were mentioned in the logs. I'd wonder what causes this result.
Tkanks.
Created by Jianing Chen RKrnn Hi @zengys,
The error your team encountered before might be due to the runtime limit at 12h, but the error was not returned - we have ensured to return the runtime limit error now.
FYI, we have made the improvement on logging the errors and the error for overloaded memory usage now is also captured, along with the error for runtime limit. We are also returning the tree structure of the output folder in your log folder.
I hope it can help you better understand the errors and ease your debugging process. Thank you. Hi @rchai ,
Thank you very much for pointing out our mistakes. It does help a lot. Hi @zengys,
I see your team's submission "9731402" and there are 18 of 27 imputed data with the correct format of filenames, `*_imputed.csv` and in the correct path. The error "5 file(s) not found" means the submission was killed before it was able to impute all the input data. It could be because of the overloaded memory usage right before the 12h. As a reminder, we do have the runtime limit at 12h and feel free to log in your model/docker (can be viewed in `{submission_id}_log.txt`) to help debug.
Below are some status of your team's another two running submission:
| NAME| MEM USAGE / LIMIT | MEM % | CREATED | OUTPUT # |
| 9731457 | 5.413GiB / 160GiB | 3.38% | 5 hours ago | 11/27 |
| 9731457 | 5.039GiB / 160GiB | 3.15% | 5 hours ago | 12/27 |
I hope it helps.
Thank you. Hi @rchai ,
I am a teammate of @@RKrnn ,
I received the email which informs that "5 file(s) not found : 'ds1c_p0125_n2_imputed.csv', 'ds1c_p0125_n3_imputed.csv', 'ds1c_p025_n1_imput...".
What bothers me is that I don't know the reason for this issue. Whether our program needs more than 12 hours to run, or the program has finished running, but no results are generated or the path of the results is wrong?
I can't find out that because the logs give no errors and I have no access to the docker container.
I?d appreciate it if you could answer my doubts.
Thanks. Hi @RKrnn,
The submission will be killed if the runtime reaches the limit (12h) for the Leaderboard phase. If so, you should be able to receive an email (Subject: ?Submission to 'Task 1 - scRNA-seq and scATAC-seq Data Analysis Challenge' invalid!? ) saying ?Submission time limit of 12h reached.
If not, can you please provide the submission ID, we can have a look.
Thank you.
Drop files to upload
Under what circumstances will a workflow be terminated after 12 hours? page is loading…