Hallo, I submitted our current docker image to the queue Tumor_Deconvolution_Course_Fast_Lane and received the "Workflow Failed" email with the following error message: ``` -packages/toil/leader.py", line 246, in run STDERR: 2019-08-09T09:54:04.736390166Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2019-08-09T09:54:04.736462147Z toil.leader.FailedJobsException ``` However, the referred logs indicate that the prediction went through successfully and the error above does not come from our code. So I assume there is an issue with the output format. As suggested by the [example in the wiki](https://www.synapse.org/#!Synapse:syn15589870/wiki/592701) the programs writes `/output/predictions.csv` inside the container. I checked the [format requirements](https://www.synapse.org/#!Synapse:syn15589870/wiki/592699) but could not find a mistake myself. Here is an example output: ``` dataset.name,sample.id,cell.type,prediction ds1,Sample_2,NK.cells,0.008460063633931704 ds1,Sample_2,B.cells,0.555693389155673 ds1,Sample_2,neutrophils,0.0235319017167217 ds1,Sample_2,endothelial.cells,0.012669776438957215 ds1,Sample_2,CD4.T.cells,0.304119121113992 ds1,Sample_2,monocytic.lineage,0.0027554503876115856 ds1,Sample_2,CD8.T.cells,0.03422140548924458 ds1,Sample_2,fibroblasts,0.058548892063868235 ds2,Sample_2,NK.cells,0.013708125093493947 ds2,Sample_2,B.cells,0.2098375007162362 ds2,Sample_2,neutrophils,0.05019118896825339 ds2,Sample_2,endothelial.cells,0.009868866143038163 ds2,Sample_2,CD4.T.cells,0.39010817902598294 ds2,Sample_2,monocytic.lineage,0.0019179995900149194 ds2,Sample_2,CD8.T.cells,0.0033699235453414625 ds2,Sample_2,fibroblasts,0.32099821691763886 ds2,Sample_3,NK.cells,0.2411763339805771 ds2,Sample_3,B.cells,0.006513521273583505 ds2,Sample_3,neutrophils,0.02958000884147315 ds2,Sample_3,endothelial.cells,0.4730599853183344 ds2,Sample_3,CD4.T.cells,0.04606008110147513 ds2,Sample_3,monocytic.lineage,0.0045850637248808284 ds2,Sample_3,CD8.T.cells,0.18148191454105675 ds2,Sample_3,fibroblasts,0.01754309121861905 ds1,Sample_1,NK.cells,0.020833045777727453 ds1,Sample_1,B.cells,0.4592073430550949 ds1,Sample_1,neutrophils,0.26332797926300655 ds1,Sample_1,endothelial.cells,0.005710324511074445 ds1,Sample_1,CD4.T.cells,0.005442816656749977 ds1,Sample_1,monocytic.lineage,0.0029847992170402545 ds1,Sample_1,CD8.T.cells,0.23438432307430562 ds1,Sample_1,fibroblasts,0.008109368445000749 ds2,Sample_1,NK.cells,0.01709091879476407 ds2,Sample_1,B.cells,0.1077550094485702 ds2,Sample_1,neutrophils,0.24295567545788063 ds2,Sample_1,endothelial.cells,0.013071939458256836 ds2,Sample_1,CD4.T.cells,0.5486985575183547 ds2,Sample_1,monocytic.lineage,0.006547779837816695 ds2,Sample_1,CD8.T.cells,0.0039038497264632403 ds2,Sample_1,fibroblasts,0.059976269757893566 ``` What am I doing wrong? Kind regards, Dominik

Created by Dominik Otto djo
@djo How big is your container? This is just a general diskspace issue, which could also be caused by writing files from your docker image to disk. Currently the 12G is just how much space there is on the instance at the moment.
Hi @djo, @andrewelamb will have to help you with this. But I imagine he's going to ask ... how much disk space do you require? Thanks, Brian
All submissions to Tumor_Deconvolution_Coarse_Fast_Lane fail without producing any logs. The `Workflow Failed` email says ``` , disk, self.maxDisk) STDERR: 2019-08-13T14:59:51.775656724Z toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 2147483648, Available: 1209143296 ``` Is our container file too large?
Yes, both fast lanes will remain open for the remainder of the challenge. We would rather teams try their images here first and confirm they work before submitting to the other queues.
Thanks, @andrewelamb Is the fast lane still open? Round 1 officially starts today, correct? Do we still have unlimited submissions to "Tumor_Deconvolution_Coarse_Fast_Lane"? I would hate to burn one of our submissions on a failed workflow.
Hi @dxs435, This was on our end, but I've resolved this. Please submit your image again. Sorry for the confusion. -Andrew
Hi @andrewelamb, I'm also have difficulty getting my docker successfully through the workflow. It runs fine locally, but when I tried submitting it to "Tumor_Deconvolution_Course_Fast_Lane", it fails with the message, ``` ues are not allowed here STDERR: 2019-08-11T23:18:02.115594350Z in "file:///var/lib/docker/volumes/workfloworchestrator_shared/_data/eaf0886d-91dd-4da0-bdcf-b8128ba96c7a/Tumor-Deconvolution-Challenge-Workflow-master/run_docker.cwl", line 57, column 31 ``` What are "ues"? And how can I fix this issue? Thanks, David
Ahh I see. Good catch!
@andrewelamb when you look at the last column of [input.csv](https://github.com/Sage-Bionetworks/Tumor-Deconvolution-Challenge-Workflow/blob/master/example_files/fast_lane_dir/input.csv) in the Github repo, you will find ``` ensg.expr.file ds1_ensg.csv ds1_ensg.csv ``` instead of the expected ``` ensg.expr.file ds1_ensg.csv ds2_ensg.csv ``` So `ds2_ensg.csv` is never referenced in this `input.csv`. I just thought this may also be the case for the `input.csv` you use for the validation.
@djo I believe your last submission is my fault, I was upgrading to a bigger instance right when you submitted. I'm going to re-do it. I don't understand this: "I think ds2 Sample_3 is missing because both, ds1 and ds2, in input.csv link to the file ds1_ensg.csv in the column ensg.expr.file. So the container probably reads in the samples of ds1 and wrongly labels them ds2. So the issue may lie with input.csv."
I think ds2 Sample_3 is missing because both, ds1 and ds2, in [input.csv](https://github.com/Sage-Bionetworks/Tumor-Deconvolution-Challenge-Workflow/blob/master/example_files/fast_lane_dir/input.csv) link to the file `ds1_ensg.csv` in the column `ensg.expr.file`. So the container probably reads in the samples of ds1 and wrongly labels them ds2. So the issue may lie with `input.csv`. For my last submission (ID 9688317), which only produced dummy predictions (zeros everywhere), the error message in the "Workflow Failed" email was different: ``` ully: 'file:///var/lib/docker/volumes/workfloworchestrator_shared/_data/54416b63-d25e-4737-b723-2619e79ec5cf/Tumor-Deconvolution-Challenge-Workflow-master/run_docker.cwl' python runDocker.py r/O/jobaDfewO RAW: $ RAW: Error grabbing logs: unexpected EOF ``` The only difference in the output to the last submission should be the prediction values and runtime.
Hi @andrewelamb, Thank you for the replay. Locally the container runs as expected and even recognizes ds2 Sample3. I will submit a quick running validation when the queue is back online. Best, Dominik
Hi @djo I agree your returned error file isn't very helpful. So I'm trying to rerun your most recent submission to see what happens, and its taking a very long time to process. The fast lane queue is a very small dataset. I'm concerned when we impose a time limit on the much larger leaderboard dataset your image won't process it quick enough. I noticed you have a submission in the queue now that started an hour and half ago and is still running. Time limits for the actual challenge will likely be 30 minutes. I would look into that first. The next thing I would do is try to get your image to run locally. You can get the fast lane input data [here.](https://github.com/Sage-Bionetworks/Tumor-Deconvolution-Challenge-Workflow/tree/master/example_files/fast_lane_dir) Then run your docker image like so: docker run --network none -v /path/to/input_files/:/input:ro -v /path/to/output_directory/:/output:rw your_docker_image This should be VERY quick, existing tools out there process this dataset in less than a minute usually. While I don't see anything wrong with the above prediction file, I did notice that that the prediction file your last submission(9688261) made looks like ``` dataset.name,sample.id,cell.type,prediction ds2,Sample_1,NK.cells,0.01882572118572418 ds2,Sample_1,B.cells,0.5949276162782124 ds2,Sample_1,neutrophils,0.02721957883527313 ds2,Sample_1,endothelial.cells,0.006663417021077549 ds2,Sample_1,CD4.T.cells,0.005435539314890897 ds2,Sample_1,monocytic.lineage,0.002719228295619729 ds2,Sample_1,CD8.T.cells,0.2897802326286082 ds2,Sample_1,fibroblasts,0.05442866644059384 ds2,Sample_2,NK.cells,0.007131439457835174 ds2,Sample_2,B.cells,0.57528326373965 ds2,Sample_2,neutrophils,0.022220388586078613 ds2,Sample_2,endothelial.cells,0.015451863651464382 ds2,Sample_2,CD4.T.cells,0.2671698132443259 ds2,Sample_2,monocytic.lineage,0.002856948268220841 ds2,Sample_2,CD8.T.cells,0.04174323777064019 ds2,Sample_2,fibroblasts,0.06814304528178493 ds1,Sample_1,NK.cells,0.019291704995738693 ds1,Sample_1,B.cells,0.5975955747595768 ds1,Sample_1,neutrophils,0.03017055554027452 ds1,Sample_1,endothelial.cells,0.00622821628524451 ds1,Sample_1,CD4.T.cells,0.0054989848855644795 ds1,Sample_1,monocytic.lineage,0.0026653934750540662 ds1,Sample_1,CD8.T.cells,0.29193735407747307 ds1,Sample_1,fibroblasts,0.046612215981073936 ds1,Sample_2,NK.cells,0.007044777479960659 ds1,Sample_2,B.cells,0.5728128467685222 ds1,Sample_2,neutrophils,0.021943882278400748 ds1,Sample_2,endothelial.cells,0.015486879498794365 ds1,Sample_2,CD4.T.cells,0.26607062211702714 ds1,Sample_2,monocytic.lineage,0.002879703684228163 ds1,Sample_2,CD8.T.cells,0.041516209316657005 ds1,Sample_2,fibroblasts,0.07224507885640973 ``` It's missing ds2 Sample_3 predictions.

unable to submit successful workflow page is loading…