This thread is the designated place to ask questions about running the **knoweng_gene_prioritization** workflow.
Created by James Eddy jaeddy Hi all!
I noticed one interesting thing, which is not a bug, but it's better to be fixed or at least noticed.
Each time when the workflow is started, `kn_fetcher` and `data_cleaning` steps fetch some data from the Internet (about 480MB).
So if the platform where I run my workflow management system doesn't have Internet connection or there are any other reasons that
prevent access to Internet from inside the container (e.g. proxy or --network="none"), the workflow will always fail. Thanks for noticing that. I updated the sharing settings for the image, should be ok now.
I don't know about PM on synapse, we can chat via Direct message on Dream Challenge slack. @bogdang : And thanks to you (and your co-workers) for working on compatibility for rabix/CGC.
There is one section of your report that I can't see (even after logging in). It says:
```
Completed task should look something like this:
Sorry, you do not have sufficient privileges for access. Forbidden
```
Also, I have some other questions about rabix, unrelated to this workflow -- can you help me with that? Is there a PM-like feature here on synapse? Basically I want to be able to run rabix on the command line on some CWL I've exported from the CGC (and have it run on my machine, no involvement of the CGC at all). I tried that and I get some errors.
Thanks.
Milt
Hi @mepstein we made the fixes required to run the workflow on [Cancer Genomics Cloud](https://cgc.sbgenomics.com/), I made a [synapse report](syn11414021) for it. Thanks again for all the help on enabling compatibility for running the workflow. Hey @vipin @mepstein
Rabix [release 1.0.2.](https://github.com/rabix/bunny/releases/tag/v1.0.2) is out and can run the gene prioritization workflow. Thanks for all the feedback and for updating the workflow to have it running.
Best,
Bogdan @vipin : I had uploaded the modified CWL files to synapse (and github), so, as @bogdang states, they are available there. So. for example, you can just re-run the "synapse get ..." command.
Thanks all for the efforts to get this working with rabix! @mepstein Thanks a lot for the updated CWL, I tested with the patched version of rabix and can confirm that everything completes and the outputs are validated. We'll make a fresh rabix release today or some time next week, so I will submit step-by-step instructions on Synapse.
@vipin You can get the latest changes either by pulling again from synapse or from [github](https://github.com/KnowEnG/cwl-gene-prioritization) Hi Bogdan,
Thanks for pointing that we all are in the same page!
Yes I saw the points which you mentioned, from the STDOUT.
@mepstein: I can try with the recent dev version of `rabix`. But can I have the recent changed version of `kn_fetcher.cwl` and `data_cleaning.cwl`. Or let me know what is the difference so that I can adjust that here in my side.
Thanks for the help!
Vipin @bogdang : OK, I've updated the CWL. At first I just updated data_cleaning.cwl, because that's the one where I had run into this problem with escape characters before. But from what you wrote about the most recent issue @vipin had, it sounds like this may also be an issue with kn_fetcher.cwl, so I updated that one too. It works with cwltool, but I don't have a patched/rebuilt version of rabix to try it with. @vipin , hopefully you can retry it. @mepstein Thanks for the info, this is the same thing I did. Good to hear that it works in cwltool as well. Could you maybe update the CWL with this workaround so it can work with both cwltool and rabix?
@vipints Thanks for testing with rabix. This is the exactly the bug that @mepstein mentioned.
```
Failed evaluating expression (inputs.genomic_file.path)
```
from the log. This happened because previous tool did not output the `genomic_file`, and this happened because the command line for that previous tool was not properly built with rabix. You can compare command lines that cwltool and rabix build for certain tools, you will see that rabix has some extra backslash characters. For example, in `kn_fetcher` the cmd line starts with `MYCMD=date && if [ \\true\\ = \\true\\ ]...` for rabix run, while for cwltool it starts with `MYCMD=date && if [ \true\ = \true\ ]...`
Hi Bogdan,
Thanks! I have followed the steps you mentioned and created a dev version of the `rabix` executable. Then start using with the dev version of rabix for my run and now it return a different error message with failure to start container. Here is the complete log:
```shell
$ ~/tools/rabix-cli-dev/rabix-cli-1.0.1/rabix --tmpdir-prefix ~/tmp/ --tmp-outdir-prefix ~/tmp/ ./gp_workflow.cwl gp_workflow_job.yml
[2017-10-12 16:22:00.514] [INFO] Job root.data_cleaning has started
[2017-10-12 16:22:00.513] [INFO] Job root.kn_fetcher has started
[2017-10-12 16:22:00.554] [INFO] Pulling docker image quay.io/cblatti3/kn_fetcher:latest
[2017-10-12 16:22:01.607] [INFO] Running command line: MYCMD="date && if [ \\"true\\" = \\"true\\" ]; then /home/kn_fetcher.sh KnowNets/KN-20rep-1706/userKN-20rep-1706 Gene 9606 STRING_experimental; else touch empty.edge; fi && date" && echo $MYCMD > run_fetch.cmd && eval $MYCMD
[2017-10-12 16:22:02.095] [INFO] Pulling docker image knowengdev/data_cleanup_pipeline:07_26_2017
[2017-10-12 16:22:04.984] [INFO] Running command line: echo "
pipeline_type: gene_prioritization_pipeline
spreadsheet_name_full_path: /home/nexcbu/dream_challenge/2017/workflow_execution/workflows/knoweng_gene_prioritization/data/Hsap.ccle.G.qnorm_probe.pos.df
taxonid: \'9606\'
redis_credential:
host: knowredis.knoweng.org
password: KnowEnG
port: 6379
source_hint: \\'\\'
results_directory: ./
" > run_cleanup.yml && \
if [ "Hsap.ccle.P.cyto_ic50.pos" != "sh" ]; then \
echo "phenotype_name_full_path: /home/nexcbu/dream_challenge/2017/workflow_execution/workflows/knoweng_gene_prioritization/data/Hsap.ccle.P.cyto_ic50.pos.df" >> run_cleanup.yml; fi && \
if [ "gene_prioritization_pipeline" = "gene_prioritization_pipeline" ]; then \
echo "correlation_measure: pearson" >> run_cleanup.yml; fi && \
date && python3 /home/src/data_cleanup.py -run_directory ./ -run_file run_cleanup.yml && date
[2017-10-12 16:22:09.433] [INFO] Job root.kn_fetcher has completed
[2017-10-12 16:22:36.230] [INFO] Job root.data_cleaning has completed
[2017-10-12 16:22:37.200] [INFO] Job root.gp_runner has started
[2017-10-12 16:22:37.214] [INFO] Pulling docker image knowengdev/gene_prioritization_pipeline:07_26_2017
[2017-10-12 16:22:39.929] [ERROR] Failed to start container.
org.rabix.bindings.BindingException: org.rabix.bindings.cwl.expression.CWLExpressionException: Failed evaluating expression (inputs.genomic_file.path).
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:234) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineParts(CWLCommandLineBuilder.java:145) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineObject(CWLCommandLineBuilder.java:87) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLBindings.buildCommandLineObject(CWLBindings.java:98) ~[rabix-cli.jar:na]
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:217) ~[rabix-cli.jar:na]
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:195) [rabix-cli.jar:na]
at org.rabix.executor.execution.command.StartCommand.run(StartCommand.java:32) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerCommand.run(JobHandlerCommand.java:51) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerRunnable.run(JobHandlerRunnable.java:60) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: org.rabix.bindings.cwl.expression.CWLExpressionException: Failed evaluating expression (inputs.genomic_file.path).
at org.rabix.bindings.cwl.expression.javascript.CWLExpressionJavascriptResolver.evaluate(CWLExpressionJavascriptResolver.java:89) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.javascriptInterpolate(CWLExpressionResolver.java:159) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.resolve(CWLExpressionResolver.java:66) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:232) ~[rabix-cli.jar:na]
... 11 common frames omitted
Caused by: org.mozilla.javascript.EcmaError: TypeError: Cannot read property "path" from null (script#1)
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3951) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3929) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3962) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3981) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.undefReadError(ScriptRuntime.java:3993) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1505) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1240) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) ~[rabix-cli.jar:na]
at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3282) ~[rabix-cli.jar:na]
at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Context.evaluateString(Context.java:1219) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.javascript.CWLExpressionJavascriptResolver.evaluate(CWLExpressionJavascriptResolver.java:69) ~[rabix-cli.jar:na]
... 14 common frames omitted
[2017-10-12 16:22:39.931] [ERROR] Failed to start 2e156a8a-0ef4-40d4-932b-dca94da9311b. org.rabix.executor.ExecutorException: Execution failed for 2e156a8a-0ef4-40d4-932b-dca94da9311b. Failed to start container.
org.rabix.executor.ExecutorException: Execution failed for 2e156a8a-0ef4-40d4-932b-dca94da9311b. Failed to start container.
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:198) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.command.StartCommand.run(StartCommand.java:32) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerCommand.run(JobHandlerCommand.java:51) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerRunnable.run(JobHandlerRunnable.java:60) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: org.rabix.executor.container.ContainerException: Failed to start container.
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:277) ~[rabix-cli.jar:na]
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:195) ~[rabix-cli.jar:na]
... 6 common frames omitted
Caused by: org.rabix.bindings.BindingException: org.rabix.bindings.cwl.expression.CWLExpressionException: Failed evaluating expression (inputs.genomic_file.path).
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:234) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineParts(CWLCommandLineBuilder.java:145) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineObject(CWLCommandLineBuilder.java:87) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLBindings.buildCommandLineObject(CWLBindings.java:98) ~[rabix-cli.jar:na]
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:217) ~[rabix-cli.jar:na]
... 7 common frames omitted
Caused by: org.rabix.bindings.cwl.expression.CWLExpressionException: Failed evaluating expression (inputs.genomic_file.path).
at org.rabix.bindings.cwl.expression.javascript.CWLExpressionJavascriptResolver.evaluate(CWLExpressionJavascriptResolver.java:89) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.javascriptInterpolate(CWLExpressionResolver.java:159) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.resolve(CWLExpressionResolver.java:66) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:232) ~[rabix-cli.jar:na]
... 11 common frames omitted
Caused by: org.mozilla.javascript.EcmaError: TypeError: Cannot read property "path" from null (script#1)
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3951) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3929) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3962) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3981) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.undefReadError(ScriptRuntime.java:3993) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1505) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1240) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) ~[rabix-cli.jar:na]
at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3282) ~[rabix-cli.jar:na]
at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Context.evaluateString(Context.java:1219) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.javascript.CWLExpressionJavascriptResolver.evaluate(CWLExpressionJavascriptResolver.java:69) ~[rabix-cli.jar:na]
... 14 common frames omitted
[2017-10-12 16:22:39.931] [ERROR] Failed to start 2e156a8a-0ef4-40d4-932b-dca94da9311b. org.rabix.executor.ExecutorException: Execution failed for 2e156a8a-0ef4-40d4-932b-dca94da9311b. Failed to start container.
org.rabix.executor.ExecutorException: Execution failed for 2e156a8a-0ef4-40d4-932b-dca94da9311b. Failed to start container.
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:198) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.command.StartCommand.run(StartCommand.java:32) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerCommand.run(JobHandlerCommand.java:51) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerRunnable.run(JobHandlerRunnable.java:60) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: org.rabix.executor.container.ContainerException: Failed to start container.
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:277) ~[rabix-cli.jar:na]
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:195) ~[rabix-cli.jar:na]
... 6 common frames omitted
Caused by: org.rabix.bindings.BindingException: org.rabix.bindings.cwl.expression.CWLExpressionException: Failed evaluating expression (inputs.genomic_file.path).
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:234) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineParts(CWLCommandLineBuilder.java:145) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineObject(CWLCommandLineBuilder.java:87) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLBindings.buildCommandLineObject(CWLBindings.java:98) ~[rabix-cli.jar:na]
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:217) ~[rabix-cli.jar:na]
... 7 common frames omitted
Caused by: org.rabix.bindings.cwl.expression.CWLExpressionException: Failed evaluating expression (inputs.genomic_file.path).
at org.rabix.bindings.cwl.expression.javascript.CWLExpressionJavascriptResolver.evaluate(CWLExpressionJavascriptResolver.java:89) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.javascriptInterpolate(CWLExpressionResolver.java:159) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.resolve(CWLExpressionResolver.java:66) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:232) ~[rabix-cli.jar:na]
... 11 common frames omitted
Caused by: org.mozilla.javascript.EcmaError: TypeError: Cannot read property "path" from null (script#1)
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3951) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3929) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3962) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3981) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.undefReadError(ScriptRuntime.java:3993) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1505) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1240) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) ~[rabix-cli.jar:na]
at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393) ~[rabix-cli.jar:na]
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3282) ~[rabix-cli.jar:na]
at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120) ~[rabix-cli.jar:na]
at org.mozilla.javascript.Context.evaluateString(Context.java:1219) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.javascript.CWLExpressionJavascriptResolver.evaluate(CWLExpressionJavascriptResolver.java:69) ~[rabix-cli.jar:na]
... 14 common frames omitted
[2017-10-12 16:22:39.933] [INFO] Failed to start 2e156a8a-0ef4-40d4-932b-dca94da9311b. org.rabix.executor.ExecutorException: Execution failed for 2e156a8a-0ef4-40d4-932b-dca94da9311b. Failed to start container.
[2017-10-12 16:22:40.025] [WARN] Job root.gp_runner, rootId: 3940f7c2-749f-47df-ae8d-6a2f61daee6d failed: Failed to start 2e156a8a-0ef4-40d4-932b-dca94da9311b. org.rabix.executor.ExecutorException: Execution failed for 2e156a8a-0ef4-40d4-932b-dca94da9311b. Failed to start container.
[2017-10-12 16:22:40.032] [WARN] Root job 3940f7c2-749f-47df-ae8d-6a2f61daee6d failed.
[2017-10-12 16:22:40.318] [INFO] Failed to execute a Job
```
Please let me know if you find something strange from the STDOUT.
Regards,
Vipin @bogdang : Thanks for posting about that backslash escape character bug/fix. I've run into that one too. Not sure you specified, but the workaround I've used there is to just not escape things, that seems to allow it to work with both cwltool and rabix. Glad that rabix and cwltool are getting more in sync! Hi Vipin,
You can do this by:
1. Pull the code and change to the `develop` branch for latest fixes
```shell
git clone https://github.com/rabix/bunny.git
cd bunny
git checkout develop
```
2. Compile using maven. Install maven if required.
```shell
mvn clean install -P all
```
3. Tarball with the compiled release will be available in `bunny/rabix-cli/target/rabix-cli-1.0.1-release.tar.gz`
Hope this helps
Best,
Bogdan Hi @bogdang,
Thank you! I saw this file from the issue #344 pointed by @mepstein. My questions is how to compile bunny from the source and create the final executable. Do you have documentation somewhere?
So far I have used the latest release version of the package available with the instruction from the page: https://github.com/rabix/bunny#installing
This is the directory structure I have for the release of `rabix` which I used for running the workflows.
```shell
tree rabix-cli-1.0.1/
rabix-cli-1.0.1/
??? bin
? ??? tes-quickstart
??? config
? ??? core.properties
? ??? ftp.properties
? ??? logback-verbose.xml
? ??? logback.xml
? ??? store.properties
??? examples
? ??? dna2protein
? ? ??? data
? ? ? ??? input.txt
? ? ??? dna2protein.cwl.json
? ? ??? inputs.json
? ? ??? README.md
? ? ??? transcribe.cwl.json
? ? ??? translate.cwl.json
? ??? README.md
??? lib
? ??? rabix-cli.jar
??? rabix
6 directories, 15 files
``` Hey Vipin,
Here is the [CWLExpressionResolver.java](https://github.com/rabix/bunny/blob/cd19f792868ec9d3712d7aa7da56ade0eebc53f7/rabix-bindings-cwl/src/main/java/org/rabix/bindings/cwl/expression/CWLExpressionResolver.java) Thanks @jaeddy for pointing to the right people!
Thank you @mepstein & @bogdang for explaining me the solution you found for the problem. Please let me know once you finish the documentation, so that I can follow your steps to resolve the issue.
@mepstein where will I see the script CWLExpressionResolver.java in `rabix` codebase? I have downloaded the version available here: https://github.com/rabix/bunny/releases/download/v1.0.1/rabix-1.0.1.tar.gz
Can you point me that ? Sorry if this is something very stupid.
Regards, Vipin
As Milt said, the dollarsign bug was fixed, and there is another bug that should be fixed: https://github.com/rabix/bunny/issues/352
I modified the CWL just a little bit to work around the second bug and got the workflow running all the way through with valid results(with rabix version with the patch for the bug that Milt explained). Once the fixes are deployed I'll make sure to document a submission both for rabix and the CancerGenomicsCloud platform. I ran into this myself, and happened to be in communication with the rabix developers, and they found and fixed a bug in rabix: https://github.com/rabix/bunny/pull/344. Don't think it's in any release yet -- maybe you can pull it from the repo, or patch it yourself somehow (it's just a two line change). It also might be possible to change the cwl to avoid this problem, not sure. (It does work with cwltool.) Not sure if this is a `rabix` issue or something specific to the `knoweng_gene_prioritization` workflow itself. I suspect @bogdang is well suited to help troubleshoot `rabix`, but I'll tag @mepstein here as well, in case he can spot any issues with the workflow.
Hello,
Here is an error when running the `knoweng_gene_prioritization` with `rabix`.
```shell
[nexcbu@wfexecutor knoweng_gene_prioritization]$ ~/tools/rabix-cli-1.0.0/rabix --tmpdir-prefix ~/tmp/ --tmp-outdir-prefix ~/tmp/ ./gp_workflow.cwl gp_workflow_job.yml
[2017-10-05 11:02:59.880] [INFO] Job root.kn_fetcher has started
[2017-10-05 11:02:59.882] [INFO] Job root.data_cleaning has started
[2017-10-05 11:02:59.914] [INFO] Pulling docker image quay.io/cblatti3/kn_fetcher:latest
[2017-10-05 11:03:01.134] [ERROR] Failed to start container.
org.rabix.bindings.BindingException: org.rabix.bindings.cwl.expression.CWLExpressionException: Substitution error, unfinished block starting at position 0 : ; else touch empty.edge; fi && date" && echo $MYCMD > run_fetch.cmd && eval $MYCMD
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:233) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineParts(CWLCommandLineBuilder.java:144) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineObject(CWLCommandLineBuilder.java:86) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLBindings.buildCommandLineObject(CWLBindings.java:98) ~[rabix-cli.jar:na]
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:203) ~[rabix-cli.jar:na]
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:171) [rabix-cli.jar:na]
at org.rabix.executor.execution.command.StartCommand.run(StartCommand.java:32) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerCommand.run(JobHandlerCommand.java:51) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerRunnable.run(JobHandlerRunnable.java:60) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: org.rabix.bindings.cwl.expression.CWLExpressionException: Substitution error, unfinished block starting at position 0 : ; else touch empty.edge; fi && date" && echo $MYCMD > run_fetch.cmd && eval $MYCMD
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.scanJavascriptExpression(CWLExpressionResolver.java:297) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.javascriptInterpolate(CWLExpressionResolver.java:175) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.resolve(CWLExpressionResolver.java:66) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:231) ~[rabix-cli.jar:na]
... 11 common frames omitted
[2017-10-05 11:03:01.135] [ERROR] Failed to start 72ab5c03-0dc9-47ec-a114-36603141b7d5. org.rabix.executor.ExecutorException: Execution failed for 72ab5c03-0dc9-47ec-a114-36603141b7d5. Failed to start container.
org.rabix.executor.ExecutorException: Execution failed for 72ab5c03-0dc9-47ec-a114-36603141b7d5. Failed to start container.
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:174) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.command.StartCommand.run(StartCommand.java:32) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerCommand.run(JobHandlerCommand.java:51) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerRunnable.run(JobHandlerRunnable.java:60) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: org.rabix.executor.container.ContainerException: Failed to start container.
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:268) ~[rabix-cli.jar:na]
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:171) ~[rabix-cli.jar:na]
... 6 common frames omitted
Caused by: org.rabix.bindings.BindingException: org.rabix.bindings.cwl.expression.CWLExpressionException: Substitution error, unfinished block starting at position 0 : ; else touch empty.edge; fi && date" && echo $MYCMD > run_fetch.cmd && eval $MYCMD
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:233) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineParts(CWLCommandLineBuilder.java:144) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineObject(CWLCommandLineBuilder.java:86) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLBindings.buildCommandLineObject(CWLBindings.java:98) ~[rabix-cli.jar:na]
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:203) ~[rabix-cli.jar:na]
... 7 common frames omitted
Caused by: org.rabix.bindings.cwl.expression.CWLExpressionException: Substitution error, unfinished block starting at position 0 : ; else touch empty.edge; fi && date" && echo $MYCMD > run_fetch.cmd && eval $MYCMD
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.scanJavascriptExpression(CWLExpressionResolver.java:297) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.javascriptInterpolate(CWLExpressionResolver.java:175) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.resolve(CWLExpressionResolver.java:66) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:231) ~[rabix-cli.jar:na]
... 11 common frames omitted
[2017-10-05 11:03:01.136] [ERROR] Failed to start 72ab5c03-0dc9-47ec-a114-36603141b7d5. org.rabix.executor.ExecutorException: Execution failed for 72ab5c03-0dc9-47ec-a114-36603141b7d5. Failed to start container.
org.rabix.executor.ExecutorException: Execution failed for 72ab5c03-0dc9-47ec-a114-36603141b7d5. Failed to start container.
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:174) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.command.StartCommand.run(StartCommand.java:32) ~[rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerCommand.run(JobHandlerCommand.java:51) [rabix-cli.jar:na]
at org.rabix.executor.execution.JobHandlerRunnable.run(JobHandlerRunnable.java:60) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: org.rabix.executor.container.ContainerException: Failed to start container.
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:268) ~[rabix-cli.jar:na]
at org.rabix.executor.handler.impl.JobHandlerImpl.start(JobHandlerImpl.java:171) ~[rabix-cli.jar:na]
... 6 common frames omitted
Caused by: org.rabix.bindings.BindingException: org.rabix.bindings.cwl.expression.CWLExpressionException: Substitution error, unfinished block starting at position 0 : ; else touch empty.edge; fi && date" && echo $MYCMD > run_fetch.cmd && eval $MYCMD
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:233) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineParts(CWLCommandLineBuilder.java:144) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLineObject(CWLCommandLineBuilder.java:86) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLBindings.buildCommandLineObject(CWLBindings.java:98) ~[rabix-cli.jar:na]
at org.rabix.executor.container.impl.DockerContainerHandler.start(DockerContainerHandler.java:203) ~[rabix-cli.jar:na]
... 7 common frames omitted
Caused by: org.rabix.bindings.cwl.expression.CWLExpressionException: Substitution error, unfinished block starting at position 0 : ; else touch empty.edge; fi && date" && echo $MYCMD > run_fetch.cmd && eval $MYCMD
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.scanJavascriptExpression(CWLExpressionResolver.java:297) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.javascriptInterpolate(CWLExpressionResolver.java:175) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.expression.CWLExpressionResolver.resolve(CWLExpressionResolver.java:66) ~[rabix-cli.jar:na]
at org.rabix.bindings.cwl.CWLCommandLineBuilder.buildCommandLinePart(CWLCommandLineBuilder.java:231) ~[rabix-cli.jar:na]
... 11 common frames omitted
[2017-10-05 11:03:01.137] [INFO] Failed to start 72ab5c03-0dc9-47ec-a114-36603141b7d5. org.rabix.executor.ExecutorException: Execution failed for 72ab5c03-0dc9-47ec-a114-36603141b7d5. Failed to start container.
[2017-10-05 11:03:01.183] [WARN] Job root.kn_fetcher, rootId: b82e2d2e-8724-4f91-946a-8428dde6cf51 failed: Failed to start 72ab5c03-0dc9-47ec-a114-36603141b7d5. org.rabix.executor.ExecutorException: Execution failed for 72ab5c03-0dc9-47ec-a114-36603141b7d5. Failed to start container.
[2017-10-05 11:03:01.191] [WARN] Root job b82e2d2e-8724-4f91-946a-8428dde6cf51 failed.
[2017-10-05 11:03:01.632] [INFO] Failed to execute a Job
```
I didn't see a submission with `rabix` and any idea what is going wrong here?
Thanks!
Vipin
Drop files to upload
knoweng_gene_prioritization workflow help page is loading…