This thread is the designated place to ask questions about running the **bcbio_NA12878-chr20** workflow.

Created by James Eddy jaeddy
Hi @mkrause ? following up on the earlier issue about teams, you can disregard my previous comment. I figured out a workaround that takes care of the problem reasonably well. See this thread for more info: https://www.synapse.org/#!Synapse:syn8507133/discussion/threadId=2967   Thanks!
Hi @mkrause ? turns out I was overly optimistic about my ability to fix the submission. Apparently, `teamId` can only be set at the time of submission, and the REST API does not have an option to update that information later.   Instead, I would need to delete the current submission and have you re-submit syn11382097 to the evaluation queue. This can be done through the Synapse browser by navigating to the entity page for syn11382097, clicking on 'Tools' -> 'Submit File to Challenge', and selecting **GA4GH-DREAM_bcbio_NA12878-chr20*. After clicking 'Next', you'll have the option to submit as part of a team.   Making a note of the original submission report here ? syn11382141 ? so that it should be easy to copy the documentation over for a new submission.   If that seems reasonable, let me know and I'll delete submission *9650685* (want to verify before creating any unwanted extra work).
Benjamin; So sorry about missing this earlier and thank you James for tagging me. If you ever need future help and I miss your post feel free to tag me or raise an issue on the bcbio GitHub (https://github.com/chapmanb/bcbio-nextgen). I think the underlying issue with your run is that you have incompatible versions of the bcbio biodata and CWL descriptions. As we've worked to port this workflow to multiple systems, we ended up using tarballs for some of the complex directories like snpEff. This resulted in an update of both the data and CWL but I think it's possible you have the most recent data but an older CWL. I believe if you update both to the latest and re-run it should hopefully work cleanly. Sorry about the issue and the changes and I appreciate you testing this out. Brad
Hi James, Thanks for the explanation and suggestion, that makes sense. Yes, if it's not too much to ask please go ahead and associate my first bcbio submission to UCSC GA4GH team. Unfortunately, I forgot to include that in the submission file. After that I will try to resubmit my second bcbio workflow. Thanks, Michael
Hi @benstory,   Sorry I missed your question from a couple months ago! I'm going to cc @chapmanb here and see if he can help troubleshoot. I'll try to take a closer later and offer any advice that might be relevant from my own experience running the workflow.   -James
Hi @mkrause,   I think I figured out what's going on. The problem was less obvious to track down than it would be in most challenges, because we have so many evaluation queues: for any given queue, participants can only submit with a single team. Because you already submitted outputs from Toil (submission `9650685`) as an individual, this means you're ineligible to submit as a member of @UCSCGA4GHDREAMChallengeTeam (at least *for this workflow*).   You should still be able to submit as an individual for the bcbio workflow; or if you would like, I can try to associate your existing submission with the UCSC team.   Best, James
Hi, My bcbio workflow has completed without problems ("Overall": true,). But when I submit using `cwltool --non-strict dockstore-tool-synapse-submit.cwl bcbio_NA12878-chr20_submit.json` I get the message shown in this image: [screenshot](https://drive.google.com/file/d/149o-HCXlDJ9vBYd7rjEnEn-eFkkHCz6A/view?usp=sharing). Just in case the link to my screenshot for some reason can't be accessed: the message shows this output: ```bash SynapseHTTPError: 403 Client Error: Forbidden The following Team members are ineligible to contribute to a Submission in the specified Evaluation at this time [3363440] ``` This although it says further above that the "...submission completed to success", and "...the process status is success". The problem is that my workflow does not show up in the section of submitted workflows, so I'm stuck (this is not the same as the one the validated section, this one I'm talking about here I ran on my local). Anyone? Thanks, Michael
I've also had some problems completing this workflow. My first attempts were using cwltool version 1.0.20170822192924 and it seems to have problems in the samtools index steps. I then went back and tried the version listed in the workflow 1.0.20170217172322 and was successful. I'm left wondering, for the challenge, the fact that a later version of cwltool didn't work seems like an important data point. How does this get tracked?
This is the only challenge I haven't been able to complete. Here is the end of the error that I get: ``` [job postprocess_variants] Job error: Error validating output record, the `vrn_file` field is not valid because is not a dict in { "vrn_file": [] } [job postprocess_variants] completed permanentFail [step postprocess_variants] Output is missing expected field file:///tmpdata/story/bcbionew/NA12878-platinum-chr20-workflow/wf-variantcall.cwl#postprocess_variants/vrn_file [step postprocess_variants] completed permanentFail [workflow variantcall] outdir is /tmpdata/story/bcbionew/tmp/dcDZ_y Workflow cannot make any more progress. Final process status is permanentFail ``` In the most recently created tmpfile in /tmp/qNcptJ/log (sub folders in the main directory where I am running everything) this is the contents of the debug file (bcbio-nextgen-debug.log): ``` [2017-09-04T11:47Z] 458290e35a63: Finalizing variant calls: NA12878-chr20, platypus [2017-09-04T11:47Z] 458290e35a63: Calculating variation effects for NA12878-chr20, platypus [2017-09-04T11:47Z] 458290e35a63: Uncaught exception occurred Traceback (most recent call last): File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 56, in process out = fn(fnargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 50, in wrapper return apply(f, *args, **kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 178, in postprocess_variants return variation.postprocess_variants(*args) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/variation.py", line 68, in postprocess_variants ann_vrn_file, vrn_stats = effects.add_to_vcf(data[vrn_key], data) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/effects.py", line 31, in add_to_vcf ann_vrn_file, stats_files = snpeff_effects(in_file, data) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/effects.py", line 244, in snpeff_effects return _run_snpeff(vcf_in, "vcf", data) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/effects.py", line 340, in _run_snpeff "Did not find %s snpEff genome data in %s" % (snpeff_db, datadir) AssertionError: Did not find GRCh38_86 snpEff genome data in /var/spool/cwl/wf-inputs/snpeff/GRCh38.86 ``` Here are the commands I've used so far (pretty standard relative to all the other pipelines) because it wasn't working I started a brand new full attempt: ``` ## local whole server install of bcbio-nextgen-vm conda install -c conda-forge -c bioconda bcbio-nextgen-vm ## copied my synapse config cp ../testfolder/.synapseConfig ./ ### get the data and stuff synapse get syn9770802 # dockstore-tool-synapse-get.cwl synapse get syn9819444 # bcbio_NA12878-chr20_get.cwl.json cwltool --tmpdir-prefix tmp/ --tmp-outdir-prefix tmp/ --non-strict dockstore-tool-synapse-get.cwl bcbio_NA12878-chr20_get.cwl.json ## run the pipeline cwltool --tmpdir-prefix tmp/ --tmp-outdir-prefix tmp/ NA12878-platinum-chr20-workflow/main-NA12878-platinum-chr20.cwl NA12878-platinum-chr20-workflow/main-NA12878-platinum-chr20-samples.json ``` This file exists: bcbionew/NA12878-platinum-chr20-workflow/wf-variantcall.cwl This file exists: bcbionew/NA12878-platinum-chr20-workflow/postprocess_variants.cwl This folder exists: bcbionew/tmp/qNcptJ/wf-inputs/snpeff/GRCh38.86 ``` cwltool 1.0.20170413194156 Docker version 1.7.1, build 786b29d/1.7.1 ``` Any ideas?

bcbio_NA12878-chr20 workflow help page is loading…