This is a question that has come up a few times, so I thought I'd share here and provide some potential answers. Some workflows produce outputs with either file- or folder names that are dynamic and/or not necessarily known up front (the most obvious example is some sort of time- or date-stamped object name). An example of this can be seen with the [**encode_mapping_workflow**](https://www.synapse.org/#!Synapse:syn10163025), where outputs are saved to a date-stamped folder (e.g., `7-26-2017T2H43M10S/`). So, how does one handle the parameterization of the corresponding checker tool for this workflow?
The most straightforward solution is to manually edit the folder name in the JSON/YAML for both the checker and submit tools ? or instruct participants to do so. This is the approach taken for the ENCODE workflow (you can see examples in the link above):
*"I first edited the file `validator.json` with the path to the outputs I want to check:"* ~ J. Seth Stratton
> **Note:** the `path` value here should be the relative or absolute path to the output directory your workflow run generated that contains the files `filter_qc.json`, `mapping.json`, `post_mapping.json`, and `xcor.json`. Where these outputs are stored will depend on the workflow runner you use. This example uses `cwltool` which saves outputs to a date-stamped folder. Do not include a trailing slash in the folder name.
This is reasonable enough for individual participants running a workflow and doesn't require much additional effort. However, it wouldn't be feasible for me to edit JSON files for each submission file before running the checker tool in the challenge backend environment. Instead, I use a little hack to "rewrite" the JSON file such that the path to the output folder is correct (this assumes that each submission is downloaded within a separate folder ? which is the standard behavior for fetching submissions through Synapse).
```python
# check whether checker cwl and param file are present
checkerPath = os.path.join(checkerDir, annotations['workflow'] + '_checker.cwl')
origCheckerParamPath = checkerPath + config['param_ext']
...
# link checker param file to submission folder
newCheckerParamPath = os.path.join(submissionDir, config['handle'] + '_checker.cwl' + config['param_ext'])
if not os.path.exists(newCheckerParamPath):
if config['handle'] == 'encode_mapping_workflow':
shutil.copy(origCheckerParamPath, newCheckerParamPath)
sed_command = ['sed', '-i', '-e', 's|path.*\"\"|path\": \"{}\"|g'.format(submissionDir), newCheckerParamPath]
subprocess.call(sed_command)
else:
os.symlink(origCheckerParamPath, newCheckerParamPath)
```
If the path doesn't need to be updated, then I can just symlink the original JSON file to the working directory for the submission (such that all relative paths are preserved).
This might not be the most graceful or sophisticated solution, but it seems to do the job. I'm curious if other people have ideas for how to do this differently.
Cheers,
James