diff options
| -rw-r--r-- | README.md | 122 | ||||
| -rw-r--r-- | doc/default.nix (renamed from doc/tools-doc.nix) | 0 | ||||
| -rw-r--r-- | examples/README.md | 25 |
3 files changed, 113 insertions, 34 deletions
@@ -1,40 +1,118 @@ <h1 align="center"> BioNix </h1> -BioNix is a tool for reproducible bioinformatics that unifies workflow engines, package managers, and containers. -It is implemented as a lightweight library on top of the [Nix](https://nixos.org/nix/) deployment system. +BioNix is a tool for reproducible bioinformatics that unifies workflow +engines, package managers, and containers. It is implemented as a +lightweight library on top of the [Nix](https://nixos.org/nix/) +deployment system. BioNix is currently a work in progress, so documentation is sparse. +Please get in contact with us for more information, help, and +contributing (see bottom of this page). -## Getting started - -Install [Nix](http://nixos.org/nix): +## Installation +BioNix requires no dependencies beyond [Nix](http://nixos.org/nix), +which may be installed by: ```{sh} curl https://nixos.org/nix/install | sh ``` -To run a sample pipeline, clone this project and run `nix-build` in the `/examples` directory: +If you do not have root access a variety of [rootless +install](https://nixos.wiki/wiki/Nix_Installation_Guide#Installing_without_root_permissions) +options are available. -```{sh} -$ git clone https://github.com/PapenfussLab/bionix -$ cd examples -$ nix-build -``` +API docs can be generated by executing `nix build` in the `doc` +directory and viewing `result/OEBPS/index.html`. + +## Examples + +Several examples are available in `./examples/`. The main example is +presented in `./examples/default.nix` and can be built using `nix build` +in `./examples/`. This sample pipeline performs variant calling using +[`platypus`](https://github.com/andyrimmer/Platypus), alignment using +[`bwa mem`](https://github.com/lh3/bwa), and preprocessing using +[`samtools`](http://www.htslib.org/). + +See the documentation in `./examples/README.md` for more detail about +this pipeline and the other examples. + +- The pipeline itself is specified in `examples/call.nix` and + `examples/default.nix`. +- The BioNix wrapper to run `platypus` is in + `tools/platypus-callVariants.nix`. +- The Nix expression for the `platypus` software itself can be found in + [nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/science/biology/platypus/default.nix). + +## Constructing workflows + +Writing workflows requires some familiarity with the Nix +programming language and deployment system. Good introductions can be +found [here](https://learnxinyminutes.com/docs/nix/) and +[here](https://ebzzry.io/en/nix/). + +To understand how to construct workflows it is recommended to study the +examples provided. Thanks to the flexibility of Nix, the workflows can +be constructed in different ways to suit the intended purposes and the +examples illustrate some of the ways one might approach various +problems. + +For constructing tool wrappers, take a look in the `./tools/` +directory for the currently existing tool wrappers. A good starting +point are the wrappers for BWA. + +## HPC execution + +BioNix supports submission of jobs to computing queues rather than +directly building them using the Nix build engine. The two supported +engines are Slurm and PBS represented by the `slurm` and `qsub` entries +in the root BioNix tree, which take an attribute set of default +parameters to a new tree of tools. Simply use tools out of these trees +to submit jobs, and specify resource requirements as ordinary +configuration options to the tools. + +The following resource parameters can be specified: + +- *ppn*: The number of cores to request; +- *mem*: The amount of memory to request (GB); +- *walltime*: A string defining the maximum walltime. + +As we rely on side effects to submit jobs sandbox builds cannot be used +and must be disabled (`--option sandbox false` with `nix-build` or +`--no-sandbox` with `nix build`). + +### Slurm specifics + +Slurm jobs are submitted by executing the `salloc` binary on the +cluster. By default this is assumed to be `/usr/bin/salloc`; if this is +not the case on your cluster then you need to additionally specify the +path to salloc via the `salloc` parameter. -The sample pipeline performs variant calling using [`platypus`](https://github.com/andyrimmer/Platypus), alignment using [`bwa mem`](https://github.com/lh3/bwa), and preprocessing using [`samtools`](http://www.htslib.org/). -BioNix will download or build all of the necessary software and create a soft link (`result`) to the workflow output. +When launching the build, it is important that the `TMPDIR` +environment variable points to a location which is on shared storage +(i.e., available from all nodes). This will be the location used for +temporary files during the execution of stages. -Next, check out the code: +### PBS specifics -- The pipeline itself is specified in `examples/call.nix` and `examples/default.nix`. -- The BioNix wrapper to run `platypus` is in `tools/platypus-callVariants.nix`. -- The Nix expression for the `platypus` software itself can be found in [nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/science/biology/platypus/default.nix). +The PBS wrapper is considerably more complicated as initiating +interactive processes is not as reliable as Slurm's `salloc`. +Consequently, jobs are submitted via non-interactive queue submissions +and the queue polled to determine when the submitted job has completed. -BioNix pipelines can be easily wrapped in shell scripts: see `examples/ex-tnpair/tnpair` for an example script that accepts a reference fasta, along with paired normal and tumor fastq files, and performs alignment, preprocessing, and variant calling with [`strelka`](https://github.com/Illumina/strelka). +The path to the PBS executables (i.e., `qsub` and `qstat`) has to be +given in the `qsubPath` attribute. Furthermore, a temporary directory +that's shared across all nodes must be specified in `tmpDir`. -Writing your own pipelines requires some familiarity with the Nix programming language and deployment system. Good introductions can be found [here](https://learnxinyminutes.com/docs/nix/) and [here](https://ebzzry.io/en/nix/). +## Distributed execution -We have successfully run BioNix pipelines in a zero-install manner (using a [statically linked binary](https://matthewbauer.us/blog/static-nix.html) and [user namespaces](https://www.redhat.com/en/blog/whats-next-containers-user-namespaces)), but this feature is currently unstable. Stay tuned! +Nix has support for distributing jobs amongst a collection of +distributed machines. See the +[manual](https://nixos.org/nix/manual/#chap-distributed-builds) and +[wiki](https://nixos.wiki/wiki/Distributed_build) for more information. -## Contact +## Getting help and contributing -Please come chat with us at [#bionix:cua0.org](http://matrix.to/#/#bionix:cua0.org). +For general questions, issues, and +[contributing](https://git-send-email.io), please +[email](mailto:bionix@cua0.org) or [subscribe +to](mailto:bionix+subscribe@cua0.org) our mailing list. You may also +chat with us at [#bionix:cua0.org](http://matrix.to/#/#bionix:cua0.org). diff --git a/doc/tools-doc.nix b/doc/default.nix index d68a6b1..d68a6b1 100644 --- a/doc/tools-doc.nix +++ b/doc/default.nix diff --git a/examples/README.md b/examples/README.md index 66ed04f..1689709 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,17 +1,14 @@ # Bionix examples -This directory has a few example workflows in bionix along with example data. -A basic workflow is defined in `call.nix`, and an example of applying it to the -sample data is in `default.nix`. To build the `default.nix` workflow, run -``` -nix build -``` -from this directory. +This directory has a few example workflows in bionix along with example +data. A basic workflow is defined in `call.nix`, and an example of +applying it to the sample data is in `default.nix`. To build the +`default.nix` workflow, run ```nix build``` from this directory. ## NextFlow and WDL translations -The directories `ex-nextflow` and `ex-wdl` contain translated examples from the -NextFlow and WDL documentation respectively. +The directories `ex-nextflow` and `ex-wdl` contain translated examples +from the NextFlow and WDL documentation respectively. The NextFlow translated example does not come with example data. It can be built with ``` @@ -25,6 +22,10 @@ nix build -f wdl-scatter-gather.nix ## Example script wrapper -`ex-tnpair` contains a shell script based example on how a front-end for users -might be constructed. It is a simple tumour-normal somatic calling workflow -using the Strelka variant caller. +`ex-tnpair` contains a shell script based example on how a front-end for +users might be constructed. It is a simple tumour-normal somatic calling +workflow using the Strelka variant caller. The script accepts a +reference fasta along with paired normal and tumor fastq files and +performs alignment, preprocessing, and variant calling with +[`strelka`](https://github.com/Illumina/strelka). + |