SeqtrimNEXT

DESCRIPTION:

SeqtrimNEXT is a customizable and distributed pre-processing software for NGS (Next Generation Sequencing) biological data. It makes use of scbi_mapreduce gem to be able to run in parallel and distributed environments. It is specially suited for Roche 454 (normal and paired-end) & Ilumina datasets, although it could be easyly adapted to any other situation.

FEATURES:

• SeqtrimNEXT is very flexible since it’s architecture is based on plugins.

• You can add new plugins if needed.

• SeqtrimNEXT uses scbi_mapreduce and thus is able to exploit all the benefits of a cluster environment. It also works in multi-core machines big shared-memory servers.

Default templates for genomics & transcriptomics are provided

You can define your own templates using a combination of available plugins:

SYNOPSIS:

Once installed, SeqtrimNEXT is very easy to use:

To install core databases (it should be done at installation time):

$> seqtrimnext -i core

Databases will be installed nearby SeqtrimNEXT by default, but you can override this location by setting the environment variable BASTDB. Eg.:

If you with your database installed at /var:

$> export BLASTDB=/var/DB/formatted

Be sure that this environment variable is always loaded before SeqtrimNEXT execution (Eg.: add it to /etc/profile.local).

To perform an analisys using a predefined template with a FASTQ file format using 4 cpus:

$> seqtrimnext -t genomics_454.txt -Q input_file_in_FASTQ -w 4

To perform an analisys using a predefined template with a FASTQ file format:

$> seqtrimnext -t genomics_454.txt -f input_file_in_FASTA -q input_file_in_QUAL

To get additional help and list available templates and databases:

$> seqtrimnext -h

CLUSTERED EXECUTION:

To take full advantage of a clustered installation, you can launch SeqtrimNEXT in distributed mode. You only need to provide it a list of machine names (or IPs) where workers will be launched.

Setup a workers file like this:

machine1
machine1
machine2
machine2
machine2

And launch SeqtrimNEXT this way:

$> seqtrimnext -t genomics_454.txt -Q input_file_in_FASTQ -w workers_file -s 10.0.0

This will launch 2 workers on machine1 and 3 workers on machine2 using the network whose ip starts with 10.0.0 to communicate.

TEMPLATE MODIFICATIONS

You can modify any template to fit your workflow. To do this, you only need to copy one of the templates and edit it with a text editor, or simply modify a used_params.txt file that was produced by a previous SeqtrimNEXT execution.

Eg.: If you want to disable repetition removal, do this:

1-Copy the template file you wish to customize and name it params.txt. 2-Edit params.txt with a text editor 3-Find a line like this:

remove_clonality = true

4-Replace this line with:

remove_clonality = false

5- Launch SeqtrimNEXT with params.txt file instead of a default template:

$> seqtrimnext -t params.txt -f input_file_in_FASTA -q input_file_in_QUAL

The same way you can modify any of the parameters. You can find all parameters and their description in any used_params.txt file generated by a previous SeqtrimNEXT execution. Parameters not especified in a template are automatically set to their default value at execution time.

NOTE: The only mandatory parameter is the plugin_list one.

REQUIREMENTS:

• Ruby 1.9.2

  CD-HIT 4.5.3 or greater

• Blast plus 2.24 or greater (prior versions have bugs that produces bad results)

• Optional

  • GnuPlot version 4.4.2 or greater (prior versions may produce wrong graphs)

• Optional

  • pdflatex - Optional, to produce a detailed report with results

INSTALL:

Installing CD-HIT

*Download the latest version from code.google.com/p/cdhit/downloads/list *You can also use a precompiled version if you like *To install from source, decompress the downloaded file, cd to the decompressed folder, and issue the following commands:

make
sudo make install

Installing Blast

*Download the latest version of Blast+ from ftp.ncbi.nlm.nih.gov/blast/executables/release/LATEST/ *You can also use a precompiled version if you like *To install from source, decompress the downloaded file, cd to the decompressed folder, and issue the following commands:

./configure
make
sudo make install

Installing Ruby 1.9

*You can use RVM to install ruby:

Download latest certificates (maybe you don’t need them):

$ curl -O http://curl.haxx.se/ca/cacert.pem 
$ export CURL_CA_BUNDLE=`pwd`/cacert.pem # add this to your .bashrc or

equivalent

Install RVM:

$ bash < <(curl -k https://rvm.beginrescueend.com/install/rvm)

Setup environment:

$ echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bash_profile

Install ruby 1.9.2 (this can take a while):

$ rvm install 1.9.2

Set it as the default:

$ rvm use 1.9.2 --default

Install SeqtrimNEXT

SeqtrimNEXT is very easy to install. It is distributed as a ruby gem:

gem install seqtrimnext

This will install seqtrimnext and all the required gems.

Install and rebuild SeqtrimNext’s core databases

SeqtrimNEXT needs some core databases to work. To install them:

seqtrimnext -i core

You can change default database location by setting the environment variable BLASTDB. Refer to SYNOPSIS for an example.

There are aditional databases, to list them:

seqtrimnext -i LIST

Database modifications

Included databases will be usefull for a lot of people, but if you prefer, you can modify them, or add more elements to be search against your sequences.

You only need to drop new fasta files to each respective directory:

DB/vectors to add more vectors DB/contaminants to add more contaminants etc…

Once the databases has been modified, you will need to reformat them by issuing the following command:

seqtrimnext -c

Modified databases will be rebuilt.

CLUSTERED INSTALLATION

To install SeqtrimNEXT into a cluster, you need to have the software available on all machines. By installing it on a shared location, or installing it on each cluster node. Once installed, you need to create a init_file where your environment is correctly setup (paths, BLASTDB, etc):

export PATH=/apps/blast+/bin:/apps/cd-hit/bin
export BLASTDB=/var/DB/formatted
export SEQTRIMNEXT_INIT=path_to_init_file

And initialize the SEQTRIMNEXT_INIT environment variable on your main node (from where SeqtrimNEXT will be initially launched):

export SEQTRIMNEXT_INIT=path_to_init_file

If you use any queue system like PBS Pro or Moab/Slurm, be sure to initialize the variables on each submission script.

NOTE: all nodes on the cluster should use ssh keys to allow SeqtrimNEXT to launch workers without asking for a password.

SAMPLE INIT FILES FOR CLUSTERED INSTALLATION:

Init file

$> cat stn_init_env 
source ~latex/init_env
source ~ruby19/init_env
source ~blast_plus/init_env
source ~gnuplot/init_env
source ~cdhit/init_env
export BLASTDB=~seqtrimnext/DB/formatted/
export SEQTRIMNEXT_INIT=~seqtrimnext/stn_init_env

PBS Submission script

$> cat sample_work.sh 
# 40 distributed workers and 1 GB memory per worker:
#PBS -l select=40:ncpus=1:mpiprocs=1:mem=1gb
# request 10 hours of walltime:
#PBS -l walltime=10:00:00
# cd to working directory (from where job was submitted)
cd $PBS_O_WORKDIR
# create workers file with assigned node names
cat ${PBS_NODEFILE} > workers
# init seqtrimnext
source ~seqtrimnext/init_env
time seqtrimnext -t paired_ends.txt -Q fastq -w workers -s 10.0.0

Once this submission script is created, you only need to launch it with:

qsub sample_work.sh

MOAB/SLURM submission script

$> cat sample_work_moab.sh
#!/bin/bash 
# @ job_name = STN
# @ initialdir = .
# @ output = STN_%j.out
# @ error = STN_%j.err
# @ total_tasks = 40
# @ wall_clock_limit = 10:00:00
# guardar lista de workers
sl_get_machine_list > workers
# init seqtrimnext
source ~seqtrimnext/init_env
time seqtrimnext -t paired_ends.txt -Q fastq -w workers -s 10.0.0

Then you only need to submit your job with mnsubmit

mnsubmit sample_work_moab.sh