Next Generation Sequencing and Data Analysis

Friday, March 6, 2015

Download from BaseSpace using R

#under shell
sudo apt-get update
sudo apt-get install libcurl4-gnutls-dev
sudo R

#under R
source('http://bioconductor.org/biocLite.R')
biocLite('RCurl')
biocLite('BaseSpaceR')
quit()


#Start R again
library(BaseSpaceR)
ACCESS_TOKEN<- ''
PROJECT_ID<- '3289289'
aAuth<- AppAuth(access_token = ACCESS_TOKEN)
selProj <- Projects(aAuth, id = PROJECT_ID, simplify = TRUE) 
sampl <- listSamples(selProj, limit= 1000)
inSample <- Samples(aAuth, id = Id(sampl), simplify = TRUE)
for(s in inSample)
{
    f <- listFiles(s, Extensions = ".gz")
    print(Name(f))
    getFiles(aAuth, id= Id(f), destDir = 'fastq/', verbose = TRUE)
}

Reference: http://seqanswers.com/forums/showthread.php?t=47633

Friday, February 20, 2015

Install bcl2fastq-1.8.4 and bcl2fastq 2.15.0 to Ubuntu 14.04 Server 64 Bit

Goal:

build a streamlined workflow from Sequencer to Analysis results directly, aka "one stop solution".

After a user starts an experiment by pushing a button on the touch screen on the sequencer, all the downstream analysis are automatically done.

The outputs from MiSeq will be transfered to another Linux machine, from where the intensities files will be converted into FASTQ files. After that, depending on the type of this experiment, an appropriate pipeline is launched on a cluster computer to processing these data. At last a report is generated and uploaded to a web server for the user to view. Everything was done automatically without any manual operations. No bioinformaticians or computer scientists involved in this process.

Here I list the steps for the installation of "bcl2fastq" on the Linux machine.

#system
Ubuntu 14.04 Server 64 bit

#package
bcl2fastq-1.8.4

#dependency
sudo apt-get install alien dpkg-dev debhelper build-essential xsltproc gnuplot -y

#make a tmp folder
mkdir -p ~/tmp ; cd ~/tmp

#download the bcl2fastq RPM package from illumina.
#The tar ball source code failed to compile on my system

wget ftp://webdata:webdata@ussd-ftp.illumina.com/Downloads/Software/bcl2fastq/bcl2fastq-1.8.4-Linux-x86_64.rpm

#http://seqanswers.com/forums/showthread.php?t=45649
sudo alien -i bcl2fastq-1.8.4-Linux-x86_64.rpm
curl -kL http://install.perlbrew.pl | bash
echo >> ~/.bash_profile "source ~/perl5/perlbrew/etc/bashrc"
perlbrew install perl-5.14.4
perlbrew switch perl-5.14.4
perlbrew install-cpanm

#install expat-2.1.0
wget http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fexpat%2Ffiles%2Fexpat%2F2.1.0%2F&ts=1424461084&use_mirror=softlayer-dal
tar -zxvf expat-2.1.0.tar.gz
cd expat-2.1.0 && ./configure && make && sudo make install

#install XML-Parser-2.41
wget http://pkgs.fedoraproject.org/repo/pkgs/perl-XML-Parser/XML-Parser-2.41.tar.gz/c320d2ffa459e6cdc6f9f59c1185855e/XML-Parser-2.41.tar.gz
cd XML-Parser-2.41 && perl Makefile.PL && make && sudo make install

#install XML module
cpanm XML/Simple.pm

#exit
exit

#the installation is done. Now make a test run.

#To run bcl2fastq we have to switch to a less-strict PERL environment
perlbrew switch perl-5.14.4

#assume "test/Data/Intensities/BaseCalls" is the output folder from your sequecning machine
/usr/local/bin/configureBclToFastq.pl --input-dir test/Data/Intensities/BaseCalls --output-dir output

#change to new output folder "output" and start the "from bcl to fastq" conversion
cd output && make -j $(grep -c ^processor /proc/cpuinfo)

#if you find "INFO: all completed successfully" in the last line of output then the test passed
#check out the output fastq files, here the "000000000-ABCDEF" is the FCID (Flow Cell ID)
ls -al Project_000000000-ABCDEF/Sample_lane1/

##################################
UPDATE!!!

interestingly, illumina claimed
"""
Use the bcl2fastq 2.15.0 conversion software to convert NextSeq 500 or HiSeq X output.
Version 2.15.0 is only for use with NextSeq and HiSeq X data.
Use bcl2fastq 1.8.4 for MiSeq and HiSeq data conversion.
The software is available for download in either an rpm or tarball (.tar.gz) format.
"""
http://support.illumina.com/downloads.html

However I found after the MiSeq's outputs were uploaded into BaseSpace, it actually
converted by bcl2fastq 2.15.0.
One key difference is that the "SampleSheet.csv" has different formats for "bcl2fastq 1.8.4" and "bcl2fastq 2.15.0".
The "SampleSheet.csv" generated by MiSeq match the format for "bcl2fastq 2.15.0".

It is a breeze to install bcl2fastq2-v2.15.0 on Ubuntu from source code.
It is a mission impossible (amlost, I spent a whole day trying different methods then gave up)to do so with bcl2fastq-1.8.4 from source code

wget ftp://webdata2:webdata2@ussd-ftp.illumina.com/downloads/Software/bcl2fastq/bcl2fastq2-v2.15.0.4.tar.gz
tar -zxvf bcl2fastq2-v2.15.0.4.tar.gz
cd bcl2fastq && mkdir build && cd build
../src/configure --prefix=/home/hadoop/tool/bcl2fastq && make && make install

#now make a test. Assuming "exp123" is our input folder
#firstly all the ".bcl" files MUST be gzipped otherwise the job will fail (need improvements here, illumina!)

find exp123 -name "*.bcl" -exec gzip {} \;

#pull the trigger
/home/hadoop/tool/bcl2fastq/bin/bcl2fastq -R exp123 -o exp123_fastq

Nice! Sorry I take back my complaining on illumina's software engineering team this morning after frustration on installing and running bcl2fastq 1.8.4

Good job, illumina team on bcl2fastq 2.15.0. You have my love again.

Monday, October 6, 2014

Install oozie on ubuntu 14.04 with hadoop-2.5.0

Assuming we use /home/hadoop/tools/ as the installation path for oozie-4.0.1 with hadoop-2.5.0 installed on /home/hadoop/tools/hadoop-2.5.0.

1. download & build


cd /home/hadoop/tools/



curl http://apache.cs.utah.edu/oozie/4.0.1/oozie-4.0.1.tar.gz | tar -zx



cd /home/hadoop/tools/oozie-4.0.1




bin/mkdistro.sh -DskipTests -Dhadoopversion=2.5.0  -DjavaVersion=1.7 -DtargetJavaVersion=1.7

2. make a new folder for the build


cp -R /home/hadoop/tools/oozie-4.0.1/distro/target/oozie-4.0.1-distro/oozie-4.0.1 /home/hadoop/tools/oozie



cd /home/hadoop/tools/oozie/bin



echo "PATH=$PWD:\$PATH" >> ~/.bash_profile



source ~/.bash_profile



#now "/home/hadoop/tools/oozie" is the home path for oozie

3. prepare for oozie web console





cd /home/hadoop/tools/oozie/



mkdir libext



cp /home/hadoop/tools/oozie-4.0.1/hadooplibs/target/oozie-4.0.1-hadooplibs.tar.gz .



tar xzvf oozie-4.0.1-hadooplibs.tar.gz



cp oozie-4.0.1/hadooplibs/hadooplib-2.3.0.oozie-4.0.1/* libext/



cd libext



wget http://extjs.com/deploy/ext-2.2.zip



rm -fr /home/hadoop/tools/oozie/oozie-4.0.1  /home/hadoop/tools/oozie/oozie-4.0.1-hadooplibs.tar.gz

4. add these two lines to "core-site.xml" under "/home/hadoop/tools/hadoop-2.5.0/etc/hadoop"




property name "hadoop.proxyuser.hadoop.hosts", property value "*"
property name "hadoop.proxyuser.hadoop.groups, property value "*"

5. Start oozie





oozie-setup.shoozie-setup.sh prepare-war



#oozie-setup.sh sharelib create -fs hdfs://localhost:8020



oozie-setup.sh db create –run



oozied.sh start



oozie admin –oozie http://localhost:11000 -status

6. Run an example


cd /home/hadoop/tools/oozie/


tar -zxvf oozie-examples.tar.gz



find examples/ -name "job.properties" -exec sed -i "s/localhost/master/g" '{}' \;



hdfs dfs -rm -f -r /user/hadoop/examples 



hdfs dfs -put examples examples



oozie job -oozie http://master:11000/oozie -config examples/apps/map-reduce/job.properties -run





#now open a web browser and access "http://master:11000/oozie", you will see the submitted job.

#reference:
http://www.quora.com/Has-anyone-tried-installing-oozie-4-0-0-with-apache-hadoop-2-2-0-and-java1-7-0_45-on-ubuntu

Monday, September 22, 2014

Learn Spark with Python

1. Install Spark


cd ~/tools/




wget http://d3kbcqa49mib13.cloudfront.net/spark-1.1.0.tgz



tar -zxvf spark-1.1.0.tgz

2. Build spark for hadoop2

cd ~/tools/spark-1.1.0

SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly

3. Install py4j

sudo pip install py4j

4. Modify ~/.bash_profile by adding two lines


export SPARK_HOME=$HOME/tools/spark-1.1.0



export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH

5. source the ~/.bash_profile


source ~/.bash_profile

6. Test. Start a python shell and type

import pyspark

Wednesday, August 27, 2014

AWS AMI for Hadoop DataNode

##Create an AWS AMI as the snapshot for hadoop datanode
##This image should contain all necessary tools, packages and libraries
##to be used by the pipeline and your hadoop-application

##assume the we already started a Ubuntu14.04-64-PV instance
##Public IP for the instance is 12.34.56.78 and our private key file was saved as
##/home/hadoop/.ssh/aws.pem

AWS_IP=12.34.56.78
KEYFILE=/home/hadoop/.ssh/aws.pem
USER=ubuntu
########################
#step1. log in our AWS instance
########################
ssh -i ~/.ssh/aws.pem ubuntu@${AWS_IP}

########################
#step2. install basic packages
########################
sudo apt-get install openjdk-7-jdk -y
sudo apt-get install make -y
sudo apt-get install cmake -y
sudo apt-get install gcc -y
sudo apt-get install g++ -y
sudo apt-get install zlib1g-dev -y
sudo apt-get install unzip -y
sudo apt-get install libncurses5-dev -y
sudo apt-get install r-base -y
sudo apt-get install python-dev -y
sudo apt-get install python-dateutil -y
sudo apt-get install python-psutil -y
sudo apt-get install python-pip -y
sudo apt-get install maven2 -y
sudo apt-get install libxml2-dev -y
sudo apt-get install gradle -y

#install R packages
sudo R
source('http://www.bioconductor.org/biocLite.R')
biocLite('edgeR')
biocLite('DESeq')
biocLite('limma')
#... all other necessary packages

########################
#step3. create folder structure
########################
TOOLS_HOME=~/tools
BIO_HOME=~/bioinformatics
APP=$BIO_HOME/app
DATA=$BIO_HOME/data

mkdir -p ${TOOLS_HOME}
mkdir -p ${APP}
mkdir -p ${DATA}

########################
#step4. install hadoop under ~/tools/
########################
cd $TOOLS_HOME
wget http://apache.cs.utah.edu/hadoop/common/hadoop-2.5.0/hadoop-2.5.0.tar.gz
tar -zxvf hadoop-2.4.0.tar.gz && rm -f hadoop-2.5.0.tar.gz && cd

#install s3tools under ~/tools/
cd $TOOLS_HOME
wget https://github.com/s3tools/s3cmd/archive/master.zip
unzip master.zip

########################
#step5. install bioinformatics applications
########################

#BWA
cd $APP
wget http://downloads.sourceforge.net/project/bio-bwa/bwa-0.7.9a.tar.bz2
tar -jxvf bwa-0.7.9a.tar.bz2 && rm -fr bwa-0.7.9a.tar.bz2
cd bwa-0.7.9a && make && cd $APP
mkdir -p $APP/bwa/0.7.9a
find bwa-0.7.9a -executable -type f -print0 | xargs -0 -I {} mv {} $APP/bwa/0.7.9a/
rm -fr bwa-0.7.9a

#bowtie1
cd ${APP}
wget http://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.0.1/bowtie-1.0.1-linux-x86_64.zip
unzip bowtie-1.0.1-linux-x86_64.zip && rm bowtie-1.0.1-linux-x86_64.zip
mkdir -p ${APP}/bowtie/1.0.1 && mv bowtie-1.0.1/* ${APP}/bowtie/1.0.1/ && rm -fr bowtie-1.0.1/

#bowtie2
cd ${APP}
wget http://downloads.sourceforge.net/project/bowtie-bio/bowtie2/2.2.3/bowtie2-2.2.3-linux-x86_64.zip
unzip bowtie2-2.2.3-linux-x86_64.zip && rm bowtie2-2.2.3-linux-x86_64.zip
mkdir -p ${APP}/bowtie/2.2.3 && mv bowtie2-2.2.3/* ${APP}/bowtie/2.2.3/ && rm -fr bowtie2-2.2.3/

#SNAP
cd ${APP}
curl http://snap.cs.berkeley.edu/downloads/snap-1.0beta.10-linux.tar.gz | tar xvz
mkdir -p ${APP}/snap/1.0beta.10/
mv snap-1.0beta.10-linux/* ${APP}/snap/1.0beta.10/ && rm -fr snap-1.0beta.10-linux

#GSNAP
cd ${APP}
curl http://research-pub.gene.com/gmap/src/gmap-gsnap-2014-06-10.tar.gz | tar xvz
cd gmap-2014-06-10 && ./configure --prefix=${APP}/gmap/2014-06-10/ && make && make install
rm -fr gmap-2014-06-10

#STAR
cd ${APP}
curl https://rna-star.googlecode.com/files/STAR_2.3.0e.Linux_x86_64.tgz | tar xvz
mkdir -p ${APP}/star/2.3.0e/
mv STAR_2.3.0e.Linux_x86_64/* ${APP}/star/2.3.0e/ && rm -fr STAR_2.3.0e.Linux_x86_64

#Tophat2
cd ${APP}
curl http://ccb.jhu.edu/software/tophat/downloads/tophat-2.0.11.Linux_x86_64.tar.gz | tar xvz
mkdir -p ${APP}/tophat/2.0.11 && mv tophat-2.0.11.Linux_x86_64/* ${APP}/tophat/2.0.11/ && rm -fr tophat-2.0.11.Linux_x86_64

#cufflinks
cd ${APP}
curl http://cufflinks.cbcb.umd.edu/downloads/cufflinks-2.2.1.Linux_x86_64.tar.gz | tar xvz
mkdir -p ${APP}/cufflinks/2.2.1 && mv cufflinks-2.2.1.Linux_x86_64/* ${APP}/cufflinks/2.2.1/ && rm -fr cufflinks-2.2.1.Linux_x86_64

#HTSeq
cd ${APP}
echo -e "y\n" | sudo apt-get install python-pip
sudo pip install numpy
sudo pip install scipy
curl https://pypi.python.org/packages/source/H/HTSeq/HTSeq-0.6.1p1.tar.gz | tar xvz
cd HTSeq-0.6.1p1 && python setup.py build && sudo python setup.py install && cd -
sudo rm -fr HTSeq-0.6.1p1

#samtools
cd ${APP}
wget http://downloads.sourceforge.net/project/samtools/samtools/0.1.19/samtools-0.1.19.tar.bz2
tar -jxvf samtools-0.1.19.tar.bz2 && cd samtools-0.1.19/ && make && cd
mkdir -p $APP/samtools/0.1.19/
find samtools-0.1.19 -executable -type f -print0 | xargs -0 -I {} mv {} $APP/samtools/0.1.19/
rm -fr samtools-0.1.19*

#picard
cd ${APP}
wget http://downloads.sourceforge.net/project/picard/picard-tools/1.114/picard-tools-1.114.zip
unzip picard-tools-1.114.zip
mkdir -p ${APP}/picard/1.114/ && mv picard-tools-1.114/* ${APP}/picard/1.114/
rm -fr picard-tools-1.114 picard-tools-1.114.zip

#bamtools
cd $APP
git clone https://github.com/pezmaster31/bamtools.git
cd $APP/bamtools && mkdir build && cd build && cmake .. && make
mkdir -p $APP/bamtools/2.3.0/ && mv $APP/bamtools/* $APP/bamtools/2.3.0/

########################
#step5. install bioinformatics annotation files
########################

#hg19
mkdir -p ${DATA}/fasta/hg19/ && cd ${DATA}/fasta/hg19/
for i in {1..22} X Y M; do wget http://hgdownload.cse.ucsc.edu/goldenPath/hg19/chromosomes/chr${i}.fa.gz; done
gunzip *.gz

#create index and dict for each chromosome files
for i in *.fa
do
j=$(echo $i | cut -d"." -f1)
echo $j
java -jar ${APP}/picard/1.114/CreateSequenceDictionary.jar R=$j.fa O=$j.dict
${APP}/samtools/0.1.19/samtools faidx $j.fa
done

#build hg19 genome index for BWA
mkdir -p $DATA/index/hg19/bwa/
${APP}/bwa/0.7.9a/bwa index -p ${DATA}/index/hg19/bwa/hg19 ${DATA}/fasta/hg19/hg19.fa

#build hg19 genome index for novoalign
mkdir -p ${DATA}/index/novoalign/hg19
${APP}/novocraft/3.02.05/novoindex -k 14 -s 1 ${DATA}/index/hg19/novoalign/hg19.nix ${DATA}/fasta/hg19/hg19.fa

#build hg19 genome index for bowtie1
mkdir -p $DATA/index/hg19/bowtie1/
$APP/bowtie/1.0.1/bowtie-build $DATA/fasta/hg19/hg19.fa $DATA/index/hg19/bowtie1/hg19
cp $DATA/fasta/hg19/hg19.fa $DATA/index/hg19/bowtie1/

#build hg19 genome index for bowtie2
mkdir -p $DATA/index/hg19/bowtie2/
$APP/bowtie/2.2.3/bowtie2-build $DATA/fasta/hg19/hg19.fa $DATA/index/hg19/bowtie2/hg19
cp $DATA/fasta/hg19/hg19.fa $DATA/index/hg19/bowtie2/

#build hg19 genome index for SNAP -require at least 64 GB of memory
mkdir -p ${DATA}/index/snap/hg19
sudo sysctl vm.overcommit_memory=1
${APP}/snap/1.0beta.10/snap index ${DATA}/fasta/hg19/hg19.fa ${DATA}/index/snap/hg19
#{APP}/snap/1.0beta.10/snap paired ${DATA}/index/snap/hg19

#TODO: build hg19 genome index for GSNAP

#TODO: build hg19 genome index for STAR

#dbsnp
curl ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606/VCF/All.vcf.gz | gunzip -c > dbsnp_hg19.vcf
mkdir -p ${DATA}/misc/ && cd ${DATA}/misc/
for chromosome in {1..22} X Y M;
do
awk -v c="$chromosome" '/^#/{print $0;next} $1~"^chr"c {print $0}' dbsnp.vcf > dbsnp_hg19.chr${chromosome}.vcf
done

Extend /home partition under VMWare+Ubuntu

Say I want to add 512GB more space onto existing Ubuntu hosted by VMWare Workstation.

0. You have to turn off your Ubuntu system then switch VMWare Workstation
Right Click the Machine-> "Settings" -> "Hard Disk" in left panel -> click "Utilities" in right panel -> "Expand" -> Enter the new total space. Since my system has 512GB already, and I want to add 512GB, the total would be "1024".

1. Check existing disk partitions. write down the biggest number of End column. here it is 3221225471

$sudo fdisk -l

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 248832 83 Linux
/dev/sda2 501758 2147481599 1073489921 5 Extended
/dev/sda3 2147481600 3221225471 536871936 83 Linux
/dev/sda5 501760 2147481599 1073489920 8e Linux LVM

2. Add new partition

$sudo fdisk /dev/sda

press "n" to create new partition, when when prompting "Partition type" , enter "p".

when prompting First sector, enter 3221225472 (3221225471 + 1)

when prompting Last sector, accept defaults (use all remaining space)

then press "w" to write to disk.

3. reboot

$sudo reboot

4. Run "sudo fdisk -l" again. This time you should see a new partition "/dev/sda4"

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 248832 83 Linux
/dev/sda2 501758 2147481599 1073489921 5 Extended
/dev/sda3 2147481600 3221225471 536871936 83 Linux
/dev/sda4 3221225472 4294967294 536870911+ 83 Linux

/dev/sda5 501760 2147481599 1073489920 8e Linux LVM

4. Find a folder under "/dev/" which ends with "-vg". In should be named as "-vg". e.g. "master-vg" if your hostname is master.

5. Extend the volume group

$sudo vgextend cloud-vg /dev/sda4

You should see screen outputs like this:
No physical volume label read from /dev/sda4
Physical volume "/dev/sda4" successfully created

Volume group "cloud-vg" successfully extended

6. Add logical volume

#add 512G

$sudo lvextend -L+512G /dev/cloud-vg/root

#or add all free space

$sudo lvextend -l+100%FREE /dev/cloud-vg/root

You should see screen outputs like this:

Extending logical volume root to 1.98 TiB
Logical volume root successfully resized

7. Final step - Reize
$sudo resize2fs /dev/cloud-vg/root

8. Check out the disk usage now

$df -h

MongoDB start and stop

#MongoDB

#create
mkdir ~/mongo && mkdir -p ~/log/mongodb

#start
mongod --fork --dbpath ~/mongo --logpath ~/log/mongodb/main.log

#shutdown
mongod --shutdown --dbpath ~/mongo