Design of the pipeline

Message queue centralized system and coupled with independent I/O Channel

Design of the pipeline

what is the hardware environment?

Script mapping

#alignment
novoalign -F ILMFQ -t120 -r0.2 -q5 -d hg19Splices34.index -f 1.txt 2.txt > 84060.novoalign

#transform to MAQ
maq novo2maq 84060.novoalign.map - 84060.novoalign

#Convert the reference sequences to the binary fasta format
maq fasta2bfa ref.fasta ref.bfa

#Build the mapping assembly
maq assemble consensus.cns ref.bfa 84060.novoalign.map 2>assemble.log

#Extract consensus sequences and qualities
maq cns2fq consensus.cns >cns.fq

#Extract list of SNPs
maq cns2snp consensus.cns >cns.snp


will be translated to :


/PATH_TO_NONOALIGN/novoalign -F ILMFQ -t120 -r0.2 -q5 -d /PATH_TO_GENOME_INDEX/hg19Splices34.index -f /PATH_TO_INPUT_FILE/1.txt /PATH_TO_INPUT_FILE/2.txt > /PATH_TO_OUTPUT_FILE/1_84060.novoalign
/PATH_TO_MAQ/maq novo2maq /PATH_TO_OUTPUT_FILE/84060.novoalign.map - /PATH_TO_INPUT_FILE/84060.novoalign
...

query, download, upload and update with rsync

#!/home/ying/py2.6.5/bin/python
"""
Use rsync to synchronize the DATA_HOME(hci-bio2, which raw data was stored) and COMP_HOME(chpc-node, 
which process raw data and return output back to source. All the rsync operations are initiated from chpc-node.

Functions:
1. query: query the source if the new data is available
2. download: download new data ( from source to destination)
3. upload: upload result(output) to source.
4. update: update the source by appending a log file from the destination to show the alignment progress on chpc. "begin, percent-finished, end, error, etc"
      
"""
import os
import time
import operator
import subprocess

DATA_HOME = 'ying@123.123.123.123:~/alignment/' 
COMP_HOME = '/home/ying/alignment/' 
RSYNC_QUERY_PARAMS = '-are ssh'
RYSNC_DOWNLOAD_PARAMS = '-avuze ssh' #archive, verbose, update-only, compress-during-transfer and trasnfer-over-ssh  
RYSNC_UPLOAD_PARAMS = '-avuze ssh' #archive, verbose, update-only, compress-during-transfer and trasnfer-over-ssh  
LOG_FILE_NAME = 'log.txt'

def query():
    #query_cmd = """rsync -are ssh ying@155.100.235.73:~/alignment/ | awk '{print $1 " " $3"#"$4 " " $5}'"""
    cmd = ["rsync",RSYNC_QUERY_PARAMS,DATA_HOME,'|',"""awk '{print $1 " " $3"#"$4 " " $5}'"""]
    query_cmd = ' '.join(cmd)
    #print query_cmd
    p = subprocess.Popen(query_cmd,shell=True,stdout=subprocess.PIPE)
    line = p.communicate()
    fnames = ''.join(line[:-1]).strip().split('\n')
    df = {}
    dt = {}
    for fname in fnames:
        prop,ctime,fn = fname.split(' ')
        if fn != '.' and fn != '..':
            if prop[0] == 'd': #it is a directory
                pass
            if prop[0] == '-': #it is a file
                parent_path = os.path.dirname(fn)
                df.setdefault(parent_path,[]).append(os.path.basename(fn))
                #Transform time to seconds, for comparing which job should be run firstly. First In First Run.   
                #'2010/08/20#16:27:40' -> 1282343260.0
                dt[parent_path] = int(time.mktime(time.strptime(ctime,'%Y/%m/%d#%H:%M:%S'))) 
    jobs = []            
    for k,v in df.items():
        if "begin" in v: #the data is ready
            if not LOG_FILE_NAME in v: #this job is not running
                jobs.append((k,dt[k]))
    #print jobs       
    return sorted(jobs,key=operator.itemgetter(1))

def download(path_to_download):
    """download a whole directory from DATA_HOME to COMP_HOME 
    """
    normalized_path = path_to_download.strip('/') #/A, /A/ or A/ -> A
    source = os.path.join(DATA_HOME,normalized_path)
    cmd = ["rsync", RYSNC_DOWNLOAD_PARAMS, source, COMP_HOME] #download the whole directory to COMP_HOME.
    download_cmd = ' '.join(cmd)
    #print download_cmd
    p = subprocess.Popen(download_cmd,shell=True,stdout=subprocess.PIPE)
    line = p.communicate()
    stdout_message = ''.join(line[:-1]).strip().split('\n')
    #print 'Message:',stdout_message
    update(normalized_path,stdout_message)

def upload(path,file_to_upload):
    """upload a file from COMP_HOME to DATA_HOME
     upload 'file_to_upload' at local path 'folder'.
    :/folder/file_to_upload will be uploaded to :/
    """
    normalized_path = path.strip('/')+"/" #/A, /A/ or A/ -> A/
    source = os.path.join(COMP_HOME,normalized_path,file_to_upload) 
    dest = os.path.join(DATA_HOME,normalized_path)
    cmd = ["rsync",RYSNC_UPLOAD_PARAMS, source,dest]
    upload_cmd = ' '.join(cmd)
    p = subprocess.Popen(upload_cmd,shell=True,stdout=subprocess.PIPE)
    line = p.communicate()
    stdout_message = ''.join(line[:-1]).strip().split('\n')
    #fn_log = log(os.path.join(COMP_HOME,normalized_src),stdout_message)
    update(normalized_path,stdout_message)

def update(path,message):
    fn = os.path.join(COMP_HOME,path,LOG_FILE_NAME)
    ofn = None
    try:
        if os.path.exists(fn): 
            ofn = open(fn,'w+')#append 
        else:
            ofn = open(fn,'w') #new file
        print >>ofn,'-'*50
        print >>ofn,time.asctime()
        for m in message:
            print >>ofn,M
        ofn.close()
        normalized_path = path.strip('/')+"/" #/A, /A/ or A/ -> A/
        source = fn 
        dest = os.path.join(DATA_HOME,normalized_path)
        cmd = ["rsync",RYSNC_UPLOAD_PARAMS, source,dest]
        subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE)
    except:
        pass
        
def clear():
    pass
#print query()
#download('A')
upload('A','hello.txt')    

Failed to run RSA key on HCI-BIO2

Password is required.

How to: Make a passwordless ssh connection

How to: Make a passwordless ssh connection between chpc(hello@delicatearch.chpc.utah.edu) and hci(world@hci-bio1.hci.utah.edu)

Suppose after we log into chpc , we want to "ssh" to hci without password.

1. Create RSA keys (on chpc).

hello@chpc:MY_HOME>ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key ($HOME/.ssh/id_rsa):

Enter passphrase (empty for no passphrase): MY_PASSPHRASE

Enter same passphrase again:MY_PASSPHRASE

Your identification has been saved in $HOME/.ssh/id_rsa.

Your public key has been saved in $HOME/.ssh/id_rsa.pub.

The key fingerprint is:

ad:9e:ab:f9:e1:1a:a4:85:16:3b:24:5f:35:b6:76:a7 user@machine

MY_PASSPHRASE is password used to encrypt the keys, not the user's passworld.

Now we have two files under $HOME/.ssh, "id_rsa" is private key, "id_rsa.pub" is public key.

2. Transfer the public key to world@hci's home directory (on chpc)

hello@chpc:$HOME>scp .ssh/id_rsa.pub world@hci-bio1.hci.utah.edu:~/

3. Append the id_rsa.pub to aothorized keys (on hci)

world@hci:$HOME>cat id_rsa.pub >>authorized_keys

4. Make the hci accept RSA key style connection.

By default, this features is off.

We need to modify the following line in /etc/ssh/sshd_config (root privilege is required)

#RSAAuthentication yes

#PubkeyAuthentication yes

to

RSAAuthentication yes

PubkeyAuthentication yes

Then we need to refresh the ssh service.

>/etc/init.d/sshd restart

5. Make a test (on chpc)

>ssh -2 world@hci-bio1.hci.utah.edu

Enter passphrase for key '$HOME/.ssh/id_rsa':

Here we need to input the passphrase of the private key, it is . If everything goes well, we will see the welcome message:

Welcome to Ubuntu!

6. Now we need to eliminate the "passphrase" step using ssh-agent and ssh-add (on chpc)

ssh-agent are used to buffer the passphrase and keep it in memory, we we do not need input passphrase next time.

#start ssh-agent

>eval `ssh-agent`

#add passphrase

>ssh-add

Enter passphrase for chpc:$HOME/.ssh/id_rsa: MY_PASSPHRASE

Identity added: chpc:$HOME/.ssh/id_rsa

7. Make a test again (on chpc)

>ssh -2 world@hci-bio1.hci.utah.edu

Welcome to Ubuntu!

8. we can test scp (on chpc)

>scp hello.txt world@hci-bio1.hci.utah.edu:~/

To avoid run "eval `ssh-agent`" and "ssh-add" every time after we log into the chpc, we can append eval `ssh-agent` to ~/.bash_profile. So it will start automatically next time. However, we still need to run 'ssh-add' manually after each log in to register the passphrase for security. if passphrase is empty, we do not even need ssh-agent and ssh-add. But this may bring security risk to the private key.

we can also use "keychain" as the frontend of ssh-agent, so we do not have to create ssh-agent for each login. With keychain, only one ssh-agent is in service no matter how many consoles we open.

>wget http://www.funtoo.org/archive/keychain/keychain-2.7.1.tar.bz2

>tar -jxvf keychain-2.7.1.tar.bz2

>cd keychain-2.7.1

>./keychain

ssh connection from chpc to bio1

Suppose we want to passwordless ssh connection from machine A(chpc) to machine B(bio1)

1. On machine B

modify the following line in /etc/ssh/sshd_config

(root privilege is required)

#RSAAuthentication yes

#PubkeyAuthentication yes

#AuthorizedKeysFile .ssh/authorized_keys

to

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile .ssh/authorized_keys

reload configuration

#/etc/init.d/ssh reload

2. On machine A

$ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/MY_HOME/.ssh/id_rsa):

Enter passphrase (empty for no passphrase): THIS_IS_MY_PASSWORD

Enter same passphrase again: THIS_IS_MY_PASSWORD

Your identification has been saved in /MY_HOME/.ssh/id_rsa.

Your public key has been saved in /MY_HOME/.ssh/id_rsa.pub.

The key fingerprint is:

ad:9e:ab:f9:e1:1a:a4:85:16:3b:24:5f:35:b6:76:a7 user@machine

$ls .ssh/

id_rsa id_rsa.pub known_hosts

id_rsa is my private key, and id_rsa.pub is the public key

$scp id_rsa.pub user@machineB:~/.ssh

3.On machine B

$cd .ssh/

$cat id_rsa.pub >>authorized_keys

Now make a test from machine A:

$ssh user@machineB

Enter passphrase for key '/MY_HOME/.ssh/id_rsa': THIS_IS_MY_PASSWORD

user@machineB$ #OK now we are using machineB.

Next, we use ssh-agent and ssh-add to avoid "Enter passphrase for key '/MY_HOME/.ssh/id_rsa'", avoid password completely.