Wednesday, July 28, 2010

RNASeq@bioserver

galaxy@bioserver>java -Xmx2G -jar /home/galaxy/tools/USeq_6.7/Apps/RNASeq
-y eland
-v D_rerio_Dec_2008
-c /home/galaxy/tmp/OverDispersedRNASeqDataset/Novoalignments/Control
-t /home/galaxy/tmp/OverDispersedRNASeqDataset/Novoalignments/Treatment
-g /home/galaxy/tmp/OverDispersedRNASeqDataset/hg19EnsGenes.ucsc
-s /home/galaxy/tmp/hello
-r /home/galaxy/tools/R-2.11.1/bin/R

Check out the Linux OS Distributor

Vm>lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid

Bioserver>lsb_release -a
LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Release: 5.5
Codename: Tikanga

#Check out where is the executable file?
>which R
/usr/bin/R

>whereis R
R: /usr/bin/R /etc/R /usr/lib/R /usr/lib64/R /usr/local/lib/R /usr/share/R /usr/share/man/man1/R.1.gz

Monday, July 19, 2010

Tomcat6 configuration

1. Download binary Tomcat6
>cd /home/ying/tools/
>wget http://mirror.cc.columbia.edu/pub/software/apache/tomcat/tomcat-6/v6.0.28/bin/apache-tomcat-6.0.28.tar.gz

2. unzip
>tar -zxvf apache-tomcat-6.0.28.tar.gz
>cd apache-tomcat-6.0.28

3. create enviromental variables
>export CATALINA_HOME=/home/ying/tools/apache-tomcat-6.0.28

>echo $JAVA_HOME
/usr/lib/jvm/java-6-sun-1.6.0.20

4. For possible mutiple instance, we want to use anther directory as a host of web applications

>mkdir /home/ying/web
>cd /home/ying/web
>mkdir webapps
>mkdir temp
>cp -r $CATALINA_HOME/conf .
>export CATALINA_BASE=/home/ying/web
>ls $CATALINA_BASE
conf webapps temp

5. Create a sample "index.html" under $CATALINA_BASE/webapps/das2/
>cat $CATALINA_BASE/webapps/das2/index.html
HELLO,WORLD

6. Start tomcat6
>/home/ying/tools/apache-tomcat-6.0.28/bin/startup.sh

7. Open a web browser and access "http://localhost:8080/das2"
HELLO,WORLD

Tuesday, July 13, 2010

DAS/2 Data

Test:

svn export -r102 https://svn.transvar.org/repos/genomes/trunk/pub/das


svn co https://svn.transvar.org/repos/genomes/trunk/pub/das das2



1. download Apache Ant

http://apache.multihomed.net/ant/binaries/apache-ant-1.8.1-bin.zip

2. append the ant to system path.

3. Restart Tomcat
/usr/share/tomcat6/bin/catalina.sh stop
/usr/share/tomcat6/bin/catalina.sh run

DAS2-genopub

==Installing the GenoPub DAS2 Server==

=== Modes ===
There are two ways to configure the GenoViz Genometry DAS2 server:
* Classic mode: The annotations are organized into a directory structure on the file system.
* GenoPub mode: The annotations are managed by the GenoPub (Genomic Annotation Publisher), which stores information about each annotation in a database. A web-based tool allows annotations to be easily added, described, and organized. Through the tool, users and groups are managed and annotation security is specified. Access to annotations is brokered by the DAS2 server so that only authorized users can gain access to these resources.

Here are instructions for installing the DAS2 server in GenoPub mode. Any java servlet container will work. Here we show how to do this with Tomcat. Likewise most relational databases can be appropriately configured. In this demonstration we are using MySQL. Lastly, these web apps were developed and run on Windows but since the majority of bioinformatics web servers use Linux, we use Red Hat Enterprise 5 in this tutorial.


=== System Requirements ===
* Java 6+, 64-bit recommended (http://java.sun.com/)
* Apache Tomcat 5.5+ (http://apache.tomcat.org/)
* MySQL database server (http://mysql.com)
* MySQL jdbc driver (http://mysql.com, download the mysql-connector-java-5.1.7.zip)
* Web accessible Linux, MacOSX, or Windows server with > 4G of RAM. This demo uses Red Hat Enterprize 5.



== Step 1: Check your java version ==
* Check that your java version is 6+ or 1.6+ by typing 'java -version' on the command line
* Download and install a new java if needed, http://java.sun.com/
* In MacOSX modify which java to use with the /Applications/Utilities/Java\ Preferences app


== Step 2: Install and Configure Tomcat 5.5+ ==
* Make a new user called tomcat
* Login as tomcat
* Download and unzip Tomcat 5.5+ in the tomcat user's home directory, see http://tomcat.apache.org/download-55.cgi
* Add environmental variables to the tomcat user's ~/.bashrc file using a text editor


#example for tomcat 5.5.25
export CATALINA_HOME=/home/tomcat/apache-tomcat-5.5.25
export JAVA_HOME=/usr
export JRE_HOME=/usr
#increase the memory available to tomcat, set to 4G or more if you have it, note 32bit machines can only use 2G
export CATALINA_OPTS=-Xmx4096m


* Load the variables into your current shell (e.g. ' source ~/.bashrc ')
* Download the MySQL JDBC driver (MySQL Connector/J) from http://dev.mysql.com/downloads/connector/j/ . Uncompress the zip file (mysql-connector-java-5.xxx.zip) and place the ~/mysql-connector-java-xxx/mysql-connector-java-5.xxx-bin.jar into your ~/apache-tomcat-5.xxx/common/lib directory. This is needed for java apps to connect to a MySQL database.
* Punch holes for port 8080 through any firewalls surrounding your computer
* Enable the tomcat shell scripts to be executed (e.g. ' chmod -R 755 ~/apache-tomcat-5.xxx/bin/ ')
* Launch tomcat using the ~/apache-tomcat-5.xxx/bin/startup.sh script
* Verify that tomcat is visible by calling tomcat from a web browser http://yourComputersIP:8080/ (e.g. http://localhost:8080/). Ask someone off site to check it too.
* Check the server log ~/apache-tomcat-5.xxx/logs/catalina.out for errors
* Shut down tomcat using the ~/apache-tomcat-5.xxx/bin/shutdown.sh script


== Step 3: Install the the GenoViz GenoPub DAS/2 Web app ==
* Download the latest das2server-genopub.zip archive from http://sourceforge.net/projects/genoviz/files/
** To build from the GenoViz project, change the '' to true in the genoviz/common.xml file and use ant to run the clean, all, release-das2server targets in the genoviz/build.xml file. The das2server-genopub.zip will be placed in the genoviz/dist/release directory.
* Unzip the das2server-genopub.zip archive in the tomcat user's home directory
* Move the ~/das2server-genopub/das2genopub directory into ~/apache-tomcat-5.xxx/webapps directory


== Step 4: Install and Configure MySQL ==
* MySQL 5+ (http://dev.mysql.com) comes as a standard install option in RedHat, if needed use yum as root to install MySQL


#To install mysql on RedHat Enterprise 5 and set the root password
su root
yum install mysql-server mysql
/etc/init.d/mysqld start
mysqladmin -u root password some_password
mysql -u root -p


* Start the MySQL daemon '/etc/init.d/mysqld start'
* Login as root 'mysql -u root -p'
* Create a database user account called 'genopub'. Here is an example mysql command:


mysql> CREATE USER 'genopub'@'localhost' IDENTIFIED BY 'some_password';
mysql> GRANT ALL PRIVILEGES ON genopub.* TO 'genopub'@'localhost' WITH GRANT OPTION;
mysql> CREATE USER 'genopub'@'%' IDENTIFIED BY 'some_password';
mysql> GRANT ALL PRIVILEGES ON genopub.* TO 'genopub'@'%' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;


* Logout of the root mysql account and login to mysql as the genopub user, 'mysql -u genopub -p'
* In mysql, run the ~/das2server-genopub/genopub.sql script to create the genopub database 'mysql> source ~/das2server-genopub/genopub.sql;'


== Step 5: Configure the GenoViz GenoPub DAS/2 Web app ==
* Modify the '~/apache-tomcat-5.xxx/webapps/das2genopub/WEB-INF/web.xml' doc in a text editor
** Change the '/path/to/das2genopub/genopubdata/' to the full path directory name where you want to store your data (e.g. '/home/tomcat/GenoPubData' )
** Create that data directory. Note it should not be the same directory used to store data from the classic mode DAS/2 server.
** Change the 'admin@example.com' to point to a trusted admin
** Change the 'http://www.example.com:8080/das2genopub/genome' replacing 'www.example.com:8080' with the URL for your machine. This is the same one you used to verify your tomcat installation.

* Modify the ~/apache-tomcat-5.xxx/webapps/das2genopub/META-INF/context.xml doc in a text editor
** Set the docBase in to the relative path of the webapp application. (e.g. )
** Set the 'your_genopub_db_password_here' in to the genopub user password you assigned in MySQL.

* (Optional) Create a 'index.html' doc describing your server and place it in the '~/apache-tomcat-5.xxx/webapps/das2genopub/' directory. This will be hot linked in some applications.
* (Optional) Likewise, add a favicon.ico to the '~/apache-tomcat-5.xxx/webapps/das2genopub/' directory representing your organization. This will be displayed in apps alongside your genopub server name.


== Step 6: Launch it! ==
* Be sure the mysql daemon is running...
* Start up tomcat...
* Launch the GenoPub tool: http://yourhost:8080/das2genopub/genopub . To programmatically access this sever using DAS/2 the URL is http://yourhost:8080/das2genopub/genome .
* Login with the username 'admin' and the password 'admin'. Click the 'Change password' link on the top right of the screen and change the admin password. Click on the Users and Groups link and add yourself as a user. If you are an admin, click the 'Admin' radio button. Create a group. Add yourself. Quit your browser and try the URL again as yourself to make sure you can login.
* Add some data. Note you need to belong to a group to be able to set the visibility permissions.
* Click the 'Reload DAS/2 Server' link to add your new data to the server


== Step 7: Register Your Das/2 Server ==
* Register your http://yourhost:8080/das2genopub/genome URL, otherwise no one will know about your most excellent data. Note only datasets you designate as public will be visable to non authenticated users.
* Currently, the [http://www.dasregistry.org DasRegistry] does not support DAS/2 services. As a temporary work around, download the 'genopubRegistry.xml' file from the [http://genoviz.svn.sourceforge.net/viewvc/genoviz/trunk/genopub_server/resources/ GenoViz Project], add an entry, and email it back to the [mailto:genoviz-devel@lists.sourceforge.net GenoViz developers] . If you have access to the GenoViz SVN, modify it directly.


== Step 8: Visualize your data in IGB ==
* The [[Software:IGB | Integrated Genome Browser]] is fully DAS/2 and DAS/1 compliant and a good application for visualizing your genomic data.
* Launch IGB using the web start links : [http://bioserver.hci.utah.edu/IGB/igb_beta_1G.jnlp IGB (1GB)] or [http://bioserver.hci.utah.edu/IGB/igb_beta_3G.jnlp IGB (3GB)]
* Under the 'Data Access' tab, click the 'Configure...' button and add your server:
** Server URL 'http://yourhost:8080/das2genopub/genome' '''NOTE''' the last word, it's genome '''NOT''' genopub
** Server name 'YourOrg'
** Server type 'DAS2'
** Login 'your genopub login name'
** Password 'your genopub password'
* Click the 'Add' button, a confirmatory message should appear
* Under the 'Data Access' tab, select a Species and a Genome Version where your data resides


Please send comments to the [http://sourceforge.net/tracker/?group_id=129420&atid=714744 GenoViz Bug Tracker (Category GenoPub)] if you encounter install issues or needed corrections.

Monday, July 12, 2010

First working version

http://h005764.hci.utah.edu

The first working version of DAS/2 client incorporated into Galaxy. Now I can download data from DAS/2 server into Galaxy's working environment directly. There are still some trivial problems to fix but the basic pipeline is done.


To do list:

1. Setup the Genopub based DAS/2 server on my virtual machine.(Now this stuck on the Tomcat startup. Will try to fix this ASAP since it is necessary to start DAS/2 server).

2. MySQL database setup to load genome data.

3. Modify the source code & protocol of DAS/2 server to allow "segments=all chromosomes".
That means download a whole genome as one big file.

4. Create and load some .bam files into the DAS/2 server.

5. Test the .bam files using Galaxy.

main.mako


<script type="text/javascript">
var xmlhttp = new XMLHttpRequest();
if (!xmlhttp)alert("Error initializing XMLHttpRequest!");

//host = "http://h005764.hci.utah.edu:8000"

function query_server()
{
//document.getElementById("server").style.visibility = "hidden";
//document.getElementById("das2_server").style.visibility = "hidden";

//document.getElementById("version").style.visibility = "hidden";
//document.getElementById("das2_version").style.visibility = "hidden";

//document.getElementById("segment").style.visibility = "hidden";
//document.getElementById("das2_segment").style.visibility = "hidden";

//document.getElementById("type").style.visibility = "hidden";
//document.getElementById("das2_type").style.visibility = "hidden";

//document.getElementById("format").style.visibility = "hidden";
//document.getElementById("das2_format").style.visibility = "hidden";
//document.getElementById("download").style.visibility = "hidden";
genome = document.getElementById("genome").value;
url = "/?action=server;genome="+escape(genome);
xmlhttp.open("GET", url, true);
xmlhttp.onreadystatechange = update_server;
xmlhttp.send();
}
function update_server()
{
if (xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
response = xmlhttp.responseText
available_servers = response.split(';')
das2server = document.getElementById("server");
das2server.style.visibility = "visible";
document.getElementById("das2_server").style.visibility = "visible";
for (i=0; i<available_servers.length; i++)
{
das2server.options[i]=new Option(available_servers[i])
}
}
}
function query_version()
{
genome = document.getElementById("genome").value;
server = document.getElementById("server").value;
url = "/?action=version;genome="+escape(genome)+";server="+escape(server);
xmlhttp.open("GET", url, true);
xmlhttp.onreadystatechange = update_version;
xmlhttp.send();
}
function update_version()
{
if (xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
response = xmlhttp.responseText
available_versions = response.split(';')
version = document.getElementById("version");
version.style.visibility = "visible";
document.getElementById("das2_version").style.visibility = "visible";
for (i=0; i<available_versions.length; i++)
{
version.options[i]=new Option(available_versions[i])
}
}
}

function query_segment()
{
genome = document.getElementById("genome").value;
server = document.getElementById("server").value;
version = document.getElementById("version").value;
url = "/?action=segment;genome="+escape(genome)+";server="+escape(server)+";version="+escape(version);
xmlhttp.open("GET", url, true);
xmlhttp.onreadystatechange = update_segment;
xmlhttp.send();
}

function update_segment()
{
if (xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
response = xmlhttp.responseText
available_segments = response.split(';')
segment = document.getElementById("segment");
segment.style.visibility = "visible";
document.getElementById("das2_segment").style.visibility = "visible";
for (i=0; i<available_segments.length; i++)
{
segment.options[i]=new Option(available_segments[i])
}
}
}


function query_type()
{
genome = document.getElementById("genome").value;
server = document.getElementById("server").value;
version = document.getElementById("version").value;
url = "/?action=type;genome="+escape(genome)+";server="+escape(server)+";version="+escape(version);
xmlhttp.open("GET", url, true);
xmlhttp.onreadystatechange = update_type;
xmlhttp.send();
}

function update_type()
{
if (xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
var response = xmlhttp.responseText
available_types = response.split(';')
type = document.getElementById("type");
type.style.visibility = "visible";
document.getElementById("das2_type").style.visibility = "visible";
for (i=0; i<available_types.length; i++)
{
type.options[i]=new Option(available_types[i])
}
}
}



function query_format()
{
genome = document.getElementById("genome").value;
server = document.getElementById("server").value;
version = document.getElementById("version").value;
type = document.getElementById("type").value;
url = "/?action=format;genome="+escape(genome)+";server="+escape(server)+";version="+escape(version)+";type="+escape(type);
xmlhttp.open("GET", url, true);
xmlhttp.onreadystatechange = update_format;
xmlhttp.send();
}

function update_format()
{
if (xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
response = xmlhttp.responseText
available_formats = response.split(';')
format = document.getElementById("format");
format.style.visibility = "visible";
document.getElementById("das2_format").style.visibility = "visible";
for (i=0; i<available_formats.length; i++)
{
format.options[i]=new Option(available_formats[i])
}
}
}
function query_download()
{
genome = document.getElementById("genome").value;
server = document.getElementById("server").value;
version = document.getElementById("version").value;
segment = document.getElementById("segment").value;
type = document.getElementById("type").value;
format = document.getElementById("format").value;
url = "/?action=download;genome="+escape(genome)+";server="+escape(server)+";version="+escape(version)+";segment="+escape(segment)+";type="+escape(type)+";format="+escape(format);
xmlhttp.open("GET", url, true);
xmlhttp.onreadystatechange = show_download;
xmlhttp.send();

}
function show_download()
{
if (xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
response = xmlhttp.responseText
download_URL = document.getElementById("download_url");
download_URL.value=response
download_BTN = document.getElementById("download");
download_BTN.style.visibility = "visible";
}
}
function start_download()
{
download_URL = document.getElementById("download_url");
window.location.href = download_URL.value
//download = document.getElementById("download");
//URL = 'http://netaffxdas.affymetrix.com/das2/genome/H_sapiens_Feb_2009/features?segment=http%3A%2F%2Fnetaffxdas.affymetrix.com%2Fdas2%2Fgenome%2FH_sapiens_Feb_2009%2Fsegments%2Fchr1;type=http%3A%2F%2Fnetaffxdas.affymetrix.com%2Fdas2%2Fgenome%2FH_sapiens_Feb_2009%2Frefseq;format=brs'
}

</script>

<DIV id='das2_genome'>
<select id="genome" onfocus="query_server();">
% for s in genomes:
<option>${s}</option>
% endfor
</select> Genome
</DIV>

<P>
<DIV id='das2_server' style="visibility: hidden">
<select id="server" onfocus="query_version();"></select>DAS2 Server
</DIV>


<P>
<DIV id='das2_version' style="visibility: hidden">
<select id="version" onfocus="query_segment();"></select>Version
</DIV>

<P>
<DIV id='das2_segment' style="visibility: hidden">
<select id="segment" onfocus="query_type();"></select>Segment
</DIV>


<P>
<DIV id='das2_type' style="visibility: hidden">
<select id="type" onfocus="query_format();"></select>Type
</DIV>

<P>
<DIV id='das2_format' style="visibility: hidden">
<select id="format" onchange="query_download();"></select>Format
</DIV>

<P>

<form action="http://h005764.hci.utah.edu/tool_runner" method="GET">
<input type="HIDDEN" name="tool_id" value="das2">
<input type="HIDDEN" id='download_url' name="URL" value="UNKNOWN">
<INPUT type="submit" id="download" value="Get Data" style="visibility: hidden"/>
</FORM>

das2.xml

<?xml version="1.0"?>
<tool name="DAS/2" id="das2" tool_type="data_source">
<description>Server</description>
<command interpreter="python">data_source.py $output $__app__.config.output_size_limit</command>
<inputs action="http://h005764.hci.utah.edu:8000" check_values="false" method="get">
<display>go to Das/2 server $GALAXY_URL</display>
<param name="GALAXY_URL" type="baseurl" value="/tool_runner?tool_id=das2" />
</inputs>
<uihints minwidth="800"/>
<outputs>
<data name="output" format="tabular" />
</outputs>
<options sanitize="False" refresh="True"/>

</tool>


Wednesday, July 7, 2010

GenoPub based DAS/2 Server installation - Part 1

1. Check available Java SDKs

#> update-java-alternatives -l
java-6-openjdk 1061 /usr/lib/jvm/java-6-openjdk
java-6-sun 63 /usr/lib/jvm/java-6-sun

2. Use Sun Java SDK
#>update-java-alternatives -s java-6-sun
update-alternatives: error: no alternatives for mozilla-javaplugin.so.
update-alternatives: error: no alternatives for xulrunner-1.9-javaplugin.so.
update-alternatives: error: no alternatives for mozilla-javaplugin.so.
update-alternatives: error: no alternatives for xulrunner-1.9-javaplugin.so.

3. Make sure Java SDK version > 1.6 (better be 64 bit)
>java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

4. Install Tomcat
>apt-cache search tomcat
>apt-get install tomcat6

#CATALINA_BASE: /var/lib/tomcat6
#CATALINA_HOME: /usr/share/tomcat6
#CATALINA_TMPDIR: /tmp/tomcat6-tmp
#JRE_HOME: /usr/lib/jvm/java-6-openjdk

5. Setup the environment for Tomcat

#>ls /usr/lib/jvm
root@ubuntu:/# ls -al /usr/lib/jvm/
total 60
drwxr-xr-x 5 root root 4096 2010-06-29 13:09 .
drwxr-xr-x 100 root root 32768 2010-07-07 10:14 ..
lrwxrwxrwx 1 root root 14 2010-06-18 13:25 default-java -> java-6-openjdk
drwxr-xr-x 4 root root 4096 2010-06-18 13:25 java-1.5.0-gcj-4.4
lrwxrwxrwx 1 root root 14 2010-06-18 13:25 java-1.6.0-openjdk -> java-6-openjdk
drwxr-xr-x 5 root root 4096 2010-06-18 13:25 java-6-openjdk
-rw-r--r-- 1 root root 2313 2010-04-14 21:25 .java-6-openjdk.jinfo
lrwxrwxrwx 1 root root 19 2010-06-29 13:09 java-6-sun -> java-6-sun-1.6.0.20
drwxr-xr-x 8 root root 4096 2010-06-29 13:09 java-6-sun-1.6.0.20
-rw-r--r-- 1 root root 2392 2010-04-23 08:54 .java-6-sun.jinfo


Append the following lines to ".bash_profile"

###########################
export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.20
export CATALINA_HOME = /usr/share/tomcat6
export CATALINA_BASE = /var/lib/tomcat6
export CATALINA_OPTS=-Xmx4096m

###########################

Then make logs directory for catalina's outputs
#>cd /usr/share/tomcat6
#>mkdir logs


6. Start Tomcat

#>cd /usr/share/tomcat6/bin
#>catalina.sh run

The ROOT directory will be:

/var/lib/tomcat6/webapps/ROOT/

7. Install MySQL

#>at-get install mysql-server #version 5.1

8. Download & Install Connector-J

#>wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.13.tar.gz/from/http://mysql.he.net/

#>tar -zxvf mysql-connector-java-5.1.13.tar.gz

#> cp mysql-connector-java-5.1.13/mysql-connector-java-5.1.13-bin.jar /usr/share/tomcat6/lib/



Thursday, July 1, 2010

Feature download


def download_feature(genome='H_sapiens',version='H_sapiens_Feb_2009',chromosome='chr1',datatype='refseq',format='brs'):
URL = 'http://netaffxdas.affymetrix.com/das2/genome'
affy = DAS2Sources(URL)
feature_url = ''.join((affy.prop_base,affy.capability('H_sapiens','H_sapiens_Feb_2009','features')))

segment_url = ''.join((affy.prop_base,affy.capability('H_sapiens','H_sapiens_Feb_2009','segments')))
segment_query = "segment="+'/'.join((segment_url,chromosome))

type_url = ''.join((affy.prop_base,affy.capability('H_sapiens','H_sapiens_Feb_2009','types')))
type_query = "type="+'/'.join((segment_url,datatype))

format_query='format='+format

query_param = encode_url(';'.join((segment_query,type_query,format_query)))

return '?'.join((feature_url,query_param))