Installing a minimal UCSC genome browser mirror in Ubuntu 10.04 64 bits

updated Feb 06, 2014

I recently (April, 2014) found this page in the UCSC wiki.

http://genomewiki.ucsc.edu/index.php/Source_tree_compilation_on_Debian/Ubuntu

Apparently, one can install the genome browser with 2 commands. I haven’t tried it yet.

$ wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/ubuntu.sh
$ source ubuntu.sh

If that doesn’t work or you want to install the browser from scratch, the following should work.

Notes

  • These instructions should work for Ubuntu server 8.04, 9.04, 10.04, 10.10, 12.04 and 14.04 Desktop and server 64 bits. It will work on 32 bit systems, but make sure you compile your binaries, because the ones provided by UCSC are 64 bits.
  • See the Troubleshooting section for issues with MySQL socket and other errors.

Goal and desired features

  1. install a minimal UCSC genome browser for a specific genome
  2. browser should not be at the root of the web dir (I wanted something like: http://www.example.edu/genomebrowser)
  3. restrict access only to the browser with .htaccess
  4. load custom tracks to be displayed permanently

Assuming:

  1. MySQL is up and running in the default port
  2. MySQL datadir is at /home/mysql (just replace this for the default /var/lib/mysql if necessary)
  3. Apache’s DocumentRoot is /var/www

Genome browser requirements

  1. there is a /gbdb directory in /
  2. a directory with html files, ex. /var/www/genomebrowser
  3. a directory with cgi-bin files, ex. /var/www/genomebrowser/cgi-bin
  4. a directory for trash that is in the same directory as cgi-bin (ex. /var/www/genomebrowser/trash)
  5. a configuration file at /var/www/genomebrowser/cgi-bin/hg.conf owned by www-data
  6. XBitHack on in /etc/apache2/httpd.conf
  7. Options +Includes in /etc/apache2/httpd.conf for the directory where html files are located
  8. system has libssl.so.6 and libcrypto.so.6

Setting up a dedicated MySQL user

Enter the commands below in your mysql> shell, editting HOSTNAME and PASSWORD.

create user 'hguser'@'HOSTNAME' identified by 'PASSWORD';
flush privileges;

Test if you can connect to the MySQL database using the user above. You may need to provide the fully qualified hostname, depending on how your hostname is defined. I had to comment bind_address = 127.0.01 at /etc/mysql/my.cnf.

Installation: Part 1. Browser engine

Provide required libraries (in 12.04 the path is slightly different, see below):

apt-get install libssl0.9.8
ln -s /usr/lib/libssl.so.0.9.8 /usr/lib/libssl.so.6
ln -s /usr/lib/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.6

Ubuntu server 12.04:

ln -s /usr/lib/x86_64-linux-gnu/libssl.so.0.9.8 /usr/lib/libssl.so.6
ln -s /usr/lib/x86_64-linux-gnu/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.6

Create a base dir for the genome browser and download html files

mkdir /var/www/genomebrowser
rsync -avzP rsync://hgdownload.cse.ucsc.edu/htdocs/ /var/www/genomebrowser/

Create a customized directory for cgi-bin (not the root cgi-bin) and download cgi-bin files:

mkdir -p /var/www/genomebrowser/cgi-bin
rsync -avzP rsync://hgdownload.cse.ucsc.edu/cgi-bin/ /var/www/genomebrowser/cgi-bin/
chown -R www-data.www-data cgi-bin

Add this to /etc/apache2/httpd.conf:

XBitHack on
 <Directory /var/www/genomebrowser>
   AllowOverride AuthConfig
   Options +Includes
 </Directory>

 # the ScriptAlias directive is crucial
 ScriptAlias /genomebrowser/cgi-bin /var/www/genomebrowser/cgi-bin
 <Directory "/var/www/genomebrowser/cgi-bin">
   AllowOverride None
   Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
   Order allow,deny
   Allow from all
   AddHandler cgi-script cgi pl
 </Directory>

Enable XbitHack

ln -s /etc/apache2/mods-available/include.load /etc/apache2/mods-enabled/

Restart Apache

/etc/init.d/apache2 restart

Create file /var/www/genomebrowser/cgi-bin/hg.conf

# Configuration file for the UCSC Human Genome server
#
# the format is in the form of name/value pairs, written as 'name=value'
#
# note that there is no space between the name and its value. Also, no blank lines should be in this file.
#
#--------------------------------------------------------------#
#
# db.host is the name of the MySQL host to connect to
db.host=YOURHOST
#
# db.user is the username used when connecting to the host
db.user=hguser
#
# this is the password to use with the above hostname
db.password=PASSWORD
#
db.trackDb=trackDb

# central.host is the name of the host of the central MySQL
# database where stuff common to all versions of the genome
# and the user database is stored.
central.db=hgcentral
central.host=YOURHOST
central.user=hguser
central.password=PASSWORD
central.domain=

backupcentral.db=hgcentral
backupcentral.host=YOURHOST
backupcentral.user=hguser
backupcentral.password=PASSWORD
backupcentral.domain=

Give ownership to www-data:

sudo chown www-data /var/www/genomebrowser/cgi-bin/hg.conf

Create other directories required by the browser:

rm /var/www/genomebrowser/trash
mkdir /var/www/genomebrowser/trash
chown www-data.www-data /var/www/genomebrowser/trash

The following links are necessary because the browser assumes it is installed in the root dir of the server. Alternatively, you can setup these links at /etc/apache2/httpd.conf

ln -s /var/www/genomebrowser/images /var/www/images
ln -s /var/www/genomebrowser /var/www/html

Providing javascript files required by the binaries. DO NOT copy js and style to /usr/local/apache/htdocs. The browser will display tracks out of alignment!

mkdir -p /usr/local/apache/htdocs/
ln -s /var/www/genomebrowser/js/ /usr/local/apache/htdocs/js
ln -s /var/www/genomebrowser/style/ /usr/local/apache/htdocs/style

Test the installation by pointing your browser to http://localhost/genomebrowser

Setup crontab to clean trash

Create a script at /etc/cron.daily (no . or _ allowed in the file name) with the following contents:

#!/bin/bash

find /var/www/genomebrowser/trash/ \! \( -regex "/var/www/genomebrowser/trash/ct/.*" \
      -or -regex "/var/www/genomebrowser/trash/hgSs/.*" \) -type f -amin +5040 -exec rm -f {} \;
find /var/www/genomebrowser/trash/    \( -regex "/var/www/genomebrowser/trash/ct/.*" \
      -or -regex "/var/www/genomebrowser/trash/hgSs/.*" \) -type f -amin +10080 -exec rm -f {} \;

You can change the clean up schedule by changing +5040 and +10080 to the number of minutes that you want.

Installation: Part 2. MySQL tables required for functionality

For a minimal installation, the genome browser requires the following databases:

  • hgcentral
  • hgFixed

Download hgcentral to your MySQL directory

wget http://hgdownload.cse.ucsc.edu/admin/hgcentral.sql
mysql -youraccountoptions -e "create database hgcentral"
mysql -youraccountoptions hgcentral < hgcentral.sql
mysql -youraccountoptions -e "grant all privileges on hgcentral.* to 'hguser'@'HOSTNAME'"

Create a dummy hgFixed

mysql -youraccountoptions -e "create database hgFixed"
mysql -youraccountoptions -e "grant select on hgFixed.* to 'hguser'@'HOSTNAME'"

Your UCSC Genome Browser should be working (no data to display, though)

Installation: Part 3. Adding one genome

Shut down your MySQL database

kill -15 `ps aux | grep mysqld | grep 3306 | awk '{print $2}'`

Let’s use mm9 as an example

mkdir /home/mysql/mm9

The database of interest requires the following tables, at least:

  • chromInfo
  • cytoBandIdeo <- not required, but shows the chromosome at the top
  • extFile <- not strictly required for minimal functionality, but necessary for zooming in
  • grp
  • hgFindSpec
  • trackDb

Download these databases

rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/chromInfo.MYD /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/chromInfo.MYI /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/chromInfo.frm /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/cytoBandIdeo.MYD /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/cytoBandIdeo.MYI /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/cytoBandIdeo.frm /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/grp.MYD /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/grp.MYI /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/grp.frm /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/hgFindSpec.MYD /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/hgFindSpec.MYI /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/hgFindSpec.frm /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/trackDb.MYD /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/trackDb.MYI /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/trackDb.frm /home/mysql/mm9

Setup permissions

chown -R mysql.mysql /home/mysql/mm9

Restart your MySQL db

/usr/sbin/mysqld --basedir=/usr --datadir=/home/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid \
    --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock &

Grant privileges on mm9

mysql -youraccountoptions -e "grant all privileges on mm9.* to 'hguser'@'HOSTNAME'"

Download gbdb data

mkdir -p /home/genomebrowser/gbdb/mm9
rsync -avzP --delete --max-delete=20 rsync://hgdownload.cse.ucsc.edu/gbdb/mm9/ /home/genomebrowser/gbdb/mm9/

Provide directory gbdb to the browser

ln -s /home/genomebrowser/gbdb /gbdb

With this, your browser should be able to display mm9 data. The first time I access the browser, I add ?db=mm9 after hgGateway (http://localhost/cgi-bin/hgGateway?db=mm9) otherwise, the browser tries to show data for hg19. After the first time, the browser somehow learned that the only database available is mm9.

Adding UCSC tracks (MySQL tables)

Select the data you want from ftp://hgdownload.cse.ucsc.edu/mysql/mm9/ and download to /home/mysql/mm9.

For example, I wanted my browser to display:

  • RefSeq
  • UCSC genes
  • SNPs and repeats
  • Conservation

I downloaded these MySQL tables.

Additional functionality

Some tables can be left out but you will probably want them to obtain extra functionality. For example, when you click on a gene or feature in the genome viewer, you expect to be able to retrieve more info.

For this, you need to install the proteome database. After shutting down your MySQL server:

mkdir /home/mysql/proteins070202

That’s it, a dummy proteins070202 does the trick.

Displaying info about RefSeq genes

rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/description.frm /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/description.MYD /home/mysql/mm9
rsync -avzP  rsync://hgdownload.cse.ucsc.edu/mysql/mm9/description.MYI /home/mysql/mm9

Displaying info about UCSC Genes

Get the UniProt database

wget -nc -c ftp://hgdownload.cse.ucsc.edu/mysql/uniProt/*

Get the proteome database

wget -nc -c ftp://hgdownload.cse.ucsc.edu/mysql/proteome/*.*

Summary of my setup

MySQL databases:

  • hgcentral (380 kb)
  • hgFixed (empty database)
  • mm9 (selected tables: 5.8 G)
  • proteins070202 (empty database)
  • proteome (full database: 8.9 G)
  • uniProt (full database: 8 G)

Other files:

  • gbdb (62 G)

Adding custom tables

How the UCSC Genome Browser displays tracks in the browser:

1. the table mm9.grp describes groups of tracks (Mapping and Sequencing Tracks, Genes and … tracks)

If you want a new group instead of adding your custom track to a pre-existing one, you must create a new group in that table:

mysql -youraccountinfo -e "insert into grp set name='$name', label='$label', priority=$priority;"

2. the table mm9.tracksDb describes how tracks are displayed in their groups

The new table must have an entry in tracksDb describing the grp they belong to, color, name, etc.

3. use the hgLoadBed utility to upload .bed files. Notice that this script reads MySQL configuration data from ~/.hg.conf

/var/www/genomebrowser/cgi-bin/loader/hgLoadBed mm9 mynewtable myfile.bed

You can write some Perl scripts to automate the process or use UCSC’s utilities.

Configuring local BLAT

1. login to your hgcentral database and update the BLAT server to be your localhost (or another computer):

update blatServers set host='myserver.university.edu', port=8083 where db='mm9';

2. build BLAT

You will need to download the “kent source” from GIT and compile BLAT in your machine.

a. setup $MACHTYPE by running “uname -a” to find your architeture. Mine was x86_64

MACHTYPE=x86_64

b. create a directory ~/bin (the README says your binaries will be in ~/bin/$MACHTYPE, but in my case, they were under ~/bin)

c. go to your kent/src directory

  • go to lib and run make (should create jkweb.a). In my system, I had to run make as root and strangely, jkweb.a was created at /jkweb.a. So I had to move it from / to lib
  • go to jkOwnLib and run make (should create lib/jkOwnLib.a)
  • run make in gfServer, gfClient, blat and faToNib
  • binaries should appear under ~/bin

d. copy or move all binaries created blat, faToNib, gfClient and gfServer to /usr/local/bin

e. start gfServer:

  • go to /gbdb/mm9 (to start a blat server for mm9). This is important. Don’t give your full mm9.2bit path. If you start the server with /gbdb/mm9/mm9.2bit, for example, blat will search for /gbdb/mm9/gbdb/mm9/mm9.2bit
  • start gfServer by doing:
gfServer start myserver.university.edu 8083 -stepSize=5 -log=/tmp/gfserver.log mm9.2bit

This is what you should see:

$ gfServer start myserver.university.edu 8083 -stepSize=5 -log=/tmp/Log.log mm9.2bit
starting untranslated server...
Counting tiles in mm9.2bit
[several minutes and a good deal of RAM later]
Done adding
Server ready for queries!

You will need >2GB of RAM to start the server. Every time you run a BLAT search on your browser, ~2GB will be used for a few seconds in your BLAT server.

The neat thing is that you can run gfServer in any computer and simply tell the MySQL table blatServers where to find the BLAT server.

You can now run BLAT from your browser. You can check the status of the BLAT server by doing:

$ gfServer status myserver.university.edu 8083

Troubleshooting

Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (13)

Strangely, I didn’t have this problem with one 10.04 server, but I had with another.

  • create a symbolic link from your actual socket to /var/lib: ln -s /var/run/mysqld/mysqld.sock /var/lib/mysql/mysql.sock
  • set permissions to the socked file: chmod 666 /var/lib/mysql/mysql.sock
  • set permissions to the socked directory: chmod 755 /var/lib/mysql/
  • test your connection from the shell:  mysql [your credentials] –socket=/var/lib/mysql/mysql.sock, you should get access to your MySQL db
  • I did not need to enter the socket path in /etc/apparmor.d/usr.sbin.mysqld, but if you still have access problems, you may try this
  • I did not modify /etc/mysql/debian.cnf
  • You do not need to restart your MySQL

(8)Exec format error: exec of ‘/var/www/genomebrowser/cgi-bin/hgGateway’ failed

  • Check that you have hg.conf under cgi-bin (with hgGateway)
  • Check that hg.conf is owned by www-data
  • manually run hgGateway. If if fails with “-bash: ./hgGateway: cannot execute binary file”, chances are you have a 32 bit system and you are trying to run a 64 bit binary. Install Ubuntu 64 bits or compile the browser from source. To check your system, use arch or uname -a, it should be x86_64

Can’t find hg19, but you want to display another organism

  • in the URL of your mirror give the code name of organism name that you found after db= as argument. For mouse: hgGateway?db=mm9, for Drosophila: hgGateway?org=dm3, etc

Browser loads, but nothing is displayed

  • make sure you installed the chromosome and ideogram tables
  • make sure there is a trash directory at the same level of the cgi-bin directory, and trash is www-data writable

Tracks appear scrambled

  1. One solution that worked for me was symlinking instead of copying directories js and style: ln -s /var/www/genomebrowser/js/ /usr/local/apache/htdocs/js and ln -s /var/www/genomebrowser/style/ /usr/local/apache/htdocs/style
  2. In another server, this solution failed. Creating a link in the DocumentRoot to genomebrowser/js and genomebrowser/style solved the problem.

Error in TCP non-blocking connect() 111 – Connection refused

If you get this error when trying to do a BLAT search on your local mirror, it means you didn’t start the BLAT server (gfServer start …).

See also

http://genome.ucsc.edu/admin/mirror.html

http://genomewiki.ucsc.edu/index.php/Minimal_Browser_Installation

http://bradbot.genomecenter.ucdavis.edu/wiki/index.php/Genome_Browser

67 thoughts on “Installing a minimal UCSC genome browser mirror in Ubuntu 10.04 64 bits

  1. Thank you for this write-up. Unfortunately when I try to test the Genome browser I get the error:

    The requested URL /genomebrowser/cgi-bin/hgGateway was not found on this server.

    However I see that it is in it’s correct location with wide-open permissions.

    Any suggestions?

  2. Oops I think I found the issue. I adjusted your directions for the cgi files:

    mkdir -p /usr/lib/cgi-bin/genomebrowser/cgi-bin
    rsync -avzP rsync://hgdownload.cse.ucsc.edu/cgi-bin/ /var/www/genomebrowser/cgi-bin

    to this:

    mkdir -p /usr/lib/cgi-bin/genomebrowser/cgi-bin
    rsync -avzP rsync://hgdownload.cse.ucsc.edu/cgi-bin/ /usr/lib/cgi-bin/genomebrowser/cgi-bin

    Now it can find hgGateway but in the apache error log i see:

    (8)Exec format error: exec of ‘/usr/lib/cgi-bin/genomebrowser/cgi-bin/hgGateway’ failed
    Premature end of script headers: hgGateway

    Other suggestions?

  3. In my setup, I have /var/www/genomebrowser/cgi-bin pointing to /usr/lib/cgi-bin/genomebrowser/cgi-bin, that’s why I downloaded data to the former. I’m glad you were able to fix it. (outdated)

    As for the second error… Check your /var/log/apache2 for hints. Did you configure /etc/apache2/httpd.conf? Is your cgi-bin directory allowed to execute scripts?
    Another possibility, do you have hg.conf in the correct path?

  4. Thanks for the reply. I found out the exec errors where because I didn’t have hg.conf in the right location.

    NOW when I try to run the browser it complains about:
    Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (13)

    I’ve tried the instructions of creating a symlink:
    ln -s /var/run/mysqld/mysqld.sock /var/lib/mysql/mysql.sock

    (located here: https://lists.soe.ucsc.edu/pipermail/genome-mirror/2010-July/001953.html)

    but I still can’t run it. Any ideas?

  5. Yep mysql is definitely running. Searching for an answer. I wonder if something drastic has changed between your instructions version of Ubuntu and 10.04.

  6. That’s sad, do you have any ideas what might have changed in 10.04 that affected the mirror?
    I have it currently running in 8.04, but before, I was running in 9.04.
    I wish I could be of more help. Hope you get it working soon.

  7. There’s definitely socket changes and I’m not sure what else. I’m just doing this setup as a favor to someone and I want to get through it and be done with it! =)

  8. I am in the process of installing a genome brower on an ubutu 10.04 (lucid). I achieved to see the browser once. Then, the mysql.sock disappeared and to solve this I tried first, to reboot the server with no results. Then I removed and reinstalled the MySQL server installing mysql-server-5.1. Now, I have a socket and MySQL is running (checked these out) but when I try to see the browser that is what I obtain:

    Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (13)

    Any ideas, please?

    Núria

  9. Hi Noboru!

    I fixed the error with new year’s inspiration!!! In fact, the solution was near: I’ve only changed the socket path in the /etc/mysql/debian.cnf file, restarted MySQL, and changed the permission access of the directory where the socket file is:

    From:

    drwxr—– 3 mysql mysql 4096 2011-01-03 09:38 mysql

    To (via chmod 755):

    drwxr-xr-x 3 mysql mysql 4096 2011-01-03 09:38 mysql

    And that’s it!!! No more socket problems for the moment!! Ieeeeei!!! I hope that can help anyone with the same socket problem!!!

    But, now I have a new problem …:

    Couldn’t connect to database (null) on localhost as centralgbuser. Client does not support authentication protocol requested by server; consider upgrading MySQL client

    And I positively know that I can do a mysql_connect() to the hgcentral DB without problem … If do you have any idea on this new issue it would be great!!!

    Cheers,

    Núria

  10. I fixed it!!! I’m seeing the genome browser right now woooooow!!!!

    What I did is:

    mysql> SET PASSWORD FOR ‘user’@’localhost’ = OLD_PASSWORD(‘passwd’);

    for each genome browser user that i have, and that’s it!

    I hope this helps someone!!

    Núria

  11. Hi Noboru!

    I have a problem. How could I see the gene search box in the browser gateway? Thanks in advance!

    Cheers,

    Núria

      1. Hi Noboru!

        Thank you for your quick replay! The answer is yes BUT I could only see a few tracks, not UCSC genes for instance. Now, after following your instructions on getting and implementing uniprot and proteome DBs, the browser stops working. The problem is with the hgTracks script:

        From apache (error.log) I get:

        Premature end of script headers: hgTracks, referer: http://bioinfold/cgi-bin/hgGateway

        From syslog:

        hgTracks[15870]: segfault at 3330 ip 00c9c50b sp bf803ffc error 4 in libc-2.11.1.so[c5c000+153000]

        From the browser:

        Internal Server Error

        The server encountered an internal error or misconfiguration and was unable to complete your request.

        More information about this error may be available in the server error log.

        I’m quite lost. Do you know something about this?

      2. Ok, I was able to reproduce your error. It occurred when copying ONLY knownGene.*. You need to copy other knownXXXX.* and kgXXXX.* tables, and have /gbdb setup.
        Check my tutorial. I provided the list of tables that I had to install in order to have:
        * RefSeq
        * UCSC genes
        * SNPs and repeats
        * Conservation
        The list is here: https://enotacoes.files.wordpress.com/2009/09/mm9_tables1.pdf

        Installing select tracks is difficult because there is a lot of dependency issues, so you have to figure out what else is needed.

  12. You’re right!!! I only downloaded knownGene* tables!!! I’m amazed!!! I didn’t copy other tables because I have the human genome and your tables are related to the mouse one and I didn’t know what to copy. Now I’m copying knownXXXX.* and kgXXXX.* tables. Let’s see!!!

    Thank you very much again!!!

    Núria

  13. Hi Noboru,

    i’m having a problem and i have no clue of what is going on… I managed to install genome browser and hg19 tracks following your totorial, thanks!

    The problem is that when i try to browse, one track is above another, as if all lines were overlapped and the “grid” would not expand… Any clues ? (was i clear enough?)

    Thanks in advance,
    Rodrigo.

    1. Hi Rodrigo, I think I know what you’re talking about. I’ve had this problem.
      What worked for me was not copying the js and style directories to /usr/local/apache/htdocs, instead, I created a symbolic link to them.

      mkdir -p /usr/local/apache/htdocs/
      ln -s /var/www/genomebrowser/js/ /usr/local/apache/htdocs/js
      ln -s /var/www/genomebrowser/style/ /usr/local/apache/htdocs/style

      However, it seems that you have done this and still got the problem? This problem bugged for days until I used the symbolic link, so I don’t know what else to do, unfortunately. I don’t even know what questions to ask to try to help…

      Did you follow the tutorial fully?

      1. Thanks for the quick tip!

        I did not fully followed the tutorial (got here too late,😀 )

        Thanks! It worked like a charm.

        Rodrigo.

  14. Hi again Noboru,

    i am inserting a new genome in my database, but i can’t enable BLAT.
    The output is “Can’t find a server for DNA database schisto51. Click here to reset to default database.”

    How do I set up this DNA database?

    Thanks again,
    Rodrigo.

      1. Just for the record. Blat is called as a web service in genome browser, so your blat server doesn’t have to be in the same server (plus you may have several blat servers or use UCSC blat servers). All you have to do is run gfServer (i.e. : gfServer start yourSever 8083 -stepSize=5 -log=Log.log yourGenome.2bit) and insert your server in blat table (mysql).

  15. Lot of problems (mysql socket, symlink, access denied, etc…) are related to AppArmor blocking access. Don’t forget to edit /etc/apparmor.d/usr.sbin.mysqld and usr.sbin.apache2 if you have AppArmor activated and reload with “service apparmor reload”

  16. Hi dude

    I am in the process of installing a genome brower on an my Server(Suse 64bits). The Page (hgGateway) shows up after I fixed some errors. But if I click “submit” to see tracks, the browser stops working.

    From apache (error.log) I get:
    [error] [client 11.11.11.40] Premature end of script headers: hgTracks, referer: http://11.11.11.31/cgi-bin/hgGateway
    Or
    [client 11.11.11.40] Premature end of script headers: hgTracks, referer: http://11.11.11.31/cgi-bin/hgTracks?hgsid=128&chromInfoPage=
    From the Brower : Internal Server Error.
    After this I try to manually run hgTracks with ./hgTracks. It fails with “ Segmentation fault” ,I thick it cannot be executed.

    node31:/usr/local/apache2/cgi-bin # ./hgTracks
    Set-Cookie: hguid=436; path=/; domain=node31; expires=Thu, 31-Dec-2037 23:59:59 GMT
    Content-Type:text/html

    Sea hare – UCSC Genome Browser v248

    document.write(““);
    function showWarnBox() {document.getElementById(‘warnOK’).innerHTML=’ OK ‘;var warnBox=document.getElementById(‘warnBox’);warnBox.style.display=”; warnBox.style.width=’65%’;document.getElementById(‘warnHead’).innerHTML=’Error(s):’;}
    function hideWarnBox() {var warnBox=document.getElementById(‘warnBox’);warnBox.style.display=’none’;warnBox.innerHTML=”;var endOfPage = document.body.innerHTML.substr(document.body.innerHTML.length-20);if(endOfPage.lastIndexOf(‘– ERROR –‘) > 0) { history.back(); }}
    JavaScript is disabled in your web browserYou must have JavaScript enabled in your web browser to use the Genome Browser

    (click to hide)

    Segmentation fault

    1. It seems that lidaof did not installed genomebrowser in a default location or lidaof has many sites in the same server, am i right?
      Either way, i had to do it, lidaof. I think i’ve created a script for that task…

      1. exactly. i am trying to have each browser for each user. in addition, install UCSC mirror not on default DocumentRoot directory need to edit the inc/*topbar* html files, to use the relative path.
        @noborujs: you didnot edit those topbar.html files? the links on your help page works?

  17. I didn’t edit topbar.html, all my links work fine.
    My document root is not default either, it is /home/www/genomebrowser/genomebrowser, but I have no problem with links.
    I used to have multiple genome browsers, but I don’t remember having problems with links.

    How are you maintaining multiple browsers in the same server?
    I used to simply configure /etc/apache/http.conf with different directories.
    But now that they are using jQuery, it seems hgTracks reads /usr/local/apache/htdocs/js and style, and my second browser doesn’t work (tracks appear all scrambled, it’s not reading js and style correctly).

    1. My tracks appear all scrambled as well, but I don’t have multiple browsers installed. The trick not to copy but to create a symlink to style and js in the htdocs file makes no difference.
      The tracks are shifted to the right an appear twice. I have no idea what to do against that.

      1. I’m glad you solved the problem!

        Just curious, did you do this step of my tutorial?
        # ln -s /var/www/genomebrowser/js/ /usr/local/apache/htdocs/js
        # ln -s /var/www/genomebrowser/style/ /usr/local/apache/htdocs/style
        According to the tutorial, http://localhost/js should already exist as /var/www/genomebrowser/js (same for style), which come with the UCSC browser engine.

        Anyway, thanks for posting the solution, I’m leaving your posts for other people.
        Thanks!

  18. If you use Ubuntu 11.10, you might get this error when you try to start the mysql server:

    Warning] Can’t create test file /home/mysql/ubuntu.lower-test

    I found the solution here http://neodon.blogspot.com/2008/09/changing-data-directory-for-mysql-in.html

    Once you know AppArmor is the problem, the fix is very easy. You simply edit /etc/apparmor.d/usr.sbin.mysqld. Underneath the two lines authorizing the default MySQL data directories, add two more with your custom directory. Make sure you have a trailing / on the directory name, otherwise it will not work (I had this problem at first). After this change, restart AppArmor:
    sudo invoke-rc.d apparmor restart

  19. Hello,
    Thank you for this tutorial, very easy to understand.
    I followed it until the end of Part 2. The front page is working fine but when I try to go on the genome browser page I get an Internal Servor Error.
    Do you have any idea of where I screwed up ?
    Thank !

    1. I’m sorry you’re having problems following my tutorial. I have tested it a couple of times, but some error might have crept in.
      Internal server errors might happen if you don’t have the cgi-bin/hg.conf config file, or it doesn’t have the correct permissions. Or maybe your cgi-bin is not properly configured in http.conf. Do you have other pages (html/php) working in the same server? If you have errors in your Apache config files, what you get is an Internal server error.

      Also, check your /var/log/apache2/error.log for clues.

      Post your solution, so others can benefit from it!

      1. Thank you for replying so fast !
        In my /var/log/apache2/error.log I get this line:

        [Thu May 24 12:37:25 2012] [error] [client 127.0.0.1] Premature end of script headers: hgGateway, referer: http://localhost/genomebrowser/
        /var/www/genomebrowser/cgi-bin/hgGateway: error while loading shared libraries: libssl.so.6: cannot open shared object file: No such file or directory

        I did the two steps of your tutorial:
        ln -s /usr/lib/libssl.so.0.9.8 /usr/lib/libssl.so.6
        ln -s /usr/lib/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.6

        Then I am not sure what is going on. Any idea?

        Thank you again !

      2. Hi guys,

        I had the same problem with the Ubuntu Server 12.10. Looking in the /var/log/apache2/error.log, I found that the symbolic links created seems to be broken

        /var/www/genomebrowser/cgi-bin/hgGateway: error while loading shared libraries: libssl.so.6: cannot open shared object file: No such file or directory

        /var/www/genomebrowser/cgi-bin/hgGateway: error while loading shared libraries: libcrypto.so.6: cannot open shared object file: No such file or directory

        ####

        Try to remove the just created links:

        rm /usr/lib/libssl.so.6
        rm /usr/lib/libcrypto.so.6

        and create them again with the new path:

        ln -s /lib/x86_64-linux-gnu/libssl.so.0.9.8 /usr/lib/libssl.so.6
        ln -s /lib/x86_64-linux-gnu/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.6

        It worked for me.

        Bye

  20. Hi,

    Thanks for providing this very helpful guide. I followed it to install the genome browser and added hg18 genome, it works fine until I try to upload user tracks.

    If I upload a file of ~10kb, the genome browser works fine. But if I upload a file of ~10mb, it can still upload, but when I click on “go to genome browser” after uploading the file, my web browser says “Internal Server Error” and could not display anything. Do you have any idea on this?

    Thanks,
    Guangyao

    1. I just tried a 15M file in my mirror and it didn’t give me an internal server error.
      Check your .bed file, are all chromosomes legal? Is the bed file correctly formatted?
      Try to upload the first 100, then 1000 lines, etc until you find the problematic entry.

      1. I think the file is correct since I can upload and display it in UCSC GenomeBrowser. I am wondering if this is because I did not download all the mysql and gbdb data for hg18 (which is huge). I have downloaded all the mysql and gbdb datasets for drosophila(dm3) (since they are not so big), and I can successfully upload and display ~10mb data in dm3. Or maybe another possibility is that the memory of my linux machine is not big enough?

      2. Thanks for the quick reply! I am just wondering have you downloaded all the mysql and gbdb datasets for mm9 to make it work? I could not afford to download them all for hg18 since I only have ~1Tb hard disk space. I have tried to download all gbdb data for hg18 since last night, it now takes ~800Gb and I have to give it up.

      3. I downloaded all gbdb files for mm9, ~100Gb and only select mm9 MySQL tables, as indicated in my tutorial.
        My first mm9 mirror ran on a 250Gb disk, so it should be possible to run on a 1TB disk.
        You will need a minimum set of MySQL tables, check my tutorial above. I learned which ones by trial and error, so if you want to add more functionality, I’m afraid you’ll have to do the same. I use the table browser and a lot of educated guessing based on file names at UCSC Genome Browser’s download page to add functionality.

        Edit: I also downloaded 100 Gb of Genbank files to display nt sequences.

    2. Thanks for your help during the process! I finally figure this out: the minimal required dateset for gbdb is .2bit or nib; and I also downloaded the mysql datasets as listed in your tutorial; the custome track issue is because the Gbrowser somehow could not display files in bedGraph format, but bedGraphToBigWig can transform them to bigWig format.

  21. Following tutorial all ok so far, fingers crossed. Should I create a file called “cron.daily” when the brackets comment says no . in name, if I call it something else how will it be found?

    Create a script at /etc/cron.daily (no . or _ allowed in the file name) with the following contents:

    1. /etc/cron.daily is a system directory in your Linux installation where scripts (files) reside and are run daily.
      You have to create a script there without . or _ in their file names. You can name it something like “cleangenomebrowser”, so the full path to it would be /etc/cron.daily/cleangenomebrowser. The contents of the script should clean temporary directories daily.

      1. I am doing right now. Just compiled mysql 5.6.16 from source. Can I post questions if I run into some problem?

      2. I ended up copying my previous genomebrowser to my new machine (installed 12.04 from scratch). I still had to perform some steps in this tutorial, but I skipped all the data download part, so if something changed in UCSC, I wouldn’t know.

        The only thing that I had to do differently was changing the path to libssl.

        Let me know if this tutorial still works for 12.04.

      3. I haven’t been able to work on it in since compiled the mysql. Will let you know if I was successful or encounter any problem.

  22. Having some trouble accessing tables through the website. There were errors in table access (Can’t start query error) which I fixed by setting mysql grants to allow user ‘readonly’@’%’ rather than ‘readonly’@’localhost’. However, now I don’t get any errors at all; the page just doesn’t load.

    1. By the way: running Ubuntu 12.04. localhost/genomebrowser works fine, but localhost/genomebrowser/cgi-bin/hgGateway?db=mm9 just doesn’t load.

  23. Problem: made it to the browser gateway page, but after entering a search term, the page loads indefinitely until a timeout error and does not complete the search. Could this have to do with the MySQL database in some way not being connected to the site?

    1. Usually, when MySQL errors occur they occur fast and with a message on screen. If you are able to browse, your MySQL is connected. I can’t think of how to fix this search problem. Can you check your Apache log for clues?

      1. I see, thanks for directing me to the log. The error was that I was missing an mm9 table (mm9.gbCdnaInfo). This is because I never finished the rsync download for the mm9 data. I’ve already downloaded 159 gigabytes of the data using a version of the command you provided, “sudo rsync -avzP –partial –progress –delete –max-delete=20 rsync://hgdownload.cse.ucsc.edu/gbdb/mm9/ /gbdb/mm9/” and the command crashes every so often, so I resume the download after reboot using the command again. But 159 GB is much larger than the value provided in this tutorial (62 GB) so am I downloading the wrong table? Or perhaps the dataset has changed? Thanks so much for responding to comments, by the way.

      2. oh wow, using the command “rsync -navzP rsync://hgdownload.cse.ucsc.edu/gbdb/mm9” shows the the full dataset is of size 2203844935342. If that’s in bytes, that corresponds to 2052 GB! Definitely the wrong dataset. What’s going on? Should the goldenpath/mm9 set be used instead?

      3. Alright, I fixed the above problems. Sorry about all the comments. The problem was that I never completed downloading the tables in the pdf you linked.
        My current problem is again that the search never completes, except now the apache error log lists “Can’t start query: select id from organism where name = ‘Mus musculus’ mySQL error 1146: Table ‘mm9.organism’ doesn’t exist”.
        I’ve also had quite a few corrupted table problems in the mysql error log, which I keep on fixing using myisamchk. The most recent error in the mysql log is “incorrect information in file: ‘./uniprot/history.frm'”. Could these be related to the apache error?

      4. /gbdb/mm9 might have changed, I see a lot of huge files associated with the ENCODE project. You will have to select a few files/directories. This is what’s in my server, but I haven’t updated since I first installed it:
        allenBrain html mm9.2bit visiGene
        bbi igtc multiz30way stsMarker wib
        knownGene.ix NIAGene targetDb
        blastzRetro knownGene.ixx RNA-img trackDb.ix
        cloneend liftOver snp trackDb.ixx

        Goldenpath has other data. You need gbdb data.

        The MySQL problem seems to be the lack of the specific database organism. I think the browser will show the specific databases missing in the error message. You should download each one. I do have that table.

      5. That ended up working, thanks so much!

        A few links in search results (all of the “Mouse Unaligned mRNA Search Results”) tend to be broken and lead to internal server errors, which the apache error log describes as premature end of script header errors in hgc. Do you know how to add functionality for these unaligned search results as well? Since the browser is working, I’d assume that it isn’t an error and that I just haven’t added that functionality yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s