Todd Harris

Ascaris suum draft genome published

tharris — Wed, 26 Oct 2011 19:41:20 +0000

A draft assembly of the 273 MB Ascaris suum genome has been published in Nature. A. suum is a model for human ascaris infection via the common round worm.

Running the Generic Genome Browser under PSGI/Plack

tharris — Sun, 11 Sep 2011 19:17:09 +0000

Here’s a simple approach for installing and running a local instance of GBrowse, leveraging the PSGI/Plack webserver <-> web application stack. You don’t need root access, you don’t need Apache, and you don’t need to request any firewall exceptions (for now).

Background

Both the current implementation and installer of GBrowse are loosely tied to Apache. By loosely, I mean that the installer generates suitable configuration and assumes installation paths as if the instance will be run under Apache. The implementation is tightly tied to the CGI specification; it’s a suite of CGI scripts. Although GBrowse will rununder any webserver that implements the CGI specification (are there any that DON’T?), this approach increases the administrative effort required for running a local instance, increases the complexity of configuration, makes it more difficult to run GBrowse under other environments, and makes it impossible to leverage powerful advances in Perl web application development.

Enter PSGI (the Perl Web Server Gateway Interface), a specification for glueing Perl applications to webservers. Plack is a reference implementation of this specification. PSGI as implemented by Plack makes it simple to run Perl-based applications (even CGI-based ones like GBrowse) in a variety of environments.

In other words, PSGI abstracts the request/response cycle so that you can focus on your application. Running your application under CGI, Fast CGI, or mod_perl is just a matter of changing the application handler. The core Plack distribution provides a number of handlers out of the box (CGI, FCGI, mod_perl, for example) and even includes a light-weight webserver (HTTP::Server::PSGI) which is perfect for development. Other webservers also implement the PSGI specification, including the high-performance preforking server Starman.

You can also do cool things via middleware handlers like mapping multiple applications to different URLs with ease (how about running the last 10 versions of GBrowse all without touching Apache config or dealing with library conflicts), handle tasks like serving static files, mangling requests and responses, etc.

What this isn’t (yet)

This isn’t a rewrite of GBrowse using PSGI. It’s just some modifications to the current GBrowse to make it possible to wrap the CGI components so that they can be used via servers that implement the PSGI specification. There is a project to rewrite GBrowse as a pure PSGI app. Stay tuned for details.

Conventions

Installation root.

Our working installation root is configured via the environment variable GBROWSE_ROOT.

No root privileges required.

You do not need to be root. Ever. In fact, one of the great advantages of this approach is the ease with which you can install a local instance.

Self-contained, versioned installation paths.

This tutorial installs everything under a single directory for simplified management and configuration. This path corresponds to the version of GBrowse being installed.

The current version of GBrowse is specified by environment variable (GBROWSE_VERSION). If you want to use the same installation path from release to release, you can also create and adjust symlinks as necessary (~/gbrowse/current -> ~/gbrowse/gbrowse-2.40, for example, and set GBROWSE_VERSION=current). This isn’t necessarily required but means that you won’t need to set GBROWSE_VERSION every time you update to a new version of GBrowse. At any rate, maintaining installations by version is a Good Practice and makes it easy to revert to older versions should the need arise.

Each installation has it’s own set of local libraries.

In keeping with the self-contained non-privileged design gestalt, we’ll install all required libraries to a local path tied to the installed version of GBrowse ($GBROWSE_ROOT/$GBROWSE_VERSION/extlib). This makes it dead simple to run many possibly conflicting variants of GBrowse all with their own dedicated suite of libraries. Awesome.

Installation

Set up your environment.

  // Set an environment variables for the your installation root and the version of GBrowse you are installing.
  > export GBROWSE_ROOT=~/gbrowse
  > export GBROWSE_VERSION=2.40

Prepare your library directory.

  // You may need to install the local::lib library first
  > (sudo) perl -MCPAN -e 'install local::lib'
  > cd ${GBROWSE_ROOT}
  > mkdir ${GBROWSE_VERSION}
  > cd ${GBROWSE_VERSION}
  > mkdir extlib ; cd extlib
  > perl -Mlocal::lib=./
  > eval $(perl -Mlocal::lib=./)

Check out GBrowse fork with modifications for running under PSGI/Plack.

  > cd ${GBROWSE_ROOT}
  > mkdir src ; cd src
  > git clone git@github.com:tharris/GBrowse-PSGI.git
  > cd GBrowse-PSGI
  # Here, the wwwuser is YOU, not the Apache user.
  > perl Build.PL --conf         ${GBROWSE_ROOT}/${GBROWSE_VERSION}/conf \
                  --htdocs       ${GBROWSE_ROOT}/${GBROWSE_VERSION}/html \
                  --cgibin       ${GBROWSE_ROOT}/${GBROWSE_VERSION}/cgi \
                  --wwwuser      $LOGNAME \
                  --tmp          ${GBROWSE_ROOT}/${GBROWSE_VERSION}/tmp \
                  --persistent   ${GBROWSE_ROOT}/${GBROWSE_VERSION}/tmp/persistent \
                  --databases    ${GBROWSE_ROOT}/${GBROWSE_VERSION}/databases \
                  --installconf  n \
                  --installetc   n
  > ./Build installdeps   # Be sure to install all components of the Plack stack:

      Plack
      Plack::App::CGIBin
      Plack::App::WrapCGI
      Plack::Builder
      Plack::Middleware::ReverseProxy
      Plack::Middleware::Debug
      CGI::Emulate::PSGI
      CGI::Compile

  // Should you need to adjust any values, run
  > ./Build.PL reconfig
  > ./Build install

Note: the curent installer script SHOULD NOT require a root password if using local paths like this example. When it asks if you want to restart Apache, select NO. It’s not relevant for us.

Fire up a Plack server using plackup.

The Build script will have installed a suitable .psgi file at conf/GBrowse.psgi. Launch a simple plack HTTP server via:

   > plackup -p 9001 ${GBROWSE_ROOT}/${GBROWSE_VERSION}/conf/GBrowse.psgi
   // Open http://localhost:9001/

Note: By default, plackup will use HTTP::Server::PSGI.

Where To From Here

PSGI/Plack is really powerful. Here are some examples that take advantage of configuration already in the conf/GBrowse.psgi file.

Enable the Plack debugging middleware:

   > export GBROWSE_DEVELOPMENT=true
   > plackup -p 9001 ${GBROWSE_ROOT}/${GBROWSE_VERSION}/conf/GBrowse.psgi
   // Visit http://localhost:9001/ and see all the handy debugging information.

Run GBrowse under the preforking, lightweight HTTP server Starman:

   > perl -MCPAN -e 'install Starman'
   > starman -p 9001 ${GBROWSE_ROOT}/${GBROWSE_VERSION}/conf/GBrowse.psgi

An introduction to cloud computing for biologists (aka the 10-minute model organism database installation)

tharris — Thu, 11 Aug 2011 19:39:48 +0000

This tutorial will explain the basic concepts of cloud computing and get you up and running in minutes. No knowledge of system administration or programming is necessary. As an example, it describes how to launch your own instance of the model organism database WormBase.

Introduction to cloud computing

If you aren’t familiar with cloud computing here’s all you need to know. At its simplest, cloud computing refers to using remote compute resources over the network as if they were a computer sitting on your desktop. These services are typically virtualized and used in an on-demand fashion.

Several vendors provide cloud computing options. Here, we’ll focus on
Amazon’s Elastic Compute Cloud (EC2).

On EC2, developers can create Amazon Machine Images (AMIs) which are essentially snapshots of a full computer system. For example, the WormBase AMI contains everything necessary to run WormBase — all software and databases with the operating system preconfigured.

Booting up an image is referred to launching an “instance”. When you do so, you choose the size of the server to allocate (for example, how many cores and how much RAM) to run the instance with. You can start, stop, or reboot the instance at any time. Terminating the instance completely removes it from your account. The original reference AMI remains; you can launch a new instance from it any time. This is what Amazon means by elastic. You can provision and decommission new servers with custom capacity in minutes mitigating overhead costs like data centers, surly IT departments, and draconian firewall regulations.

Amazon’s EC2 service is a “pay-for-what-you-use” service; running an instance is not free. You are charged nominal rates for 1) the size of the instance allocated; 2) the amount of disk space the instance requires even if it isn’t running; 3) the amount of bandwidth the instance consumes; 4) how long the instance is running.

A complicated model organism database like WormBase typically require a “large” instance (see below). Running 24/7, the estimated cost would be approximately $2700/year. Costs can be mitigated by starting and stopping the instance when needed, pausing the instance in its current state. This is conceptually similar to puting a desktop computer to sleep. Alternatively, if you aren’t modifying the data on the server, you can safely terminate it when you are done, avoiding disk use charges, too. Simply launch a new instance from the original WormBase AMI. Launching from an AMI requires slightly more time (several minutes) than restarting a stopped instance (< minute). Requesting a dedicated instance in advance from Amazon further reduces the cost by approximately 30%.

caveat emptor: these are back-of-the-napkin calculations. Costs can vary dramatically especially if you start making many, many requests to the website. Bandwidth charges for accessing the website are nominal.

Example: Personal Instances of WormBase through Amazon’s EC2

In the past running a private instance of WormBase has been a time-consuming process requiring substantive computer science acumen.

Today I’m happy to announce WormBase Amazon Machine Images (wAMIs, pronounced “whammys”) for Amazon’s Elastic Compute Cloud (EC2). The WormBase AMI makes it absolutely trivial to run your own private version of WormBase.

Running your own instance gives you:
* Dedicated resources
* A feature-rich data mining platform
* Privacy

Contents of the WormBase AMI

* The WS226 (and beyond) version of the database
* The (mostly) full WormBase website
* The Genome Browser with 10 species
* A wealth of pre-installed libraries for data mining (to be covered in a subsequent post)

The first WormBase AMI is missing a few features:
* WormMart
* BLAST

Launching your own instance of WormBase

Here’s a really bad screen cast. You might want to read through the rest of the tutorials for details.

View the screencast in full size.

The general steps for launching an instance of a new AMI are as follows. Note that in the management console it is possible to execute many of these steps during the process of launching any one specific instance, too.

1. Sign up for an Amazon Web Services account

See up for an account at aws.amazon.com. You’ll need a credit card.

2. Create a keypair

Note: You can also complete this step when you launch your instance if you prefer.

When you launch an instance Amazon needs to ensure that you are who you say you are (read: that you have the ability to pay for the resources that you consume), as well as give you a mechanism for logging into the server. This authentication process is handled through the use of secret keys. Even if you only intend to use the web interface of WormBase and not log in directly to the server, you will still need to generate a keypair.

To do this, log in to your Amazon AWS account and click on the EC2 tab. In the left hand pane, click on “Keypairs”. You’ll see a small button labeled “Create Keypair”. Click, and create a new kaypair. You can name it whatever you like. When you click continue a file will be downloaded to your computer. You will need this file if you intend to log on to the server. Store it in a safe place as others can launch services using your account if they get access to this file!

3. Configure a new security group

Note: You can also complete this step when you launch your instance if you prefer.

Security groups are a list of firewall rules for what types of requests your instances respond to. They can be standard services on standard ports (HTTP on port 80) or custom, and they can range from allowing the entire internet to a single IP address. They are a quick way to lock down who gets to use your instance. For now, we’ll create a security group that is very permissive.

Click “Create new group”, give the group a name and description. From the dropdown, select “HTTP”. Click Add Rule. Repeat, this time selecting SSH. Although not required, enabling SSH will allow us to actually log into the server to perform administrative or diagnostic tasks. Click Add Rule, then Save.

4. Find and launch an instance of the WormBase (WS226) AMI

Now we’re ready to launch our own instance. See the video tutorial for description.

5. Get the public DNS entry for your new instance

Your new instance is elastic; it gets a new IP address every time it is launched (although Amazon has services that let it retain a static address, too). You need to get the hostname so that you can connect to the server. Click on “Instances”, select the running instance, and in the bottom pane, find the “Public DNS” entry. Copy this entry, open a new tab in your browser and paste in the URI. It will look something like this:

ec2-50-17-41-111.compute-1.amazonaws.com

6. Stopping your instance

When you are done with your instance, shut it down by going to the EC2 tab > Instances. Select the instance and from other the “Instance Action” drop down or by right clicking, select “Stop”. You’re instance will be paused where you are. Repeat these steps selecting “start” to restart it. Note: you will continue to accumulate charges associated with disk storage while the instance is stopped, but will not incur compute charges. Alternatively, you can choose to “terminate” the instance. Once you do so, be sure to visit the “Volumes” and select the EBS volume that had been attached to the instance — it will be 150GB in size. It will cost about $7/month to save this volume.

In a subsequent tutorial, I’ll show you how to go beyond the web browser to use the powerful command line data mining tools packaged with every WormBase AMI.

Questions? Contact me at todd@wormbase.org.

Happy belated birthday, Mendel!

tharris — Thu, 21 Jul 2011 17:00:12 +0000

Photoshop art from my old grad school days.

Debugging xinetd configuration problems

tharris — Sun, 19 Jun 2011 13:54:14 +0000

xinetd is great when it’s working but can be a complete pain to debug when things go wrong. As a start, try launching it in the foreground in debugging mode:

   /usr/sbin/xinetd -d -dontfork

GitHub’s “Organizations” for distributed #bioinformatics dev; migrating from Mercurial

tharris — Sat, 12 Feb 2011 16:57:41 +0000

GitHub.com’s “Organizations” is a great tool for distributed bioinformatics teams. Here’s how I migrated some of our repositories from Mercurial to Git to take advantage of this feature

After much evangelizing, weeping, and wailing, I finally convinced everyone at one highly geographically and functionally distributed projects that we should at least try consolidating our code in one place.

Currently we have old legacy repositories in CVS, mid-range projects in SVN, new development in Git and Mercurial, and AFAIK a bunch of code in no SCM system at all.

Given that DVCS doesn’t have the directory level granularity of SVN, we definitely don’t want to consolidate everything in a single repository. So far, it seems that GitHub offers the best solution with its “Organizations” feature. This lets a team group multiple repositories under a single umbrella with a shared news feed and administration. Perfect.

hg-git looks like a useful tool if you want to maintain code in both git and mercurial. I don’t. Here’s how I handled a full-scale migration of our repositories:

todd> cd ~/projects
todd> git clone http://repo.or.cz/r/fast-export.git
todd> mkdir new_git_repository ; cd new_git_repository
todd> git init
todd> ../fast-export/hg-fast-export.sh -r ~/projects/old_hg_repository
todd> git checkout HEAD
todd> git remote add origin git@github.com:[organization]/[reponame].git
todd> git push origin master

Bing! And you’re done. Or whatever.

Amazon Elastic Block Store for facile sharing and archiving of biological data

tharris — Thu, 10 Feb 2011 18:09:21 +0000

Amazon’s Web Services offers enormous potential for people who need to process, store, and share large amounts of data.

And it’s a huge boon for bioinformatics. It’s cost effective and it’s fasta. Hah. Get it? It’s “>fasta”. Archiving and sharing data has never been easier.

Here’s a quick tutorial on creating an Elastic Block Store volume that you can share with your colleagues.

1. Create a volume

From the AWS Management Console, click on the EC2 tab, then on “Elastic Block Store > Volumes”
Click on “Create Volume”.
Pick an appropriate size for your volume. For EBS volumes that I am going to use to store and archive data, I create a volume 1.5 times the size of the data. This lets me store an unpacked version and a packed version simultaneously, making it easy to update data at a later date.
Add some informative tags.

2. Attach the volume to an EC2 instance.

From the Volumes window in the Management Console, select the new volume, then right click and Select “Attach”. I attach devices starting at

3. Format the volume.

Once you’ve created and mounted a volume, you’ll need to attach it to an EC2 instance. Fire one up and SSH in.

ssh -i @yourdns.amazonaws.com
> sudo mkfs.ext3 /dev/sdf

Mount points are available at /dev/sdf through /dev/sdp.

4. Mount the volume

> sudo mkdir /mnt/data
> sudo mount -t ext3 /dev/sdf /mnt/data

If you are potentially going to be dealing with many versions of data overtime, you might want to version your mount points. This will allow you to attach multiple EBS volumes at different sensible directories:

> sudo mkdir /mnt/data-v0.2
> sudo mount -t ext3 /dev/sdf /mnt/data-v0.2

Alternatively, you might consider handle versioning when creating snapshots of your volume.

5. Set the EBS volume to mount automatically (optional)

> sudo emacs /etc/fstab
/dev/sdh /mnt/data ext3 defaults 0 0

And you’re done! Now what?

Throw some data on there. Do some computes. Go nuts.

Share your data

Sharing your data is as easy as creating a snapshot.

1. Create a snapshot

Power down your instance. From the Management interface, select the volume and choose “Create Snapshot”.

Tips for effective data archiving and sharing

1. Add informative tags.

Be sure to add informative tags such as the release date and version of the data.

Release Date = 02 Jan 2011
Source = Todd’s Data Emporium
Contact = data@tharris.org

2. Include informative READMEs on the volume itself.

3. Be sure to make the snapshot public!

Updating your data

Updating your data to the next release of your resource is simple. Mount the original volume to an instance, copy in new data, then create a new snapshot.

Hide ‘n Seek: What to do with empty data fields?

tharris — Tue, 08 Feb 2011 19:45:05 +0000

We’ve been working on a fundamental website redesign for a hefty biological database.

One design dilemma has been what to do with empty data fields. For example, on a Gene Summary we might have a “Variation” field listing variations found in the gene. Obviously, not all genes have variations.

Displaying field labels with empty contents clearly delineates the limits of our knowledge or curation, but at the same time leads to more visually confusing pages.

Current options we’re considering are:

1. Omit the field entirely.

Known unknowns (apologies to D. Rumsfeld), if you don’t know what you might know, you don’t know how much you do know. Or something like that.

2. Display the field label, but with empty contents.

Variations:

3. Display the field label with a string:

Variations: no data available

This offers the same advantage as above, namely that gaps in our knowledge or curation are clearly indicated. But sparse entries become visually thick very fast.

We’re currently experimenting with other design patterns for handling this situation, too, including using color to de-emphasize empty fields or allowing users to turn off their display as a configuration option.

What do you prefer? Would you rather see all available data fields on a report page even if they’re empty? Or are you a minimalist and prefer that empty field be hidden?

On my way to Science Online ’11. Biological databases, represent! #scio11

tharris — Thu, 13 Jan 2011 14:50:08 +0000

Another early morning for the 6:05AM from Bozeman. 4 AM doesn’t feel so bad when the stars are shining and its 30° F outside.

Today I’m on my way to the Science Online ’11 meeting — in fact, I’m posting this in the air between Bozeman and Minneapolis. This is the first year I’ve been able to attend, having been stymied by conflicting advisory board meetings the past two years.

Humbly joining luminaries from science writing and blogging, my motivation for attending is a bit different. I’m most interested in exploring how we can make use of online tools and communities to make the process of science more transparent to other scientists, more accesible to the public, and in general, easier and more efficient.

Publicly-accessible web-based databases have become an essential component of daily research in biomedical sciences. I’m the project manager and lead developer of one such database. We know from user surveys that a vast majority of our users visit the site every day. Most databases — including ours — are referential in nature. You log on, look something up, and log off. But these resources could be so much more than that. We owe it ourselves to look at success cases in other fields to make these websites more interactive and useful.

At the moment, we are currently in the middle of a ground up rewrite of our site. Inspired by the rise of web 2.0 social media and networking, we’re building a number of new tools into the site not commonly found on biological websites.

For example, can we glean biologically meaningful information from the browsing patterns of users? I’ve tried to do this a number of times in the past using log file analysis with no limited success. In our new site, we’ve built a tool that does this in real-time to collect the most popular objects. When correlated with unique users, we can also use this as an Amazon-style suggest feature (“Users interested in this gene were also interested in gene Y”). We’ve extended this concept to a common “favorite this” design pattern to make possible matches even more relevant.

Features like this that revolve around community intelligence pose interesting questions for privacy and transparency. One approach that we are considering is to only tally and only present results to users who have specifically opted in.

Well, we’re descending below 10K feet. Time to post.

Volume 18, Number 3 of the Worm Breeder’s Gazette now available

tharris — Sun, 26 Dec 2010 20:55:57 +0000

Volume 18, Number 3 of the resurrected, open access research newsletter of the Caenorhabditis elegans research field is now available. Go get it while the gettin’s good!

The next issue of the Gazette will be release in June 2011, just prior to the 18th International Worm Meeting. You can submit articles now online at the Worm Breeder’s Gazette. The deadline for submissions is June 1, 2011.