The $24 Poor Man’s Social Media Expression Pattern Database (PoMaSoMeExpPaDa)

Expression pattern images are some of the most information-rich data housed at model organism databases. They are time consuming to generate. They are time consuming to collect and annotate.

Moreover, copyright restrictions mean that many images remain captive at publisher’s websites, unable to be placed within the rich intellectual framework that exists at sites like WormBase and FlyBase. How many near identical images are stashed away in darkened confocal rooms? How many possibly informative rejects are tossed out due to the puny limitations of publication? Gabijillions?

I wanted to build an easy to use expression pattern image resource that got around these limitations. The system would allow people to add their own photos for display within a broader intellectual context, comment on photos, add tags, search for a variety of criteria, etc. The problem? Developer cycles. This is a lower than low priority project and there aren’t enough hands to go around as it is.

I started wondering if I could leverage a site like Flickr to create a Poor Man’s Expression Pattern Database. Flickr is a key exemplar for Web 2.0 community style features. Tags, contacts, comments, an API.

The images

I took approximately 6000 public, highly curated expression pattern images from WormBase. We display these on Expression Pattern Summary pages.

Uploading images

I wrote a script exploiting Flickr’s REST-like API to programmatically upload images.

For each image, the script added a text description of the expression pattern with hyperlinks back to WormBase genes, anatomy ontology terms, gene ontology terms, strains, transgenes, etc. Images were posted to a dedicated user named, ahem, wormbase.

Tags were added to each image corresponding to the unique gene ID, public gene names, and anatomy ontology terms.

Here’s an example image on Flickr.

Integration with WormBase

I wasn’t happy with the current Perl interfaces to the Flickr API so I wrote my own (Flickr::API::Simple; note that I haven’t released this to CPAN yet and probably never will).

To pull the correct images, tags, and comments from Flickr, individual expression pattern pages levy a query for images at Flickr from the wormbase user with tags corresponding to either the expression pattern ID or the current gene being displayed. Information is displayed inline on the page but served from Flickr.

Posts from the community

Since the goal of this project is to allow *anyone* with images to post them for display at WormBase, we needed a open account or group to do that. But since people are nuts, I wanted to constrain it a bit to people who I knew. Attribution is critical. To do this, I created an invite-only group on Flickr called, erm, WormBase.

If a user has an image that they would like to share on the WormBase site proper, all they need to do is:

* Upload the image to their account
* Post the image to the WormBase group on Flickr
* Tag the image with the unique gene ID

These images will automatically be displayed on WormBase Expression Pattern pages using the exact mechanism as above: Expression Pattern pages search Flickr for images belonging to the WormBase group (instead of user), tagged with the current gene.

Summary

That’s it! A Poor Man’s Expression Pattern database with integration and cross links to a public genomics repository.

We get tagging, searching (clustered tag analysis), social features like commenting and blog integration for (nearly) free. We don’t have to spend six months time in development.

Cost: $24 bucks a year for a Flickr Pro account. This gives 24 GB of storage. Ridiculous. No electricty costs. No sysadmin. No maintenance. $24 dollars or 6 pints.
Time: about 2 hours of programming time to figure out the Flickr REST-like API. About 2 days of running time to upload images (I’m on a slow link).

Share on FriendFeed | Comments | TrackBack URI
  • September 17, 2008 at 10:09 am Cameron Neylon
    Wonderful! Of course, you know what I'm going to ask for next don't you...?
  • September 17, 2008 at 11:07 am Bill Hooker
    Brilliant!
  • September 17, 2008 at 5:17 pm Todd Harris
    Thanks all! This was a fun little diversion. I'm curious to see if anyone actually *adds* any images...
  • September 17, 2008 at 5:49 pm Neil Saunders
    This is great. Makes you wonder if we could build quite complex data management systems using APIs to existing web services (Flickr, various Google tools, FriendFeed...) and aggregating to a portal.
  • September 17, 2008 at 6:05 pm Ricardo Vidal
    This looks so cool and simple that I can't really grasp why nobody hasn't done this before. Not sure what Flickr's TOS says about this but it's quite brilliant.
  • September 17, 2008 at 6:48 pm Todd Harris
    Cameron: happy to contribute! Ricardo: I think as long as images are linked back to Flickr, this is within the TOS. We'll see the next time someone decides to LWP::Simple their way through our expression patterns!
  • September 17, 2008 at 6:58 pm Todd Harris
    Neil: I've been working along the exact same lines, building resources for less well-characterized (and less well-funded) organisms. You need a foundation upon which to layer third party services. I'm using a genome browser with rudimentary annotations: gene models with stable identifiers. It's a great leg up if you have limited resources and presents some intriguing mashup possibilities for larger operations, too.
  • September 18, 2008 at 1:22 am Cameron Neylon
    The question I was going to ask was about the license - could it be CC-BY by default? But yes, using Flickr for research image management is really a no brainer - it would be great to wire up something more complex and automated. To a certain extent this is what Jean-Claude already does but withou automated aggregation.
  • September 18, 2008 at 5:03 am Todd Harris
    Ah, I see. I got my 2.0 communications lines crossed. I tend to think the licensing should be left up to the contributor. I'm not familiar with Jean-Claude's work. Linky?
  • September 18, 2008 at 9:14 am Bill Hooker
  • September 19, 2008 at 5:53 am Marcin
    Great idea

Add a comment on FriendFeed




Does Nobel Prize == quality teacher?

| , ,

In a speech to the NAACP, John McCain wrote:

Many thousands of highly qualified men and women have great knowledge, wisdom, and experience to offer public school students. But a monopoly on teacher certification prevents them from getting that chance. You can be a Nobel Laureate and not qualify to teach in most public schools today. They don’t have all the proper credits in educational ‘theory’ or ‘methodology’ — all they have is learning and the desire and ability to share it.

First, how many Nobel Prize winners would want to teach in America’s public schools. I don’t think he realizes that this would be a serious pay cut for most Nobel Laureates.

Second, there is much more to be an effective teacher than simply having “knowledge, wisdom, and experience.” Nobel laureates are usually good for ribald seminars but I’m not certain I’d want them teaching my kids. And I don’t even have kids ;)

Share on FriendFeed | Comments | TrackBack URI

Add a comment on FriendFeed




Implementing a simple web-log based recommender system

I’ve now implemented such a system as an extension to Catalyst, the open source Perl web framework. The system isn’t yet ready for general distribution, but I’d like to share my approach.

First, I’ve gathered ten years of web access logs from WormBase, a generic model organism database where I work as the project manager.

Next, I correlated IP addresses with requests and tried to trace browsing patterns from one object to the next. This isn’t an exact science since we haven’t historically tried to uniquely identify users.

Data is loaded into a simple MySQL schema with object and object2related tables. Expediently simple.

Share on FriendFeed | Comments | TrackBack URI

Recommender systems for biological databases

Recommender systems [Wikipedia] seek to provide users with information related to what they are currently browsing. These are now ubiquitous in e-commerce sites such as Amazon, where each page contains a list of items viewed or purchased by other users.

I’ve long felt that a recommender system could revolutionize the browsing and mining of biological data. The idea would be to provide users with a list of related objects based on browsing history of cadres. See this post for some preliminary implementation notes.

I am worried that a recommender system won’t be received with open arms. Given that my current implementation is based on web log analysis it presents serious privacy issues. It’s conceivable that it’s use could inadvertently reveal the identity of uncloned and unpublished loci.

Share on FriendFeed | Comments | TrackBack URI

GUI prototyping add-on

| , , ,

Check out the GUI prototyping add-on Pencil Project for Firefox.

Share on FriendFeed | Comments | TrackBack URI

Bizarro Perl problem

|

I just came across a fairly strange Perl issue:

Consider:

protein => {
title => '%s release [%s]‘,
qq{formatdb_cmd => ‘-p T -t ‘%s’ -i %s’},
filename => ‘protein.fa’,
},

Yes, there’s a typo, with the qq enclosing an entire hash entry. This doesn’t raise an error, but obviously obscures the hash entry!

Share on FriendFeed | Comments | TrackBack URI

Bioinformatics is dead! Long live bioinformatics!

| , ,

A few years ago, my boss Lincoln Stein prognosticated the end of bioinformatics.

According to Google Trends he just might be onto something. Searches for bioinformatics have dropped in half in the past four years. C’est incroyable! Not surprisingly, the top-ten sites for searches for bioinformatics are all in India.

Gory details available by searching Google Trends for bioinformatics.

Interesting parallels in the trends between bioinformatics, molecular biology, proteomics, and C. elegans:

Share on FriendFeed | Comments | TrackBack URI

Installing Trac

| , , ,

We’re already using 37 signals Basecamp for project management. It works well for collaborative work and management of distributed teams but is tedious when used as a feature tracker.

Today we discussed the need for a bug/issue tracker. In the past, we’ve considered RT since it integrates with our existing email flow. But frankly, RT is a PITA to configure.

I know that some people are hot and heavy on Mantis. I’m not. For one, I don’t like it’s name or it’s logo.

Here’s how I installed Trac:


// install Python
cd ~/src
wget http://python.org/ftp/python/2.5.2/Python-2.5.2.tgz
cd ../build
tar xzf ../src/Python*
cd Python*
./configure; make ; sudo make install

// Install easy_install.py
cd ~/src
wget http://peak.telecommunity.com/dist/ez_setup.py
sudo ./ez_setup.py

// Install Trac
sudo easy_install Trac==0.11rc2

// Install mod_wsgi
cd ~/src
wget http://modwsgi.googlecode.com/files/mod_wsgi-2.0.tar.gz
tar xzf mod_wsgi*
cd mod_wsgi*
./configure
make
sudo make install

// Set up Trac environment
trac-admin /usr/local/wormbase/trac
... // follow the configuration prompts

// Test it
tracd --port 9001 /usr/local/wormbase/trac
http://localhost:9001/

// Configure wsgi
import os

os.environ['TRAC_ENV'] = ‘/usr/local/wormbase/trac’
os.environ['PYTHON_EGG_CACHE'] = ‘/usr/local/wormbase/trac/eggs’

import trac.web.main
application = trac.web.main.dispatch_request

// Configure apache

// apache conf
LoadModule wsgi_module libexec/mod_wsgi.so
AddModule mod_wsgi.c

// wormbase/conf/httpd.conf
WSGIScriptAlias /trac /usr/local/wormbase/cgi-perl/misc/trac.wsgi


WSGIApplicationGroup %{GLOBAL}
Order deny,allow
Allow from all


AuthType Basic
AuthName “WormBase Trac”
AuthUserFile /usr/local/wormbase/trac/.htpasswd
Require valid-user

Share on FriendFeed | Comments | TrackBack URI