arraysearch

ARACNE results

Sat, 26 Dec 2009 03:31:26 GMT

This is the gene regulatory network ARACNE inferred from the top most 50 expressive genes (as measured by their L2 norms across the sample space):

Click for larger image

Gene 25127_at appears to be a major regulatory gene, so I will check and see what else is known about it.

I also have generated network adjacency matrices for the top 200 and the top 1000 genes as ranked by their respective L2 values, but the network graphs are pretty unwieldy to work with. This graph was generated using Cytoscape - which is a very cool program with a lot of plug-ins, and I am still working my way around it.

More ARACNE results to be forthcoming. The program also comes with Matlab scripts to estimate optimal parameters for a given dataset, and I will run these the next time.

ARACNE

Wed, 23 Dec 2009 23:17:24 GMT

Deleted gene 245127_at (line 3640) from the dataset. ARACNE cannot run if there are missing values. And for some reason across more than half the experimental samples, no expression values exist for this gene. Weird.

Filtering issues of the atgenexpress dataset

Wed, 23 Dec 2009 10:17:25 GMT

Because the atgenexpress dataset is so huge, I have been trying to filter out genes which are not expressed across any of the samples. To this end, I have made use of the mas5calls function in the affy library of R which you get when you install the Bioconductor package. This function performs the Wilcoxon signed rank-based gene expression presence/absence detection algorithm on the microarray data. So in addition to ending up with an Excel file containing gene expression data (the rows are the genes and the columns are the experimental samples), you end up with another Excel spreadsheet of the same size, where in place of each expression value, there is now either the letter P, M, or A. P = present, M= marginal, A = absent (indicating whether the gene was detected or not in the microarray).

My first thought was to use this as a way to filter out genes. By summing the Ps, Ms, and As along each row (gene), I formed the ratio: R = P/(P+M+A). Then sorting the genes from largest to smallest by this value would give me the genes which were detected the most frequently. Genes which have R = 0 I am obviously not interested, so I deleted these from consideration.

There are other factors to consider though. For one, a gene which is highly expressed in one or two samples and could play an important role in the the genetic response of Arabidobsis to a stimulus could be important as well - but have a low R value. If I just filter by R-value, I am only catching those genes which are highly expressed across all experiments. I need to consider the standard deviation of each gene.

I am also interested in the L2 norm value of each gene. Viewing each gene as a vector containing expression values for each sample, I calculate the L2 norm. This is also a way to rank genes.

So there are in fact three ways to rank genes in terms of importance (by L2 norms, standard deviations, and their R values). I am working on a way to link them together into one metric. Below are scatterplots of the various interactions.

Click for larger image

Conclusions:

The linear relationship which appears to exist between the L2 norm and R was kind of unexpected. The mean of all the L2 norms for the genes is 208.5677, and so the concentration of genes near the lower left-end of this graph also share low R values. Higher R values also appears to suggest higher L2 values.
There are a few genes which have L2 values above the mean and R values below 0.1 (suggesting genes expressed highly for a small number of samples), but only a very small number. In fact I think I could be safe in deleting all genes with an R value less than 0.5.

Click for larger image

Conclusions:

The mean of all standard deviations is: 1.1074, and this plot seems to bear that out. What is interesting about this one is its similarity to the previous plot. There appear to be two large groupings of genes in the dataset. Ones with high R values and ones with very low R values, and a few genes randomly dispersed between these two groups.

Click for larger image

Conclusions:

This is the most interesting plot. As I mentioned previously, the mean of the standard deviaitons is 1.1074, and the mean of the L2 norms is 208.5677. There appears to be a large clustering of genes which are highly expressed across a few experiments and have a standard deviation which falls below the mean for all genes.

Below are the histograms for the three metrics I have looked at:

Click for larger image

Click for larger image

Click for larger image

Future Course of Action: I will be posting the results of running ARACNE shortly (within a day or so).

Preliminary results from atgenexpress dataset

Thu, 17 Dec 2009 16:09:42 GMT

Click for larger image

Click for larger image

Link to spreadsheet of top 500 most active genes across the sample space (as determined by the l2 norm

And here are graphical plots of the heatmaps for the metagenes obtained through the NMF algorithm. I will post a list of genes for each metagene when I am finished finding the optimal dimension for NMF for this dataset. k=50 worked well for the previous dataset, but there are many more samples in this set and there appears to be a lot of overlap between genes for that value.

Click for larger image

Click for larger image

Arraysearch bugs which need to be fixed

Mon, 05 Oct 2009 09:02:33 GMT

A user should be able to enter a list of genes in the first search box, and the results displayed should be the same as when they search by gene id and fold value, except that in place of the Pearson coefficient, the p-value should be displayed. Also - the search results should be sorted by p-value.

Status: Solved this weekend. I created an approximation of the incomplete irregularized beta function (which you can then use to calculate the p-value for the t-distribution given the degrees of freedom of the data set). I used the approximation found in Numerical Recipes in C (1992).
Additionally, this approximation also uses another approximation (that of the natural log of the gamma function), and I used a modified version of the famous Sterling approximation to solve this (based on a suggestion by Robert H. Windschitl suggested it in 2002 as a formula convenient for inputting into programmable calculators because it was fast and did not consume a lot of memory. Scroll down to the bottom for his formula). I tested some of the results against the t-test in R and the results seemed ok. I would like to do a more advanced plot of errors on data with different degrees of freedom, however.
The default search for the first box (genes + fold values) - should input the fold values in log2.
Search by Gene Title (the third search) takes a really long time to filter search results and display an answer. I need to figure out what is up with that.
Related to this - find a way to speed up all SQL queries in general (if there is a way)
Fix the heatmap - by overlaying the heatmap with a clickable image map.
Based upon the size of the result set - the site should return either a heatmap or a bargraph.

Status: This has been partially solved. For moderately large results sets - a heatmap will be returned. Unfortunately - for really large results sets (like 100 genes or so) - this will kill the webserver and the script will time out. It would be nice to have a zoomable heatmap - but I'm not sure how easy it will be to implement given the constraints of the hardware we are currently running the site on. And of course - if you get more than two people hitting the site at the same time and trying to generate their own heatmap, it will totally die. I might be able to implement this functionality in PHP and then put it on the site if we ever go to a fully dedicated server. From my Googling searches, I do not believe it has ever been done before in PHP (and maybe there is a good reason for that). What we need is really fast image generating code (perhaps in Python?) which I can run on the server, and then use PHP to deliver the images client-side. Pretty sure downtown hosting will not let me run my own C-code to generate images on their server (even if we have a dedicated server). But they do have python installed.

Updates for September 9, 2009

Fri, 11 Sep 2009 00:24:17 GMT

I have been going over the PHP code for the website and cleaning it up. The sections of code for pChart which I hacked to create the heatmap and bargraph plots are really ugly, and I will have to tackle them next. Anyway - Dr. Ge gave me a to-do list of things to finish as soon as possible.

Instead of 'Searching by Gene Fold Value', the title for this search should be 'Search for Similar Expression Profiles'
Widen the text area for the searches (especially the first one)
Input list of genes and fold values in log2 scale
Display graphical as well as textual results in this first search (ie. show a heatmap/bargraph)
Currently we are sorting by maximum positive Pearson coefficient. Should also return correlations whith high negative correlations.
The query can could also be just a list of genes. Use t-test to return significant contrasts.
Dynamically switch between bargraph and heatmap.
search_ontology.php : This page looks messed up in IE even though it looks great in Firefox. Find out why.

Additionally, I am thinking strongly about moving this journal over to wikidot (the support for latex code being a prime motivating factor).
Note to self: Set up office hours as soon as possible.

LJ Developer Blog created

Fri, 05 Jun 2009 07:13:49 GMT

This is the new development blog for the Arraysearch research project at South Dakota State University.   Primarily I intend to use this site to document bugs in the source code for the arraysearch website, as well as discuss where I am currently in developing new functions and algorithms.

Yesterday: I changed the DNS nameservers at GoDaddy so that now they point to Downtown Hosting.

http://www.arraysearch.org now redirects to http://zanshin3d.net/arraysearch/arraysearch.php

Currently working on this function:

function getExpressionMatrix($db,$query){

    $expressions = array();

    $contrasts = array();

    $dtls = ExecuteQuery($db,$query);

    $getColumns = "SHOW columns FROM tbl_signature";
    $cNames = ExecuteQuery($db,$getColumns);
    $i=0;

    while($row = mysql_fetch_array($cNames,MYSQL_BOTH)){
        $contrasts[$i]=$row[0];
        $i++;
     }//end while


    $numCols = count($contrasts);
    $rowNum = 0;

    while($row = mysql_fetch_array($dtls,MYSQL_ASSOC)){

     for($colNum = 1;$colNum<count($contrasts);$colNum++){

      $expressions[$rowNum][$colNum]=$row[$contrasts[$colNum]];

      }
      $rowNum++;

     }

     $numRows = $rowNum;

    return array($expressions,$numRows,$numCols);
}

The matrix returned from this function is sometime HUGE - and I need to limit it's potential size - while allowing the user to navigate through the data quickly and easily.