A Short History of Quantitative Biology at Cold Spring Harbor Laboratory

In my spare time, I enjoy reading about the history of science, and since arriving at Cold Spring Harbor Laboratory last August, I have been particularly interested in the role of CSHL in the history of genetics and molecular biology.  I have been especially struck by the large number of prominent quantitative biologists who have been affiliated with the Laboratory, in one way or another, and I thought it would be useful to draft a short history focused on quantitative biology at CSHL.  Below is an initial attempt at this document.  I’d be interested in comments or corrections any history buffs out there might be able to offer.

Adam Siepel

A Short History of Quantitative Biology at Cold Spring Harbor Laboratory

While quantitative biology was not considered a distinct area of focus at CSHL until recently, quantitative methods have long been important in research at the Laboratory. Charles Davenport, who introduced genetics to the Laboratory in 1898 and founded the Station for Experimental Evolution in 1904, arrived with a strong appreciation for quantitative methods. Early in his career, before his prominent and controversial role in the American eugenics movement, Davenport was an enthusiastic supporter of the “biometric” approach to the study of heredity, pioneered by Francis Galton and Karl Pearson in England and characterized by its heavy reliance on newly emerging statistical techniques. For a time, Davenport was a co-editor of Pearson’s influential journal, Biometrika. He later split from the biometricians and became a supporter of Mendelian models of inheritance but remained enthusiastic about mathematical modeling in genetics. (The major divide between the biometricians and Mendelians would eventually be reconciled by the great statistical geneticist, R.A. Fisher.)

Sewall Wright

Sewall Wright

In the early to mid 1900s, the summer program was central to research at the Laboratory. One summer visitor was a young Sewall Wright, who would later become one of the founders of theoretical population genetics, with Fisher and J.B.S. Haldane. Wright spent rewarding and formative summers at the Laboratory as a student in 1911 and 1912. He enjoyed his time at Cold Spring Harbor so much that he returned in the summer of 1920, while on leave from a position at the USDA. As it happened, he met his future wife, Louise Williams (also a biologist) at the Laboratory that summer.

Reginald Harris

Reginald Harris

Davenport’s son-in-law, Reginald Harris, became director of the Biological Laboratory at Cold Spring Harbor in 1924. In 1928, Harris introduced a program in biophysics that was critical in establishing the Biological Laboratory as a true year-round research institution. A few years later, in 1934, Harris launched the first Cold Spring Harbor Symposium in Quantitative Biology. The Symposium went on to become a hugely influential forum for the dissemination of research across many areas of biology, but particularly in quantitative biology. Harris’s successor, Eric Ponder, continued to promote quantitative biology, focusing the Symposium on topics in biophysics through 1940.

Delbruck and Luria

Delbruck and Luria relaxing at Cold Spring Harbor

In the middle decades of the 20th century, biophysics gave way to genetics as the major focus of research at the Laboratory, but an undercurrent of quantitative thought remained. Perhaps the most influential CSHL-affiliated quantitative biologist of this time was Max Delbruck, a prominent physicist who had moved into biology in the late 1930s and became a key figure in the emergence of molecular biology. Delbruck was particularly important in stimulating interest in biology from other physical scientists, including Erwin Schrodinger, whose popular 1944 book “What is Life?” made an impression on both Watson and Crick, among many others. In 1945, Delbruck launched the Phage Course at Cold Spring Harbor Laboratory, which would continue until 1970 and was highly influential in training many pioneers in molecular biology (including a young Jim Watson), a number of whom were recruited from physics. A group led by Delbruck and two close associates who would later share the Nobel prize with him, Salvador Luria and Alfred Hershey, met regularly in the summer at Cold Spring Harbor Laboratory and became known as the Phage Group. Hershey moved his laboratory to CSHL in 1950, and soon afterward he and Martha Chase carried out the famous Hershey-Chase experiment, which helped establish that DNA was the molecule that carried genetic information.

While the best known research at the Laboratory during this era was focused on molecular genetics, there was one prominent evolutionist and population geneticist, Bruce Wallace, based at the Laboratory from 1947 through 1958. Among many other things, Wallace contributed to an influential CSHL Symposium on Population Genetics in 1955, which was attended by Wright, Dobzhansky, Kimura, Crow, and Mayr, among other luminaries of the field.


Claude Shannon


Barbara Burks

One interesting episode in this mid-century era concerned Claude Shannon, the father of Information Theory and one of the most influential applied mathematicians of the 20th century. It is a little known fact that Shannon’s Ph.D. thesis, submitted in 1940 to the Mathematics Department at MIT, was focused not on communication theory but on population genetics and was based in large part on work carried out at Cold Spring Harbor Laboratory. At the time, Shannon’s Ph.D. supervisor at MIT, the great inventor and engineer Vannevar Bush, was president of the Carnegie Institution of Washington, the parent institution of the Department of Genetics at Cold Spring Harbor, and he arranged for Shannon to visit the Laboratory to work with Barbara Burks, a highly respected behavioral geneticist with interests in mathematics and statistics. Shannon spent the summer of 1939 at the Laboratory and wrote a highly original thesis on an algebra that described genetic changes in an evolving Mendelian population. Sadly, he never published the work and it had little impact in genetics.


Rich Roberts at CSHL in 1975

When the Biological Laboratory and Department of Genetics at Cold Spring Harbor finally merged in 1962, the new institution was formally called the “Cold Spring Harbor Laboratory of Quantitative Biology,” a name that lasted until 1970. The 1970s saw the introduction of computers to CSHL, led by Rich Roberts and Jim Garrels. Gradually, bioinformatics became increasingly developed to support growing efforts in DNA sequencing and genomics. In the 1990s and early 2000s, two computer specialists on the CSHL faculty, Lincoln Stein and Michael Zhang, became prominent in the bioinformatics community. Other investigators, such as Mike Wigler and Dick McCombie, made increasingly heavy use of computational tools and techniques in their research programs in genetics.


The Hillside Campus at CSHL, home to the Simons Center for Quantitative Biology

For a number of years, quantitative biology research continued in the absence of a designated unit, but in 2008, the Center for Quantitative Biology was created, with Mike Wigler as Director. The Center would later be rechristened the Simons Center for Quantitative Biology in recognition of generous donations from the Simons Foundation.


Crow, J. F. (2001). Shannon’s brief foray into genetics. Genetics (Vol. 159, pp. 915–917).

Provine, W. B. (1989). Sewall Wright and Evolutionary Biology. University of Chicago Press.

Provine, W. B. (2001). The Origins of Theoretical Population Genetics: With a New Afterword. University of Chicago Press.

Susman, M. (1995). The Cold Spring Harbor Phage Course (1945-1970): a 50th anniversary remembrance. Genetics, 139(3), 1101–1106.

70-Year Archive of Cold Spring Harbor Symposia on Quantitative Biology. http://symposium.cshlp.org/site/misc/index_archive.xhtml

FitCons Method Published in Nature Genetics and Rebuttal to Cooper et al.’s Critique (‘Don’t be a CADD’)

Our paper on our new computational method for computing probabilities of fitness consequences for point mutations in the human genome, originally released as a preprint on the bioRxiv, appeared online today in Nature Genetics.  As I described in an earlier blog post, this paper is the fruit of more than three years of labor led by Brad Gulko and Ilan Gronau in my research group.  It is satisfying to finally see it in print not only because this has been a long, arduous project with a number of false starts, but also because this work neatly weaves together several threads in the research my group has pursued over the last decade, including the use of evolutionary conservation to characterize genomic function, the detection of natural selection based on joint patterns of polymorphism and divergence, estimation of the share of the human genome that is under selection, and characterization of the impact of evolutionary turnover on patterns of genetic variation.

In addition, this paper provides an excellent example of a publication model about which I am increasingly enthusiastic: rapid publication of a preprint in a fully open fashion, improvement of the manuscript based on community feedback, rigorous peer review, further revision, and publication in a high-quality journal.  Whatever opinions one holds about the Nature Publishing Group—and they are undoubtedly controversial in the circles I move in—one must acknowledge that they deserve some credit for their willingness to consider and publish papers, like ours, that have been fully “in the open” since before submission. 

I want to turn now to the less pleasant issue of a critique of our article that was posted on Haldane’s Sieve by Greg Cooper and colleagues a couple of months ago.  Let me begin with my perspective on the background for this critique. 

While preparing our manuscript, we came across a paper by Cooper, Jay Shendure, Martin Kircher and colleagues on a method called CADD (an acronym for the rather cryptic name “Combined Annotation-Dependent Depletion”) that purports to solve a similar problem to the one addressed by our fitCons method.  In the months prior to publication, CADD was heavily promoted in meetings and conferences and at the time of publication it was beginning to show signs of serious “market penetration” in the human genetics community.  We spent some time closely evaluating the CADD paper and, in all honesty, found the approach more than a little naive from both a machine learning and a molecular evolution standpoint.  In addition, while CADD evidently does perform quite well in coding regions, we found the claims of its effectiveness in predicting “pathogenicity” in noncoding regions to be highly unconvincing.  As we studied the paper more closely and carried out some of our own analyses of the CADD scores, we also identified some troubling features of the reported validation experiments.  In our hands, the CADD scores did not seem to perform nearly as well outside of coding regions as suggested in the paper.

We decided to approach this issue head-on by including CADD in our validation tests and comparing it directly with fitCons (along with several other methods), focusing on prediction performance at putative cis regulatory elements.  We also included a paragraph on CADD in our discussion speculating on the reasons for its much poorer performance on our tests than on those reported in the CADD paper.  Among other things, this paragraph called out the CADD paper for some features of their validation tests that we found misleading.

Greg Cooper was unhappy with our treatment of CADD, and within a few days responded to our preprint on the bioRxiv with a long, highly critical email.  Among other things, Greg felt that we had inaccurately described what our fitCons scores actually represent, unfairly compared fitCons with CADD, and used validation experiments that were strongly biased in favor of fitCons. 

I took some time to mull over these criticisms during a trip to Oxford, and then responded about two weeks later with a rather carefully written letter that spelled out the principles behind our approach, why we think the two methods are fundamentally very similar and deserve to be compared, and the rationale behind our validation experiments.  I have posted the full text of this letter below, because I think it still does a reasonably good job of capturing my perspective on these issues.

There followed a series of emails that were at times cordial and constructive, but never resulted in resolution of our differing points of view.  During this exchange, my group performed two sets of substantial additional validation experiments to address concerns that Greg had raised: an analysis of high information content positions within TFBSs and a reanalysis of eQTLs that corrected for certain ascertainment biases.  We felt that these analyses provided a convincing refutation of Greg’s most serious criticisms, and we incorporated them into a revised preprint and, eventually, the published manuscript.  Nevertheless, Greg remain unconvinced by our arguments, and he evidently decided to go on the offensive and publish his critique of our paper as an extended blog post.

At this point, it hardly needs to be stated that we disagree with almost everything in Greg’s critique.  His blog post mischaracterizes our goals, our claims, our approach, and our results.  And it fails to recognize several highly relevant points we had communicated clearly to him by email, for example, concerning the rationale behind our approach and our validation experiments, and the reasons why fitCons performs poorly at the three loci that were experimentally analyzed by the Shendure group.

I recognize that few readers will have patience for this level of detail, but, in the interest of setting the record straight, I provide a point-by-point rebuttal of Greg’s critique in the post that follows.  I want it to be clear that we stand by our paper, all of our results, and our comparative evaluation of fitCons and CADD. And I fully expect that independent evaluations of performance of these methods in characterizing fitness consequences of mutations in regulatory regions will support our findings.

Thoughts on: Probabilities of Fitness Consequences for Point Mutations Across the Human Genome

Posted on October 23, 2014 by Joe Pickrell

This guest post is by Greg Cooper, Martin Kircher, Daniela Witten, and Jay Shendure, and is a comment on Probabilities of Fitness Consequences for Point Mutations Across the Human Genome by Gulko et al.


Recently, Gulko et al. (2014) described an approach, FitCons, to estimate fitness consequences for point mutations using a combination of functional genomic annotations and inferences of selection based on human variant frequency spectra.

This is not quite an adequate summary of our approach.  We consider patterns of divergence between primates as well as allele frequencies in human populations.  Our inferences of natural selection are based on patterns divergence and polymorphism at the sites of interest in contrast to those at nearby neutral sites.  We interpret estimates of fractions of sites under selection within a class of genomic positions as probabilities that point mutations within that class will have fitness consequences.

On the basis of comparisons with several maps of regulatory element features, they concluded that FitCons is substantially better at inferring deleterious regulatory effects of variants than other metrics, including an annotation we developed named Combined Annotation Dependent Depletion (CADD, Kircher et al. 2014). However, we believe that the comparisons of FitCons and CADD for detecting deleterious regulatory variation are misleading, and that methods to predict fitness effects of point mutations should evaluate variants with demonstrable effects rather than variants assumed to have an effect by virtue of being within a functional element.

This is a distorted characterization of the difference between our validation approaches.  Cooper et al. have not evaluated variants having any more “demonstrable effects” than the ones we have used, particularly vis-a-vis cis-regulatory elements.  To the contrary, we believe that, on the whole, our validation experiments are more direct and more comprehensive than those used in the CADD paper (see arguments below).     

We find that FitCons is substantially less effective than CADD at separating variants, both coding and regulatory, with functional and/or phenotypic effects from functionally inert and/or organismally benign variants. For example, CADD is much more effective at enriching for mutations in two enhancers and one promoter that have been experimentally shown to have large transcriptional effects.

As Cooper and colleagues know from our correspondence with them, the results of their saturation mutagenesis analysis at their three hand-picked loci are largely a consequence of the gene expression patterns at these loci.  Our approach is designed to make use of cell-type-specific data.  Consequently, it has little predictive power at loci not active in the analyzed cell types, and that happens to be the case in our cell types for two of these three loci (see details below)  Furthermore, the correlation coefficients cited by Cooper and colleagues suggest that all of the methods tested are performing very poorly by their benchmarks.  It is hard to see how the authors can justify claiming a great victory when their method appears to explain less than 10% of the variance in their own benchmarking statistic!

Further, in contrast with CADD, FitCons does not separate highly deleterious variants that cause Mendelian disease from high-frequency benign variants, nor does it separate complex-trait associated variants, which are enriched for deleterious regulatory effects, from matched control variants. We believe that it would be more appropriate to characterize FitCons as a predictor of cell-type specific regulatory elements, and to compare it to other tools directed specifically at this task, rather than variant fitness consequences.

Notice the leap of faith above to prediction of “highly deleterious variants that cause Mendelian disease” and “complex-trait associated variants”.  These are distant goals well beyond what we have claimed to address in our paper, and Cooper et al.’s “demonstration” that CADD  is achieving them requires a good deal of creative license (as we discuss more specifically below).  In fact, both CADD and fitCons are really just predicting the local action of purifying selection over relatively long evolutionary time periods, based on various types of covariates along the genome.  There are differences in the particular covariates considered, the prediction methods used, and the evolutionary time periods represented but the set-up for both methods is fundamentally the same.  Thus, in both cases, any predictive power for Mendelian or complex traits depends on correlations between these traits and long-term purifying selection on individual nucleotides.  We expect that purifying selection will be at least somewhat useful as a proxy for these phenotypes, but many phenotype-associated variants will not show signs of selection, and many sites under selection will not have direct phenotypic links.  For this reason, we think it is more meaningful to speak of “probabilities of fitness consequences” than “pathogenicity”.  In addition, we think that it is more useful and informative to evaluate the performance of these predictors on molecular phenotypes in particular cell types, such as ChIP-seq-based transcription factor binding sites, rather than on statistically associations with complex phenotypes through unknown mechanisms.

FitCons, recently developed by Gulko et al (2014), is a method to estimate fitness effects of point mutations at both coding and non-coding positions in human genomes. FitCons works by first defining regional boundaries on the basis of clusters of functional genomic signals (“fingerprints”) and then estimating selective effects, inferred from allele frequency distributions within human populations, for variants with the same fingerprint. On the basis of comparisons with enhancers, transcription factor binding sites (TFBSs), and expression quantitative trait loci (eQTLs), Gulko et al. concluded that FitCons is substantially better at inferring variant regulatory effects than other metrics, including an annotation we developed named Combined Annotation Dependent Depletion (CADD, Kircher et al. 2014).

We never describe our method as “estimating fitness effects of point mutations” but instead say that it estimates the “probabilities that point mutations will have fitness effects” across groups of sites. This apparently subtle difference is important because the first phrase suggests prediction of specific effects of specific mutations, whereas we have never made any other claim than that we can produce a useful average measure of the probabilities that individual mutations will have fitness effects across coarse-grained classes of sites.  Clearly, the more specific type of prediction would be more valuable than the more general one, were it possible to do accurately, but we believe even these coarser predictions of average effects represent an important step forward in noncoding regions of the genome.  As we discuss later in this response, we also think that the evidence is very weak that CADD’s highly specific predictions carry much useful information about the actual impact of regulatory mutations.

While FitCons is an interesting approach with potentially useful attributes, we believe that the comparisons of FitCons and CADD for detecting deleterious regulatory variation are misleading. Clarification is needed as to the purposes and performances of these metrics. Below, we first describe what we believe to be important general distinctions between CADD and FitCons and then detail their relative effectiveness at differentiating several sets of functional and/or pathogenic variants from inert and/or benign variants. Finally, we consider how correlations between the bin definitions and validation datasets used in Gulko et al., rather than fitness per se, may underlie the performance of FitCons for cell-type specific regulatory element prediction.

CADD is an allele-specific measure of variant deleteriousness that accounts for a wide variety of discrete and quantitative information at the allelic, site, and regional levels (Kircher et al. 2014). CADD scores can vary among the possible alleles at a given site, across sites within a given region or functional element, and between and across variants within differing classes of functional elements.

It is true that CADD considers many sources of information as input and that it produces different scores for different alleles.  However, it is not at all clear that it is really able to make good use of all of that information and that those allele-specific differences in score are meaningful, particularly in noncoding regions.  In other words, while CADD has access to this information, whether or not it truly “accounts” for it is an open question. Our evaluations suggest, to the contrary, that CADD’s scores have quite limited predictive power for fitness consequences at cis-regulatory elements.

FitCons, on the other hand is driven by a small number of cell-type-specific, regional features with reduced or absent variation within regions. As a result, FitCons is in practice a segmentation method: the median length of uniformly scored segments is 72 bases, the average segment length is 196 bases, and 50% of all scored bases in hg19 lie within a segment over 950 bases long. Furthermore, 30% of all bases in hg19 are assigned to the mode value (0.062), 60% are assigned one of two FitCons values, and over 80% are assigned one of 10 possible values. Thus, FitCons is in practice a regional annotation of cell-type-specific molecular activity, not a site or allele-specific metric of variant deleteriousness.

FitCons first clusters sites according to several functional genomic covariates, then predicts average probabilities of mutational fitness consequences within each cluster.  As we have stated clearly in the paper, these clusters are fairly coarse-grained.  This feature of the approach is by design.  In our view, the coarse-grained nature of the clusters is well matched to the information in the data.  There is simply very little information about the precise fitness consequences of particular, position-specific mutations across most of the noncoding genome.  We opt to produce a meaningful low-resolution score instead of a higher resolution score that produces a misleading impression of precision.

That a large fraction of sites are assigned a uniform low value is also part of the design.  These are the sites in our “null” class, with no signal from DNAse-seq, RNA-seq, or histone modifications, and they are assigned a “background” value representing our estimate of average probabilities of fitness consequences in the absence of informative functional genomic data.

These observations should not be surprising to anyone who has taken the trouble to read our paper.  The blockiness of the scores and the fact that relatively few distinct scores dominate the genome wide distribution are natural consequences of the functional genomic covariates considered and the approach for estimating class-wise probabilities of fitness consequences.  Our genome-wide tracks and ROC plots demonstrate that this simple design leads to remarkably informative scores.

The basic structures of FitCons and CADD are crucial to interpreting the data presented by Gulko et al. In particular, they measure utility by assessing coverage of bases within functional elements, namely TFBSs and enhancers, relative to genomic background. While such an approach is reasonable to evaluate a method to annotate functional elements, it is not informative for a method to estimate organismal deleteriousness since many mutations within functional elements are evolutionarily neutral, including many that lack even a molecular effect. To wit, by FitCons/INSIGHT estimates, most sites within the enhancers and TFBSs evaluated have fitness effect probabilities below 0.2 (Gulko et al. 2014). While likely somewhat higher among high-information TF-binding motif positions and lower among the enhancers used (mean size of 888 bp), a decisive majority of positions in these nucleotide groups are mutable without consequence. Performance evaluations that reward uniformly high coverage of bases in these regions, rather than the particular subset of variants therein that actually have deleterious molecular effects, are therefore not meaningful for estimates of point mutation fitness consequences.

More of the same.  Cooper and colleagues have an admirable ideal in mind—a computational method that can precisely predict the fitness consequences of any possible mutation at any position across the genome, distinguishing mutations that alter TF binding affinity or miRNA binding preferences or ncRNA structure from those that do not.  The problem is that their dream so far has no basis in reality.  The evidence that CADD, in particular, is making fine distinctions of the kind they describe across the noncoding genome is exceedingly weak.  FitCons is at least producing coarse-grained predictions of average effects that are useful in practice for distinguishing signal from noise across the genome.

We firmly believe that methods to predict functional or fitness effects of mutations should be evaluated on mutations for which we have data relevant to function and fitness, not large aggregates of genomic regions or bases within which mutations are simply assumed to be phenotypically relevant. When tested on such mutation sets, we find that FitCons fails to capture a considerable amount of site- and allele-specific information that is captured by CADD (and between-species conservation metrics to a lesser extent). This loss of information, in turn, has profound effects on FitCons’ ability to identify variants with functional, pathogenic, or deleterious effects, including for regulatory changes.

We acknowledge that our benchmarks rely on indirect measures of “function and fitness effects of mutations”, but the measures of performance Cooper and colleagues have actually examined—correlations of scores with reporter-gene-based saturation mutagenesis experiments at three loci, ROC plots for very small numbers of ClinVar variants, and enrichments for GWAS hits—are no more direct than our evaluations of transcription factor binding sites, eQTLs, and enhancers.   Moreover, Cooper et al. fail to mention that, in response to their privately relayed criticisms of our validation experiments, we performed a follow-up analysis that considered only the high-information-content positions within our ChIP-seq-supported transcription factor binding sites, which should be strongly enriched for the particular positions at which fitness consequences are most severe.  As shown in the latest version of our paper on the bioRxiv (and the version published in Nature Genetics), we observed no substantial difference in our ROC plots when focusing on this set, indicating that the excellent performance of fitCons in those experiments is not an artifact of our focus on whole regulatory elements (rather than individual positions) in our tests.

First, FitCons has no predictive power for separating pathogenic variants in ClinVar (Landrum et al. 2014) from benign, high-frequency polymorphisms matched for genic effect category (e.g., missense, nonsense, etc): the distributions of FitCons scores for pathogenic and benign variants are nearly identical (Figure 1). While most of these variants are protein-altering, this same pattern holds for the subset of pathogenic/benign variants that do not directly alter proteins (Figure 1, right). In contrast, CADD and conservation measures like GERP (Cooper et al. 2005) strongly differentiate pathogenic from high-frequency variants, and, although more weakly, also differentiate non-protein-altering pathogenic from benign variants (for further details, see Kircher et al. 2014). The inability of FitCons to distinguish these highly pathogenic/deleterious variants from clearly benign variants runs counter to the general narrative in Gulko et al. in which FitCons scores are claimed to correlate with mutational fitness effect probabilities.

Here Cooper et al. double down on ClinVar, which, as we have pointed out, is highly problematic in an evaluation of genome-wide performance because it is dominated by protein-coding variants.  In the analysis above, Cooper and colleagues try to address this criticism by looking at sites in ClinVar outside of coding regions, but what they don’t mention is just how tiny this set is.

When we last reviewed this database we found that, of roughly 10,000 autosomal pathogenic positions in ClinVar only about 3.4% fall outside of coding sequences (CDSs) and fewer than 1% are more than 100bp away from a CDS.  As far as we could tell, ClinVar also gives no indication of whether these are true (transcription-associated) cis-regulatory variants, as opposed to, say, variants that disrupt splicing or variants that fall in CDSs in alternative isoforms of a transcript.  So the analysis of noncoding variants above reflects at most a few hundred variants, with some unknown mixture of variants in cis-regulatory elements and variants more directly associated with protein-coding function, and presumably, highly variable activity across cell types.  In short, this database is totally inadequate for the kind of validation we are interested in, and it is not surprising that fitCons would perform poorly on it.  We expect that the elevation in CADD scores at these variants is largely driven by conservation, proximity to exon boundaries, mutations in known splice sites, and other gene-annotation-based features, rather than by an ability to accurately estimate fitness effects associated with cis-regulatory function.

Second, as Gulko et al. emphasize the detection of regulatory variation (title of the manuscript not withstanding), we performed a detailed examination of three regulatory elements for which saturation mutagenesis data exist (Patwardhan et al. 2009; 2012). While not global, these data are comprised of directly measured, not assumed, regulatory effects.

These saturation mutagenesis data are undoubtedly informative, but they are hardly definitive as functional characterizations of regulatory elements.  They are based on in vitro reporter gene experiments that may or may not capture in vivo regulatory patterns in particular cellular contexts.  The contribution of experimental noise in these measurements also appears to be relatively large. Overall, it is not clear to us that these data are all that much more direct in measuring “pathogenicity” than are binding patterns of transcription factors as assessed by ChIP-seq and informatics analysis (the basis of our first set of validation experiments).  And when one considers the much larger size of our ChIP-seq data set (55,844 TFBSs compared to three elements), we think it is clear that our benchmarks are much more likely to be representative of the genome than are the ones reported here.

Nevertheless, why does fitCons perform so poorly at the three loci examined by Cooper et al.?  As we have explained to the authors, what fitCons is doing here is no mystery: the fitCons scores at these loci are a direct consequence of the expression patterns and functional genomic marks at these loci in the three cell types we have examined.  It just so happens that two of the three loci (the HBB promoter and ALDOB enhancer) have almost no functional genomic signal in our data—they mostly fall in our “0” classes for RNA-seq and DNase-seq and are annotated as “quiescent” by ChromHMM in all three cell types (with the exception of a weak RNA-seq signal in H1-hESC over part of the ALDOB enhancer).  This absence of signal can be clearly seen in the fitCons tracks on our browser mirror (http://genome-mirror.cshl.edu/) at approximately chr11:5,248,210-5,248,401 (HBB) and chr9:104,195,570-104,195,828 (ALDOB) in the hg19 assembly.  As a result, fitCons scores these regions at or slightly above the genomic background across these elements, exactly as designed.  We have made the decision to produce cell-type-specific scores based on cell-type-specific functional genomic information.  As we show in our paper, this design leads to excellent performance in many cases, but it does have the necessary and inevitable consequence of eliminating any prediction power at genomic loci that are not active in any of the cell types we have examined. 

In the 70-bp promoter region of HBB (Patwardhan et al. 2009), FitCons assigns all bases to the genome-wide mode (0.062). However, mutations in this region exhibit substantial variation in both transcriptional and disease consequences. Mutational effects on in vitro promoter activity range from no effect to a >2-fold change in transcription, and some of the strong in vitro effect mutations cause beta-thalassemia by disrupting normal transcript regulation. CADD and GERP correlate significantly with the regulatory (CADD Spearman’s rho=0.23, GERP rho=0.11) and disease consequences of these mutations (details in Kircher et al. 2014).

Incidentally, CADD’s Spearman’s rho of 0.23 in this region seems to support our point that it is at best weakly informative about regulatory function.  While this value is significantly different from zero, it represents a very poor correlation in absolute terms.  Spearman’s rho is simply Pearson’s r for ranks.  Therefore, it is fair to say that CADD has an r^2 of 0.23^2 = 0.053 for rank order in transcriptional output as assessed by saturation mutagenesis.  In other words, it is able to explain only about 5% of the total variance in ranks.

Within each of two enhancers tested by saturation mutagenesis (Patwardhan et al. 2012), FitCons scores are correlated with mutational effect (ECR11 rho=0.32, ALDOB rho=0.26) similar in magnitude to CADD (ECR11 rho=0.25, ALDOB rho=0.36). However, in both elements, the FitCons correlation is due to a higher score segment overlapping a more transcriptionally active region (Figure 2); no predictive power within the active regions exists. For example, most of the mutations with regulatory effects in the ECR11 enhancer reside in the last ~100 bases, which in turn reside within a single 168-bp FitCons segment. Within this segment, considerable mutational effect variation exists: 209 of 504 possible SNVs, distributed across 110 of the 168 sites, have no discernible effect on transcription (p >= 0.1). Concordantly, these inert mutations have significantly lower CADD scores (Wilcox test p=5.9 x 10-25) than their interdigitated SNV neighbors with at least some evidence for functional effect. Furthermore, within the set of mutations that have at least some evidence for effect (p<0.1; other arbitrary thresholds yield similar results), transcriptional effect sizes vary considerably and correlate with CADD (rho=0.33).

The ECR11 locus is the one of the three at which a functional genomic signal is present in our data—namely, we have promoter and weak-enhancer ChromHMM states and a moderate DNase-seq signal (broad peaks) here in the GM12878 cell type (see chr2:169,939,082-169,939,701 in the browser), leading to three blocks of modestly elevated scores (as shown in Cooper’s Figure 2). We would argue that the fitCons scores are doing exactly what they are designed to do at this locus, by distinguishing regions with stronger average functional effects from those with weaker effects, without providing misleadingly specific predictions of fitness effects.  CADD, by contrast, makes very precise predictions about the functional consequences of each possible mutation at each position, but there is little support for most of these predictions.  Indeed, in this case, CADD explains only 6%-13% of the variance in ranks in transcriptional output. In addition, it is very difficult to tell what is driving these predictions—CADD is completely a “black box” for the user. 

Next, as suggested by Gulko et al., we quantified coverage of discretely thresholded regulatory variants to evaluate the extent to which FitCons and CADD could enrich for “large-effect” regulatory mutations. Specifically, there are 108 mutations that alter transcriptional activity by at least two-fold within the three elements tested (29 mutations across 19 bases in ECR11, 76 mutations across 41 bases in ALDOB, and 3 mutations across 3 sites in HBB). We compared coverage of these 108 mutations at various thresholds relative to coverage of hg19, and find that CADD is much more effective at enriching for them than is FitCons (Figure 3). For example, 95 (88%) of the large-effect regulatory variants have a scaled CADD score above 10, a threshold that includes 10% of all possible hg19 SNVs (~9-fold enrichment above genomic background). Enrichment peaks at a CADD score of 15, a threshold that includes 53.7% of large-effect regulatory variants but only 3.2% of hg19 SNVs (~17-fold enrichment). In contrast, FitCons enrichment peaks at a threshold of 0.08, wherein only ~27% of all large-effect mutations are covered (~4-fold enrichment above background).

We next evaluated the ability of FitCons to distinguish trait-associated SNPs identified in genome-wide association studies (GWAS). Such SNPs are as a group enriched for regulatory variants with pathogenic, likely deleterious effects, a hypothesis supported by numerous anecdotal and systematic studies (e.g., Hindorff et al. 2009; Musunuru et al. 2010; Nicolae et al. 2010; ENCODE Project Consortium et al. 2012). These variants are overwhelmingly non-protein-altering (98%), with ~83% being intronic or intergenic SNPs not near to an exon. We previously showed that CADD scores significantly separate lead GWAS SNPs from matched neighboring control SNPs (Wilcoxon p=5.3 x 10-12). This separation remains highly significant for the 83% that are intronic or intergenic (p=1.26 x 10-9), indicating it is not driven solely by coding or near-coding variation. In contrast, FitCons scores do not separate lead GWAS SNPs from controls, either considering all variants (p=0.32) or intronic/intergenic only (p=0.57).

Cooper et al. are evidently much more impressed with GWAS enrichments than we are.  The problem with analyses like this one—showing that a group of sites statistically enriched for some property, are also statistically enriched for another property—is that they are notoriously hard to interpret.  Typically the fold enrichments are modest (it is telling that they were not reported above) and very often the statistical differences can be explained by relatively uninteresting covariates (GC content, overall conservation, proximity to exons, etc.). The genome is large and statistically significant differences between collections of sites are easy to find.  The real question, in our view, is does one have any predictive power to identify individual variants of interest.  For this reason, we have focused on ROC and ROC-like plots in our validation experiments.  In our hands, CADD has very little predictive power for noncoding sites of known functional significance, and what it does have often appears to be largely a consequence of local evolutionary conservation or proximity to genes.

With respect to separation of eQTL SNPs from controls, Gulko et al. used all common variants that were part of the study as a background/control set. We believe the results from such a test are difficult to interpret. They do not control for effects of minor allele frequency, for example, a key property that correlates with both technical (e.g., eQTL discovery power) and biological effects (e.g., eQTL deleteriousness). Additionally, they do not control for genomic location. By virtue of the annotations it uses, FitCons scores will tend to be higher near transcribed genes in the cell-type of choice, which are in turn the only genes for which eQTLs can be identified. Therefore, this analysis confounds the information content resulting from a focus on cis, rather than trans, eQTLs, with that intrinsic to the scores themselves. While this may be an advantage, relative to a more general predictor like CADD, for predicting cell-type-specific function, it likely comes at a cost of reduced accuracy in terms of predicting deleteriousness per se (see below). Furthermore, in practical terms it is not likely to be useful given that cis-effects are the first and major focus of most eQTL discovery efforts, and it is furthermore unclear that FitCons would outperform other cell-type specific regulatory element annotations, such as those from integrative predictions of enhancers and promoters (Ernst and Kellis 2012; Hoffman et al. 2012). In any case, an analysis in which eQTL SNPs are matched for MAF, genomic location, distance to TSS and other confounders would provide a more meaningful evaluation of the utility of FitCons in a realistic eQTL discovery/fine-mapping analysis.

The conjecture that fitCons only shows good performance at eQTLs because we have not controlled for minor allele frequency or genomic location is wrong.  We have repeated these experiments after stratifying the data by proximity to TSS and MAF and see very little difference in the ROC plots.

Finally, we believe that the discrepancies in performance metrics defined here vs. within Gulko et al. are also influenced by the potential circularity of cell-type-specific information within both the model definition and validation. Indeed, while INSIGHT adds value by scoring the 624 potential feature-defined bins, the correlations between the bin definitions (expression, DNase, and ChromHMM states) and validation data (TFBSs, enhancers, eQTL candidates) are quite likely the primary drivers of performance of FitCons as measured in Gulko et al.

But a correlation between the definitions of the clusters and the validation data is only a problem if it is not driven by the underlying biology.  In a sense, defining the clusters so that they are correlated with true cell-type-specific functional roles is the whole point of the approach!  And we have taken care to use measures of cis-regulatory function (such as eQTL association and GRO-seq-based enhancer identification) that are as different as possible from the covariates used to define the scores.

In fact, the strong correlation between INSIGHT and PhyloP as a bin scoring method suggests that metrics of evolutionary constraint could replace INSIGHT in the FitCons framework.

As discussed in our comparison of the fitCons scores with their divergence-based counterpart (called fitConsD), much of the information in fitCons does come from divergence but there is also an important contribution from polymorphism.  The difference between the fitCons and fitConsD scores is the basis of our analysis of evolutionary turnover.

More generally, the use of cell-type specific features in both the metric definition and validation obscures a crucial trade-off in analyses of this sort. Evolutionary constraint-driven metrics (including CADD, which is strongly influenced by evolutionary constraint) emphasize variant effects on organismal fitness, which often have nothing to do with molecular function in any given cell-type; this means that constraint-driven metrics may be sub-optimal when trying to predict molecular function within said cell-type. However, the converse is also true, in that efforts to predict deleteriousness that emphasize cell-type specific molecular functionality will often be misleading when a variant has no molecular effect in that cell-type but strong fitness consequences due to functional effects elsewhere and/or has a molecular effect with no fitness consequence. Obviously, optimal choices in this trade-off depends greatly on analytical context and goals, but in our opinion the goal of predicting “fitness consequences for point mutations” dictates that performance metrics focused on organismal deleteriousness are more appropriate.

We don’t disagree that there are tradeoffs concerning the use of cell-type-specific data, and, indeed, were quite frank in our paper that improved methods for integrating across cell types are an important area for improvement of our methods (see last paragraph of Discussion).  CADD also attempts to use cell-type-specific data for prediction—it just does it in a rather crude way, by simply including summaries across cell types as covariates and hoping that its SVM can sort them out.  We also agree that the use of cell-type-specific data for validation provides an imperfect measure of organismal fitness.  But, as discussed above, we believe that the measures considered in the CADD paper are at least as imperfect.

As a final illustrative example, Weedon et al. (2014) identified a set of noncoding SNVs that disrupt the function of an enhancer of PTF1A and cause pancreatic agenesis. CADD scores these variants between 23.2 and 24.5, higher than 99.5% of all possible hg19 SNVs and higher than 56% of pathogenic SNVs in ClinVar (most of which are protein-altering); much of the CADD signal for these variants results from measures of mammalian constraint (not shown). FitCons, on the other hand, places these variants in a 5-kb block of sites all scored at the genome-wide mode (0.062). This is in part a result of not having functional genomic data from cells in which the enhancer is active; however, the absence of such data in disease studies is common given that the relevant cell-types are frequently unknown or inaccessible. Further, even if DNase, RNA, and ChromHMM data were all generated for this cell type, given the general distributions of FitCons scores within regulatory elements observed in other cell types and lack of inter-species conservation information, it is unlikely that FitCons would have ranked these variants within the top 0.5% of all possible SNVs.

We are not trying to duplicate the functionality of tools such as phastCons, phyloP, or GERP (which presumably work fairly well at this locus) but to provide something complementary, based on cell-type-specific functional genomic data.  At loci unexpressed or inactive in the cell types we have analyzed, one is clearly better off using evolutionary constraint than fitCons.  It is unclear, from our experiments, how much value CADD adds at such loci.

In any case, Gulko et al. demonstrate that FitCons is reasonably effective, and more so than CADD, at predicting the approximate boundaries of regulatory elements in cell types on which it is trained. However, claims that it better predicts functional or fitness effects of variants in either coding or non-coding regions are unsupported. Indeed, when challenged to separate point mutations with demonstrable effects from appropriate sets of control SNVs, CADD and other metrics that include evolutionary constraint information are substantially better as predictors of both coding and non-coding variant impact.

It is simply wrong to dismiss our benchmarking analyses by saying that they merely demonstrate that fitCons can find “approximate boundaries of regulatory elements in cell types on which it is trained.”  We have shown that fitCons outperforms CADD at high information content positions of TFBSs and at eQTLs, both of which should be highly enriched for positions at which point mutations have fitness effects.   We have also shown that our methods generalize reasonably well to cell types other than the ones we have trained on (Supplementary Figure 10).

We suggest that it would be more appropriate to characterize FitCons as a predictor of cell-type specific regulatory elements rather than variant fitness consequences, and to compare it to other tools directed at this task, such as ChromHMM (Ernst and Kellis 2012) or Segway (Hoffman et al. 2012).

ChromHMM and Segway are essentially clustering methods, with no use of evolutionary information.  They produce multivariate outputs (assignments of genomic positions to multiple classes) rather than a univariate score meant to serve as a means for ranking genomic positions for follow-up study, as CADD and fitCons produce.  We maintain that the comparison with CADD is appropriate.

Letter to Greg Cooper

[email to Gregory Cooper sent July 26, 2014, cc’d to Jay Shendure, Martin Kircher, Brad Gulko, and Ilan Gronau]

Hi Greg,

I’ve spent some time thinking about your letter and have composed a response (below). There were quite a few criticisms and allegations packed into those few paragraphs, so I thought it was best to break out what I saw as the major points and address them one by one.  Please let me know if any of these responses are unclear. I would like to make this exchange as constructive as possible for both groups.

I’ve copied Brad Gulko and Ilan Gronau, who have been the leads in my group on the fitCons project.



Claim: You are not really predicting fitness consequences of alleles, and the resolution of the fitCons scores along the genome is very coarse.  Your title and the name “fitCons scores” are misleading.

Response: First, note that we have not claimed to predict fitness effects of mutations in terms of allele-specific selective coefficients or some similar measurement.  We think we make it quite clear in the title and the manuscript that our predictions are of “probabilities that the nucleotide identity at individual genomic positions influences fitness”.  I should add that by this we mean “influence fitness to a sufficient degree to substantially influence patterns of genetic variation within and between species”.  We are not making predictions about nearly neutral mutations, with |2Ns| ~ 1, but mutations having |2Ns| of at least about 5-10.  (Our papers on INSIGHT explore these issues in some detail.)

Second, you are correct that we have not made separate predictions for each alternative allele at each site, but we do not believe that this is an important difference between our methods and yours.  If a user were to insist on allele-specific mutations, we can easily produce them — our model simply assumes that the probability of fitness consequences for each candidate point mutation are equal, so the score for each alternative allele would be the same.

On a philosophical level, we are fundamentally Bayesian in our approach to prediction tasks in computational genomics, and I think there is a clear Bayesian rationale for defining our scores as “probabilities of fitness consequences”.  The question we wish to address is the following: given a collection of functional genomic covariates along the genome and a measure of natural selection based on patterns of genetic variation, what is the probability that each nucleotide position is under natural selection, hence that it will have fitness consequences if mutated?  Our approach is to group sites into large, relatively coarse-grained categories, to pool data about genetic variation within these groups, then to treat an estimate of the fraction of sites under selection as a probability of fitness consequences per site.  Under the assumption of exchangeability of sites within each class, this is a legitimate Bayesian estimator.  Obviously, this strategy ignores considerable heterogeneity within each class and results in scores that are “blocky” along the genome.  But there is nothing fundamentally unfair or misleading about treating these estimates as our “best guesses” for probabilities of fitness consequences given the data at hand.  How good those “best guesses” are is, of course, another question (to be addressed below).

Claim: The assumptions underlying the fitCons scores are unrealistic.  For example, your approach effectively assumes that all nucleotide positions in a transcription factor binding site experience the same degree of selective constraint, which is obviously wrong.

Response: Our modeling approach does indeed make some rather strong simplifying assumptions, but we are mindful of the fact that all models are approximate and any prediction method should be guided by what is realistically possible given the information in the data.  I am sure we do not have to explain to you that prediction methods that make fewer simplifying assumptions are not necessarily better, because of factors such as overfitting, bias/variance tradeoffs, etc.  Frankly, in our view, CADD is being at least as unrealistic as we are by imagining that it will be possible to make meaningful position- and allele-specific predictions of pathogenicity across unannotated noncoding regions of the genome, by simply throwing a large number of covariates into an SVM with a linear kernel.  We just don’t believe that there’s enough information in the data to support these types of predictions, and even if there were, that your linear-kernel SVM framework would be adequate for exploiting it.

To illustrate the point, consider the slightly absurd analogy of an insurance company that chooses to predict not only whether a customer will have an automobile accident over the next year and approximately how much it will cost, but also on what day and in what town an accident will occur.  Obviously, those predictions would be meaningless, and, worse, potentially misleading for the unsophisticated user.  We believe that, at least in noncoding regions, it is more honest and realistic to work at the coarse-grained level than to provide highly specific predictions of the type CADD attempts to make.

In any case, there’s no point in having a philosophical discussion about the costs and benefits of alternative modeling approaches when they can be compared empirically.  The proof is in the pudding.  Which brings me to the next point…

Claim: Your validation experiments are unfair to CADD because they measure “bulk coverage of elements” rather than allele-specific pathogenicity.

Response: In answer to this claim, I want to make two arguments: first, that CADD and fitCons scores are more similar than they are different and therefore can reasonably be compared, and second, that our evaluation of candidate cis-regulatory elements is a reasonable and fair way to evaluate the performance of both methods in noncoding regions.

Despite differences in resolution and allele-specificity, as discussed above, CADD and fitCons scores both essentially address the problem of measuring the “deleteriousness” of mutations at individual sites along the genome, given a collection of covariates and a “readout” of deleteriousness based on patterns of genetic variation.  To be sure, the two methods use rather different modeling and prediction strategies.  In particular, instead of explicitly clustering genomic positions into distinct classes, CADD is implicitly grouping sites with similar patterns of covariates using an SVM regression strategy.  And instead of a model-based inference strategy for its evolutionary readout, it is using a simulation-based measure of constraint.  Nevertheless, we believe that these are two different approaches for addressing the same fundamental problem.

If it is true that two methods are addressing the same, or at least quite similar, problems, then what is an appropriate way to compare their performance?  Let’s focus in particular on noncoding regions, which is our main interest (we are happy to concede that CADD will outperform fitCons scores in coding regions).

Your paper considered several validation methods, but, honestly, we found them rather unconvincing about noncoding performance.  Several of these were basically anecdotal and did not involve comparisons with conservation-based methods, which would likely have performed quite well in these settings (e.g., correlations with derived allele frequencies, single-locus results for MLL2, HBB, and TP53, correlations with expression changes in saturation mutagenesis experiments, exome-based ASD and intellectual disability analysis).  Another was more statistical but also included no comparison (analysis of GWAS hits).  As far as we could tell, the only real statistical comparisons with other methods were all based on ClinVar (as shown in your Figures 3 and 4), which is heavily skewed toward coding regions (by our accounting, 95% of pathogenic ClinVar variants overlap CDSs).  To me, it is not surprising at all that CADD would outperform standard conservation scores in coding regions.  Indeed, it is perhaps somewhat surprising that it does so by so slim a margin!  In any case, we could quibble about how informative or noninformative these validation experiments are, but I think you’ll agree that they do not provide a compelling demonstration that CADD outperforms other measures of “pathogenicity” (or selection) at transcription factor binding sites and other regulatory elements.

[NB: Greg pointed out in response to this letter that they did actually compare CADD with conservation-based methods on several of these benchmarks, and found an improvement; results are reported in supplementary tables]

So we need a more direct measure of performance in regulatory elements.  We have the problem, of course, that there is quite limited data on the specific pathogenic effects of regulatory variation.  Our proposal was to use what we see as the next best thing: reasonably high-quality predictions of cis-regulatory elements.  Our assumption is that most mutations that fall in these elements will be disruptive and hence pathogenic.  It is not a perfect assumption, but we would argue that this the best we can do at present, and it is certainly a more direct and general evaluation of noncoding pathogenicity than any considered in the CADD paper.

Now, what of your criticism that the resolution used in these tests favors fitCons over CADD?  There is some merit to this argument if in fact CADD is effectively differentiating among, say, informative and degenerate positions in transcription factor binding sites (which, I want to reiterate, remains an unproven assumption).  However, even if this is true, I do not believe the penalty to CADD will be as great as you imagine.  First, I don’t think this argument is relevant for our eQTL tests, which apply to individual nucleotides.  Not all of these eQTLs are indeed causal loci, of course, but many of them are, and false positive eQTLs shouldn’t alter the relative performance of the different scoring systems.  Notice that the fitCons scores outperform CADD scores in eQTLs by nearly as high a margin as in the other elements.  For the TFBSs, keep in mind that we are examining quite short binding sites (6-8 bp) under ChIP-seq peaks, after trimming off noninformative flanks of the motifs.  It is true that these TFBSs may contain some degenerate positions, but I don’t think they could possibly account for more than about a third of sites.  So in the best case, with perfect discrimination between informative and noninformative positions within binding sites, the sensitivity of CADD shown figures 5, S2, and S3 would rise by a factor of 3/2 if only informative sites were considered, which would not be sufficient for it to outperform fitCons.  (We actually quite like the idea of performing a version of this experiment that considers only informative positions in motifs and plan to add this to the manuscript soon.)  The Gerstein enhancers are admittedly a rather low-resolution set but keep in mind that a low density of functional sites hurts fitCons as well as helping it, because it means that the scores will tend to be lower and it will be harder to obtain good sensitivity without compromising specificity (that is the beauty of ROC plots).  So even if it is not a great validation set, I don’t think that it is unfair to consider it.

[NB: We did in fact carry out follow-up experiments with eQTLs and high-information-content positions in TFBS and found very little change in our results, as reported in the published manuscript.]

Claim: CADD is not biased toward coding regions

Response: We can only speculate on this issue, of course, but I think there is a problem with the way you fit the CADD model to the data that is likely to make it perform considerably better in coding regions than in noncoding regions.  As I understand it, CADD is trained by a global strategy, in the sense that a single set of parameter values are selected (for a given choice of training set and generalization parameter) to obtain an optimal fit, on average, across all examples in the training set.  Thus, if it is true that different covariates are relevant in coding and noncoding regions, as we expect, then the method will have to make tradeoffs between these types of sites.  If the “signal” for training (i.e., the contribution to the SVM’s objective function) is stronger in one type of region than another, it is likely that the tradeoff will favor that type of region.  Because constraint will be strongest, on average, in coding regions, leading to higher rates of difference between simulated and observed variants in these regions, it seems likely that these regions will indeed dominate in the training procedure, which will tend to make CADD sacrifice performance in noncoding regions in order to improve performance in coding regions.

Author post: Probabilities of Fitness Consequences for Point Mutations Across the Human Genome

(This article originally appeared as a guest post on Haldane’s Sieve, dated September 12, 2014.  It is re-posted here for completeness.)

This guest post is by Adam Siepel on his group’s paper Gulko et al. Probabilities of Fitness Consequences for Point Mutations Across the Human Genome, bioRxived here.

Four Genomicists in a Subaru

The idea for this paper emerged during a long drive across New York State, from Cold Spring Harbor to Ithaca, after the 2011 Biology of Genomes meeting. Two postdocs in my research group, Ilan Gronau and Leo Arbiza, were riding with me in my old Subaru, trying not to express too much alarm at my distracted driving. Also with us in the car was Ran Blekhman, who was at the time a postdoc with Andy Clark. (Ran is now an assistant professor at the University of Minnesota.)

Our conversation turned to important open questions in computational genomics, and in particular, to ways of making better use of the vast quantities of functional genomic data being pumped out of projects such as ENCODE. At the time, Ilan, Leo, and I were thinking a lot about how to use patterns of within-species polymorphism and between-species divergence to shed light on the influence of natural selection on regulatory sequences. Along these lines, we had spent much of the spring developing our new INSIGHT method [1,2], and we had just presented this work for the first time at Biology of Genomes. Ran, however, was pushing us to think less about abstract evolutionary questions and more about genomic function and disease association. He made a strong case that the biggest obstacle to progress in medically related human genomics was the absence of adequate functional annotations in noncoding regions of the human genome.

For a while the conversation went in circles, as we grasped for ways of measuring “functional potential” across the genome that would make use of genomic data yet be grounded in evolutionary theory. Then it suddenly dawned on us that we already had in hand a key piece of what we needed. The INSIGHT program was designed to estimate, for any collection of nucleotide positions across the genome, the fraction (denoted ρ) of those positions that were directly influenced by natural selection, in the sense that point mutations at those positions tended either to increase or to decrease the fitness of an organism. We realized that an alternative way of interpreting ρ was as a probability that the nucleotide at each position in the analyzed collection influenced fitness, assuming exchangeability of sites (as INSIGHT does).

All that we needed, therefore, was a general way of partitioning nucleotide positions from across the genome into distinct classes that were reasonably homogeneous in their functional roles. We could then estimate ρ for each class using INSIGHT, and assign to each genomic position the estimate of ρ for the partition to which that position belonged. This procedure would produce a “score” across the genome that looked roughly like widely used evolutionary conservation scores, but instead of representing local divergence patterns across the mammalian phylogeny, the score at each position would be estimated from groupwise patterns of polymorphism and divergence and would be directly interpretable as a probability of fitness consequences. Later Ilan would dub these “fitCons” scores, to emphasize this fitness-related interpretation. (“FitCons” also nicely parallels “phastCons,” our first conservation-scoring method.)

Because INSIGHT measures selection on recent time scales, fitCons scores would circumvent a major shortcoming of standard evolutionary conservation scores—that they require functional roles to have remained consistent over very long evolutionary time periods (tens to hundreds of millions of years) in order to be detectable in divergence patterns at individual sites or small loci. In principle, fitCons scores should be able to detect selection (hence potential function) at sites whose functional role had emerged quite recently, perhaps even along the human lineage.

The Problem of Grouping

The piece that was still missing in our plan was a particular scheme for grouping together similar genomic sites from across the genome. We did not get to the point of working this problem out in any detail during our revelatory drive to Ithaca, and, as it happened, it took several more months to settle on a solution. By this time, a Ph.D. student from Computer Science, Brad Gulko, had joined the project and assumed the lead in implementing a prototype of the scores.

At first, Brad, Ilan, Leo, and I spent some time thinking about fancy algorithms for clustering genomic sites that would consider functional and evolutionary information jointly. However, it did not take long to realize that this was a hard problem. Eventually, we decided to move forward with a simple grouping scheme, based on functional genomic data alone. This would allow us to cluster genomic sites in a pre-processing step and avoid the need for an iterative solution. Our hunch was that the scores would not be too sensitive to the grouping scheme as long as it was reasonable. As we discuss in our article, it may be worthwhile to revisit this clustering approach eventually, but it appears to be adequate for our current purposes. (I hope to convince Brad to discuss some of the technical issues with the clustering problem in an upcoming blog post.)

Relevance to the “Share Under Selection” in the Human Genome

By the fall of 2012, we had finally settled on an initial set of fitCons tracks and were beginning to observe decent prediction performance for cis-regulatory elements, when the human genomics community was thrown into a frenzy by a deluge of publications and accompanying press releases from the ENCODE Consortium. This event led to the now-famous controversy over what fraction of the genome is truly “functional” and whether ENCODE’s measures of “reproducible biochemical activity” (which apply to over 80% of the genome) were comparable in any meaningful way to the “share under selection” (SUS) estimated from comparative genomics (which generally came out to 5–10%).

I do not wish to rehash the familiar terms of this debate here, but I do want to focus on one aspect of it that was particularly relevant to our work. Many of the criticisms of ENCODE reminded readers that comparative genomic analyses pointed to a SUS of ~5–10%, suggesting that 80% might be a gross over-estimate of the functional content of the genome. However, others pointed out that these comparative-genomic estimates applied only to the fraction of the genome that had been under long-term selective constraint, because evolutionary turnover of functional elements—if it occurred at appreciable rates—could bias estimates based on long-term genomic divergence substantially downward. (For the latest chapter in this saga, see a recent paper by Gerton Lunter, Chris Ponting, and colleagues [3].)

We realized that the fitCons scores could help address aspects of this controversy, because they were based on patterns of variation over much more recent time scales and should therefore be much less sensitive to turnover than scores based on divergence patterns across the mammalian phylogeny. Moreover, the fitCons scores, by making use of INSIGHT to interpret patterns of polymorphism and divergence, might provide substantially better estimates of the quantities of interest than simple analyses of SNP densities or allele frequencies, a few of which had appeared among the ENCODE publications. Finally, the INSIGHT-based estimates are unique in that they directly predict the SUS, without the need for separate thresholding, mixture deconvolutions, or enrichment analyses.

Somewhat surprisingly, when we estimated the SUS based on fitCons scores, we obtained values (4.2–7.5%) that were quite similar to those based on conservation patterns in mammals. There are a number of tricky technical issues involved in this type of estimation—for example, concerning the corrections for local mutation rates and coalescence times—but violations of our modeling assumptions generally should tend to push our estimated upper bound (7.5%) to conservatively high values, implying that the true value is lower than 7.5%. In addition, the correction we have applied to obtain our lower bound (4.2%) is quite conservative, making it likely that the true value is higher than 4.2%. Therefore we have high confidence that the fraction of the genome under detectable selection from the available polymorphism and divergence data is indeed fairly close to 5%. As we discuss in the paper, it is important to bear in mind that the absolute values of these estimates reflect constraint on the identities of individual nucleotides only, and do not take into consideration higher order constraints, for example, on element lengths or spacing. Nevertheless, the similarity of the estimates based on mammalian divergence and human polymorphism suggest that evolutionary turnover has not produced a major downward bias in conservation-based estimates of the SUS.

Ilan and I soon realized that we could go a step further in this analysis and compare the fitCons-based estimates with parallel estimates based on the same functional categories but a measure of natural selection based on divergence only. The idea here was to perform a direct “apples to apples” comparison of the fraction of the genome under selection as measured on two different time scales: the 1–5 million-year time scale measured by fitCons and the ~30 million-year time scale measured by an analogous method based on divergence patterns in four primate genomes (human, chimpanzee, orangutan, and rhesus macaque), which we called “fitConsD” (the “D” is for “divergence”). I won’t attempt to describe this analysis in detail here, but our general conclusion is that the estimates of selection are highly similar on these two different time scales, suggesting further that evolutionary turnover has not had a dramatic effect on the functional content of the human genome over the past 30 million years or so. It is worth noting that Lunter and colleagues’ recent analysis is not strictly incompatible with ours (they estimate 7.1–9.2% constraint at present and focus on turnover over longer time scales) but their qualitative interpretation suggests large amounts of turnover, while ours suggests modest amounts.

Scooped by CADD… or Perhaps Not

As grant proposals, other manuscripts, and job searches led to delays in writing up our work through 2013, we began to hear rumblings on social media about a method called CADD, developed by Greg Cooper and Jay Shendure’s groups, that sounded alarmingly similar to fitCons. Then, in early 2014, a paper by Kircher, Witten et al. describing CADD appeared in Nature Genetics [4]. When we saw this paper, our initial impression was that we had been scooped by sitting on a good idea for too long. CADD was described as a method that integrated functional and evolutionary data and produced a measure of “relative pathogenicity” across the entire genome, and it was motivated, in part, by its potential usefulness in noncoding as well as coding regions. The paper included several impressive-looking ROC plots in which CADD apparently outperformed conservation based methods such as phastCons, phyloP, and GERP by a significant margin. In addition, CADD made use of a support vector machine (SVM), which was potentially a highly flexible and powerful means for considering large numbers of covariates with arbitrarily complex correlations.

We decided, with a certain amount of dread, that we needed to add CADD to our empirical performance comparisons on putative regulatory elements. At the time, fitCons was showing clear advantages in predictive power for cis regulatory elements compared with conservation-based methods and a functional annotation database called RegulomeDB. FitCons had several potential advantages over CADD—for example, it made direct use of polymorphism data for prediction, it considered covariates in a cell-type-specific manner, and it avoided a need for brute-force simulations through its use of INSIGHT for inference—but we thought that the use of the SVM in CADD made it unlikely that fitCons could compete with it in a pure classification task. Nevertheless, Brad dutifully downloaded the CADD scores, added them to his experiments, and displayed curves for CADD in his ROC plots for three types of regulatory elements (ChIP-seq-supported transcription factor binding sites, eQTLs, and enhancer predictions based on chromatin marks).

To our surprise, fitCons significantly outperformed CADD in all of these tests. This was true for three different types of putative regulatory elements, and true whether or not we considered cell-type-specific test sets. In fact, CADD performed essentially no better than conventional conservation scores in these tests, in apparent contradiction to the results presented in the CADD paper.

A closer reading of the CADD paper revealed a possible explanation for these observations. While the method was motivated, in part, by its applicability to the entire genome, the authors’ validation experiments heavily emphasized coding regions. In fact, it appears that even the ROC plot for “genome-wide” results (Figure 3a in the paper) is actually based almost exclusively (>92% by our interpretation of the paper) on missense variants in coding regions. The experiments that included substantial numbers of noncoding sites, in turn, were much more indirect—for example, by showing correlations with derived allele frequencies (Figure 2), known disease-causing status (Figure 4), and changes in expression in saturation mutagenesis experiments at two enhancers and one promoter. It is possible to have correlations of this kind without having substantial predictive power for regulatory variants.

When Greg Cooper saw our initial preprint on bioRxiv, he raised two major objections to our validation experiments. First, he pointed out that we were measuring the sensitivity of the fitCons scores in terms of bulk coverage of elements, when those elements actually consist of a mixture of sites at which mutations are deleterious and sites at which mutations are neutral or nearly neutral (such as degenerate positions in transcription factor binding sites). This approach to measuring sensitivity may be overly generous to the fitCons scores, which are relatively “blocky” along the genome, varying little from one site to the next, in comparison to higher-resolution prediction methods that properly distinguish between functionally important and neutral sites within elements. Second, Greg pointed out that we were using a naive genome-wide background set for our eQTL, which did not properly account for the ascertainment scheme used for eQTL identification.

We felt that these were fair and reasonable criticisms, and needed to be addressed. Therefore, we revised our validation experiments to consider only high-information-content positions in transcription factor binding sites (a proxy for functionally important nucleotides), and to use a more appropriate control for eQTL. The details of these follow-up experiments are described in our revised preprint (now on bioRxiv), but the bottom line was that they had almost no effect on our ROC plots. In other words, the apparent performance advantages of fitCons over CADD and other divergence-based methods is not an artifact of our experimental design but appears to reflect real advantages of the method. While Greg is correct that the coarse, “blocky” nature of the fitCons scores is a limitation of our current methods, the method still appears to perform significantly better than any competing method in distinguishing putatively functional regulatory nucleotides from background sequence. In other words, while scores that exhibit more variation from one nucleotide to the next—such as CADD, GERP, and phyloP—may appear on the surface to have higher predictive resolution, much of that variation is uninformative about regulatory function, and, on balance, the “blocky” fitCons scores are more useful in prediction.

We have spent some time trying to understand the differences in performance between fitCons and CADD, and believe we have some insights into why fitCons performs significantly better on regulatory elements. (What follows are our conjectures only; the authors of CADD do not agree with our analysis.) While the SVM in CADD is potentially a strength, we believe that it is substantially limited in this case by the use of a linear kernel and by pooling features across cell types, rather than focusing separately on each cell type of interest. In addition, we think there is a fundamental problem with the optimization scheme used by CADD. The SVM in CADD is trained by a global strategy, in the sense that a single set of parameter values is selected (for a given choice of training set and generalization parameter) to obtain an optimal fit, on average, across all examples in the training set. Thus, if it is true that different covariates are relevant in coding and noncoding regions, as expected, then the method will have to make tradeoffs between these types of sites. If the “signal” for training (i.e., the contribution toward the SVM’s objective function) is stronger in one type of region than another, it is likely that the tradeoff will favor that type of region. Because constraint will be strongest, on average, in coding regions, leading to higher rates of difference between simulated and observed variants in these regions, it seems likely that these regions will indeed dominate in the training procedure, and this may explain CADD’s superior performance in coding regions and its weaker performance in noncoding regions. FitCons avoids this problem by applying INSIGHT separately to each class of sites.

These observations raise the interesting possibility of a modified CADD that addresses some of these limitations. There is no reason why CADD couldn’t be trained separately on noncoding and coding regions, perhaps with different sets of covariates for each type of sites. Moreover, regulation-associated covariates could be treated in a cell-type-specific manner. A modified CADD designed along these lines (regulatory CADD, or rCADD?) could provide an interesting alternative to fitCons.


When we first discussed the idea for the fitCons scores during our drive across New York State three years ago, I envisioned a quick spin-off project that could be completed in perhaps half a year. As so often happens in research, several unanticipated challenges arose in completing this work, but we also found unexpected opportunities to connect our analysis with important open questions in the field. In addition, we were stimulated to think about the problem of combining functional and evolutionary data in new and deeper ways by another paper that addressed a similar problem but in a fundamentally different way. The end result is a paper I am quite proud of—one that provides what I think will be a useful resource to the genomics community and that also offers new insights into longstanding evolutionary questions.


[1] Gronau, I., Arbiza, L., Mohammed, J., & Siepel, A. (2013). Inference of Natural Selection from Interspersed Genomic Elements Based on Polymorphism and Divergence. Molecular Biology and Evolution. doi:10.1093/molbev/mst019

[2] Arbiza, L., Gronau, I., Aksoy, B. A., Hubisz, M. J., Gulko, B., Keinan, A., & Siepel, A. (2013). Genome-wide inference of natural selection on human transcription factor binding sites. Nature Genetics, 45(7), 723–729. doi:10.1038/ng.2658

[3] Rands, C. M., Meader, S., Ponting, C. P., & Lunter, G. (2014). 8.2% of the Human genome is constrained: variation in rates of turnover across functional element classes in the human lineage. PLoS Genetics, 10(7), e1004525. doi:10.1371/journal.pgen.1004525

[4] Kircher, M., Witten, D. M., Jain, P., O’Roak, B. J., Cooper, G. M., & Shendure, J. (2014). A general framework for estimating the relative pathogenicity of human genetic variants. Nature Genetics. doi:10.1038/ng.2892

Our Paper: Genome-wide inference of ancestral recombination graphs

(This article originally appeared as a guest post on Haldane’s Sieve, dated July 2, 2013.  It is re-posted here for completeness.)

This guest post is by Adam Siepel (@asiepel) on his paper with Matthew Rasmussen (@mattrasmus): Rasmussen and Siepel “Genome-wide inference of ancestral recombination graphs” , arXived here.

Inference of Ancestral Recombination Graphs

As the title indicates, our paper is about the problem of inferring an “ancestral recombination graph,” or ARG, from sequence data.  This is a topic that may strike many readers as impenetrably obscure and technical, so I will first try to explain, in plain language, what the ARG describes and why it has so much potential to be useful in many kinds of genetic analysis.  Then, I will tell the story of how I and members of my research group have become increasingly fascinated by this problem over the years, how we have struggled with it, and how we finally achieved the conceptual breakthrough that is described in our paper.  As will become evident, Matt Rasmussen, a former postdoc in the group and lead author of our paper, was central in this achievement.

What is the ARG?

The ARG is an elegantly simple yet superbly rich data structure that describes the complete evolutionary history of a collection of genetic sequences drawn from individuals in one or more populations.  It was invented in the mid 1990s by the mathematicians Bob Griffiths and Paul Marjoram.  The ARG captures essentially all evolutionary information relevant for genetic analysis of such sequences.  Statisticians say that it fully defines the “correlation structure” of the sequences, meaning that it explains most similarities and differences among the sequences in terms of their patterns of shared ancestry.

The ARG is something like a family tree, only richer, because it not only defines the relationships among individuals, but it also traces the histories of specific segments of DNA sequences.  For example, if you were to replace your family tree with an ARG, you could tell exactly which pieces of your genome came from your eccentric great grandmother and which pieces you share with your charming, intelligent, and handsome third cousin.

Now, this is obviously all a bit vague and superficial.  What does the ARG actually describe?  First, the ARG is a “graph” in the mathematical sense, meaning that it consists of a set of “nodes” and a set of “edges” between nodes.  Traditionally, nodes and edges are depicted with circles and connecting lines, respectively, making graphs look a bit like those molecular modeling kits you may have used in your high school chemistry class.  In the case of the ARG, there are two types of nodes, representing two key classes of events in the history of the sequences: (i) recombination nodes, which describe how a sequence is assembled by concatenating two ancestral sequences when DNA is mixed-and-matched during the production of sperm and egg cells or other gametes (meiosis); and (ii) coalescence nodes, which describe how two sequences trace back to a common ancestor. Edges between these nodes represent lines of descent, or lineages, for segments of DNA.  Each node is annotated with the time of the corresponding event.  In addition, for reasons that will become apparent in a moment, recombination nodes are annotated with the position along the sequences at which the ancestral sequences were cut and then joined together.

The significance of recombinations and coalescences comes from the fact that these are the two ways in which lineages can join or split over time.  The best way to understand them is to think about the behavior of lineages as one looks backward in time.  The graph is typically laid out with time on the vertical axis, so that the bottom of the graph represents the present time and each node is assigned a height above this baseline indicating the time before the present at which the associated event occurred. Therefore, to look backward in time, we look upward in the graph.  As we do so, we see that recombination events cause a single lineage to split into two ancestral lineages (representing the two sequence fragments that were joined together by the recombination in forward time), and coalescence events cause two lineages to join into one. Therefore, recombination nodes have one edge coming in and two going out, and coalescence nodes have two edges coming in and one going out.  One way of thinking about it is that, given a fragment of modern DNA, recombinations have the effect of increasing its set of ancestors, while coalescences have the effect of decreasing its set of ancestors.

All of this joining and splitting results in a complex network that, to the uninitiated, might vaguely resemble a map of the London Tube.  For the ARG-literate, however, this graph is richly informative about evolutionary history.  The key to interpreting the graph is to make use of the positional information associated with the recombination nodes (the positions at which the recombinations occurred). For any position along the sequence, one can extract an evolutionary tree, or genealogy, for the samples in question, simply by following their lineages upward on the page and taking a left or a right turn at each recombination node based on whether the genomic position in question falls to the left or the right, respectively, of the recombination event.  It is easy to see that a subgraph extracted in this way will contain only joins (coalescences) and no splits, and must eventually reduce to a single lineage, and therefore will define a tree.  Thus, the ARG has embedded within it a “local tree,” or genealogy, for every position in the sequence!

I have to pause here to say that I find this property of the ARG incredibly beautiful.  In a highly compact manner, the graph manages to capture both a complete record of all recombination events and a genealogy for every position in the sequence, and it does so in a way that shows how neighboring genealogies can be reconciled in terms of historical recombination events!  Figure 1 of our paper illustrates it with a simple ARG for four sequences.  Don’t be confused by the absence of explicit nodes (circles) in the diagram.  By tradition, nodes are not drawn as circles in ARGs, but are simply understood to be the points at which edges meet.

It is worth emphasizing that this representation is very general.  It can be used to describe the history of a particular gene of interest in individuals from a single well-defined population, or the history of whole genomes (with one ARG per chromosome) for individuals from many diverse populations.  It can even be used to describe the histories of sequences from representatives of different species, such as humans, chimpanzees, and gorillas.  As long as the sequences in question are orthologous and collinear—meaning, essentially, that they are derived from a common ancestral sequence in the absence of duplication and rearrangement events—then the coalescence and recombination events defined by the ARG are sufficient for describing precisely how the sequences derive from their common ancestor, and, hence, how they are correlated with each other.

Why is the ARG useful?

Many biologists are familiar with the value of “tree-thinking” in understanding evolutionary relationships.  For example, a diagram of a tree intuitively and precisely captures the fact that humans and chimpanzees are more closely related to each other than either humans or chimpanzees are to rhesus macaques.  Similarly, it explains why even neutrally evolving sequences (“junk DNA,” if the term is not too loaded) will tend to be more similar between humans and chimpanzees, and between rats and mice, than between humans and mice (this is essentially what we mean by “correlation structure”).  Against the background of neutral evolution, trees help us to identify genes and noncoding functional elements, to detect evolutionarily conserved and adaptively evolving sequences, to date speciation events, and so on.

The problem with trees on population genetic time scales, however, is that they change along the sequence, due to recombination.  As noted above, the ARG precisely describes these trees and the way they change.  Therefore, it enables tree-thinking with population genetic data.

Viewing population genetics in terms of the ARG can clarify one’s thinking about many problems of interest.  For instance, the ARG makes it clear that divergence times for genetically isolated populations can be estimated by looking across the ARG for the most recent coalescences that cross population boundaries.  Similarly, given an estimated divergence time, the rate of gene flow or migration between populations can be estimated, in a fairly straightforward manner, in terms of the rates of inter-population coalescence events across the ARG.  Ancestral effective population sizes can be estimated from the density of coalescence events in the ARG over time. Signatures of natural selection, including hitchhiking and background selection, can be detected by various kinds of local distortion of the ARG.  In general, the ARG provides a unifying framework for the field, and many challenging statistical problems in population genetics can properly be seen as problems of revealing relevant features of the ARG.

What would a reconstructed ARG mean in practical terms?  First, I should be clear that we have no intension of actually drawing an ARG for dozens of complete human genome sequences.  Such a drawing would be far too large and complex to be useful.  Rather, the value of a reconstructed ARG is as a rich data structure that could be interrogated for many features of interest, such as local trees, recombination events, mutation ages, or regions of identity by descent.  Because these features would be derived from a unified description of the evolutionary history of the sample, they would be guaranteed to be internally consistent, unlike ones based on simpler estimators.  In this way, the ARG would be useful in many problems of interest in statistical genetics, ranging from demography inference (e.g., estimation of population divergence times or rates of gene flow between populations), to the detection of regions influenced by natural selection, to the detection of genotype/phenotype associations.

Why is it so difficult to find a good ARG?

In practice, most population geneticists do not work with ARGs, but instead work with surrogates such as principle components, site frequency spectra, and spectra of identity by descent.  The reason people work with these simpler, lower-dimensional summaries of genetic data, of course, is that explicit ARG reconstruction is forbiddingly difficult.  From a statistical and computational perspective, there are two major issues in reconstructing the ARG.  First, the problem of searching all possible ARGs for one that best fits the data is computationally intractable, even in a restricted, parsimony-based formulation of the problem (it belongs to the class of problems computer scientists call “NP-hard”).  Second, and perhaps more importantly, in most cases of interest there is simply not enough information in the data to reconstruct a single ARG with high confidence.  Rather, in general, a large family of ARGs will be more or less equally compatible with observed sequences.

For these reasons, it would be misleading to suggest that there is any hope of producing a magical computer program that will allow the user to input a collection of sequences and obtain the true ARG for those sequences as output.  Instead, we must consider many possible ARGs, weighting them in some way by their plausibility.  In other words, we must consider a statistical distribution of ARGs given the data.

Because of the awkwardness of the space of ARGs (each ARG is a complex, combinatorial object, difficult to summarize in terms of low-dimensional features), we and others have come to the conclusion that the best way to get at these distributions is by making use of statistical sampling methods.  In our case, we use an approach, called Markov chain Monte Carlo (MCMC), that chooses samples that are guaranteed to be representative of the distribution of ARGs given the data and the model, provided the sampling program is run long enough.  After collecting fairly large numbers of samples, we can make useful statements about general features of the ARG even if we have limited confidence in each individual sample.  For example, the average of the times to most recent common ancestry (TMRCA) in the sampled ARGs at a particular position along the sequence can be used as an estimator of the true TMRCA at that position.  We show that our methods can be used to summarize various useful features of this kind, including recombination and coalescence rates, and the ages of mutations that are polymorphic in the sample, as well as TMRCAs.

How we became interested in the problem of ARG inference—or, the evolution of an obsession

The ARG seems to have a unique ability to inspire obsession in a particular kind of mathematically inclined geneticist.  As discussed above, it is potentially richly informative but extremely difficult to obtain, which makes it a natural “Holy Grail” for evolutionary genetics.  At the same time, the ARG is fairly simple to describe, it is straightforward to simulate, and sequence data are strongly informative about many of its features, making solutions to the inference problem seem tantalizingly within reach.

I first encountered the ARG and the problem of ARG inference in 2005 when I picked up the book Gene genealogies, variation, and evolution: a primer in coalescent theory, by Hein, Schierup, and Wiuf.  The authors of this book had obviously been bitten by the ARG bug (in fact, the lead author, Jotun Hein had already been working on variations on the ARG inference problem for more than 15 years) and I found their enthusiasm contagious.  At the time I was primarily a phylogeneticist, new to population genetics, so the book was a perfect introduction to the topic for me.  It is written largely from a phylogenetic perspective with a strong emphasis on combinatorial analysis and algorithms and without a lot of intimidating population genetics jargon.  Not long afterward I got to know Jotun Hein at leisurely meeting in Barbados and became further interested in the topic through my conversations with him.

As time went on, and I attended more and more talks on population genetics (they are hard to avoid at Cornell!), I became increasingly convinced that the ARG provided the right framework for thinking about a wide variety of problems in population genetics.  While population mixture models, principle components analysis, copy-and-paste models, models of the site-frequency spectrum, and similar approaches clearly have their place, for a committed tree thinker like me, there is something deeply unsatisfying about the use of these techniques in evolutionary modeling and inference.  I had a strong feeling that it must be possible to do better by more directly modeling the true evolution process that gives rise to the observed sequences.

In 2009, I was invited to write a review article on primate comparative genomics forGenome Research.  This exercise led me to carry out a broad review of the literature and reflect on many of the problems of current interest in population genetics.  Perhaps inevitably, the ARG ended up playing a central role in my discussion of the field.  Not many people have cited this rather long-winded review but the act of writing it was extremely useful in clarifying my thinking about population genetics and the ARG.

Soon afterward, I began to try to interest students and postdocs in tackling the problem of explicit ARG inference in earnest, with an eye toward applications involving dozens or possibly hundreds of complete genomes.  I had a hunch that it should be possible to improve on existing methods by making carefully selected approximations and drawing from our standard bag of tricks for probabilistic modeling.  I did manage to convince a couple of students to work on the problem but progress was slow and limited, and they moved onto other problems before obtaining publishable results.

When Matt Rasmussen joined the group as a postdoc in 2011, he jumped at the chance to tackle the ARG inference problem.  It quickly became clear that Matt was an ideal match for the problem, given his background in molecular evolution and phylogenetics, his strong skills in algorithms and probabilistic modeling, and his programming talents.  He made early progress by setting up a forward simulator for ARGs, studying the properties of simulated ARGs in the presence of selection, and developing intuition about which features in the data were most informative for inference.

Still, the inference problem proved challenging.  Matt was focusing on an approach I had suggested involving the use of a small set of “representative genealogies” that were selected to span, as effectively as possible, the (infinitely large) space of all genealogies.  His plan was to use these genealogies to define a phylogenetic hidden Markov model, which could be used for approximate inference.  He had devised a number of clever heuristics for selecting these genealogies, for example, based on the collection of “splits” (bipartitions of sequences) implied by informative sites in the data, but it gradually became clear that this approach was limited in effectiveness and had poor scaling properties.  It was just too hard to pick a good set of representative genealogies.  Almost a year into the project, we faced a choice between settling for an approach that was clearly suboptimal or going back to the drawing board and beginning anew.

Sequence “threading”—exaptation of an idea

Our breakthrough came some time during the winter of 2012.  I remember the occasion vividly.  It was one of those rare moments when the outlines of a solution to a difficult puzzle suddenly come into focus at all once—as close to a mythical “eureka” moment as I have experienced.

Matt and I were reviewing possible solutions to the problem of selecting representative genealogies, when I was reminded of an idea I had discussed with a student a couple of years earlier.  Could there be a way to build the ARG up one sequence at a time, by “threading” an nth sequence into an ARG of size n–1?  This idea had intuitive appeal but the previous student and I had never been able to make it work with full genealogies.  However, Matt immediately picked up the idea and ran with it.  It fit naturally with the way he was representing the local genealogies and thinking about how they were related and he guessed that it ought to be possible to think of it as a problem of adding one branch to each local tree, position by position along the sequence.

As we discussed further how one might set up a solution to the threading problem, I was suddenly struck by a parallel to a problem my group had worked on several years earlier.  That problem had to do with gene prediction in a multi-species setting, where the exon-intron boundaries for genes were allowed to differ from one species to the next.  The approach we had settled on was to repeatedly sample the gene annotations for one species conditional on those for all of the others, using a probabilistic model called a phylogenetic hidden Markov model.  It turned out that this sampling problem could be elegantly solved by dynamic programming, using an algorithm for hidden Markov models known as the stochastic traceback algorithm.

Matt’s threading problem, it was evident, had the same essential structure, although it differed in many details.  Therefore, it ought to be possible to carry out the threading operation exactly, in time proportional to the sequence length, using the stochastic traceback algorithm.  Furthermore, it ought to be possible to structure this operation so that repeated applications would guarantee that we sampled from the true distribution of ARGs given the data and our model (i.e., so that we had a proper Gibbs sampler, in statistical jargon).  It was immediately clear to both of us that this was the right path forward.

As usual, there was a good deal of work in taking these ideas from white-board scratchings to a fully defined model and a working implementation.  Not least among these was the formidable task of generalizing the threading method to multiple sequences, which proved necessary for good MCMC convergence properties with more than a few sequences.  Matt was exceptionally diligent, resourceful, and creative in working through all of these steps, and eventually demonstrated that the method performs remarkably well, perhaps better than either of us anticipated.  Nevertheless, it was in that one exciting discussion about the threading operation that the project came into focus, making it clear that we had fundamentally new way of tackling this longstanding problem that was worth the additional months that would be needed to see it through to a full-fledged implementation.

Most evolutionary biologists are familiar with the concept of “exaptation,” popularized by Stephen Jay Gould and Elizabeth Vrba.  Exaptation refers to the process by which evolution takes a phenotypic characteristic developed for one purpose, and co-opts it for use in another way.  Examples include the gas bladders used for buoyancy in some fish (derived from lungs) and the use of bird feathers for flight (thought to have evolved for temperature regulation).  The phenomenon, of course, is also common in the evolution of ideas.  To me, the threading idea was a perfect example of an algorithmic exaptation, the reuse of an idea developed for one problem in evolutionary sequence analysis and co-opted for use in another.  In this case, the threading operation proved considerably more powerful and useful for ARG inference than for gene prediction.

Concluding notes

In this post, I have tried to convey some of my excitement about genome-wide ARG inference, a topic that might strike the casual reader as dry and obscure.  Space does not allow for a discussion of all of the possible ways in which we are thinking of making use of Matt’s ARG sampling program (called ARGweaver), but our manuscript does include a fairly lengthy Discussion section, which lays out some of our ideas for future work.  Clearly there is much still to be done, but I am convinced that this is going to be an enormously powerful tool for population genomic analysis.

Finally, I should say that, while I have focused on our own efforts in this post, we are far from the only researchers actively working on this topic.  As described in our paper, there is a great deal of closely related recent work by Yun Song, Richard Durbin, Thomas Mailund, Asger Hobolth, Mikkel Schierup, and others.  In short, there is no shortage of opportunities for exaptation of ideas in this vibrant field.