Monday, May 22, 2017

The zombie grant


Not to the swift, the race: 
Not to the strong, the fight: 
Not to the righteous, perfect grace: 
Not to the wise, the light. 

But often faltering feet
Come surest to the goal; 
And they who walk in darkness meet
The sunrise of the soul.


Excerpt from ‘Reliance’ by Henry Van Dyke

They say ‘good things come to those who wait’. I’m writing today to say that this is at least sometimes true. We waited. And waited. And waited. And something really good, that we are really proud of, has finally arrived. To be more precise, Katie Peichel and Andrew Hendry and I wrote a grant (2009), and rewrote it (2010), and rewrote it (2011), and rewrote it (2011), and were all set to give up when we got funded. Then we hired [postdoc Yoel Stuart, most significantly], planned, did the field work, collected the data, analyzed the data, wrote the paper. We bounced it across several good journals. And now, the core paper from the study has come out. We are pretty excited, but also a bit stunned by the start-to-finish time. To put this in perspective, the idea for this project was conceived when my first child was this old:
Me, with more hair and a 4-month-old, just before the Evolution meeting in Minnesota in 2008 where we hatched our plan
and I had a lot more hair. The resulting paper was born nine years later when she was old enough to play piano, build her own spectroscope, and get 1st place in the Austin Regional Elementary School Science Fair for figuring out the elemental composition of the sun:
That 4-month old can now build a homemade spectroscope and interpret Fraunhaufer Lines to infer elemental composition of the sun. This picture was taken a couple months before our paper was accepted.

Grad students, maybe that helps you feel like your PhD isn’t taking so long now, after all.

My goal for this blog post isn’t to delve into the science of our paper: you can read the paper, or read Yoel Stuart’s blog post on Nature.com  for that. My goal is to tell the story of the process. A story of perseverance that maybe will make you feel better when you get that 4th grant proposal submission rejected yet again (for this, also see the Heard-Hendryblog on serial rejection). And maybe a broader lesson about the glacial pace of science and what we might be able to do about it.

There’s a recent trend towards Zombie remakes of classic literature. I’m thinking specifically of Pride and Prejudice and Zombies, a Zom-Com-Rom take on Jane Austen’s classic book. In several real senses, this is a Zombie story too. A Sci-Zom. You’ll see why soon.


I. Zombie films sometimes involve some initial chance encounter
Andrew Hendry and I had been doing field work on stickleback in the same general area of Vancouver Island, for over 5 years, when we finally crossed paths in 2006.

The field crew gathering in 2006. Left-to-right: Kate Hudson, On Lee Lau, Dan Bolnick, Andrew Hendry (crouching), Ann-Catherine Grandchamp, Jean-Sebastian Moore, Daniel Berner (crouching),  Credit: Tania Tasneem (not pictured)    
His crew showed up at the cabin we rent on Vancouver Island (Roberts Lake Resort), and crashed on our floor for about a week. We got a bottle of good scotch in return, which Andrew drank most of (then proceeded to build a duct-tape spiderweb in the middle of the cabin at 3 AM). Between doing our respective field tasks, we spent a lot of time playing McGill-vs-UTAustin ultimate games, and talking science late into the night. We came up with some grant proposal idea, sketched out on the wobbly dining table, wrote it up that fall, submitted it in early 2007, and got it soundly rejected later that year. The wobbly table may have been a metaphor. Oh well. I don’t even quite recall what we proposed to do. Something about a continuum of ecological speciation, or lack thereof.


II. The main characters hatch a plan to go to the woods. Always a bad idea in Zombie stories.
Before the 2008 Evolution meeting in Minneapolis, I emailed Katie Peichel and Andrew Hendry to say that it would be fun to develop a collaborative project between the three of us. Something on lake-stream evolution that fused Katie’s genetics know-how, with Andrew’s experience with the lake-stream system. Not sure what I contributed, really.We met after the BBQ social and sat and talked for many many many hours. What emerged was a plan to make a plan. The general outline was clear: we wanted to know whether ‘parallel’ evolution of lake-stream stickleback was a reflection of parallel environments, driving parallel selection, causing changes at the same genes, leading to heritable lake-stream differences in the same traits, in each of many replicate lake-stream pairs (pictured). 
Comida Lake and its outlet stream. Photo by Thor Veen

We wanted to go big, with many lake-stream pairs. And we wanted to fuse QTL mapping, genome-wide association mapping, field measures of selection, quantitative genetics, and ecology. With this, we could ask whether the loci that diverge most strongly & repeatedly from lake to stream (GWAS) are also loci linked to traits (QTL) that are highly divergent (survey data) and under divergent selection in the wild (field selection experiment). Many different layers of data would be needed: environmental data, diet data, parasite data, trait data, functional performance measures, genotypes, and fitness. We left the Evolution meeting with two definite plans. First, we would meet again in December 2008 to write a grant. Second, Andrew’s postdoc Renaud Kaeuffer would genotype six lake-stream pairs at several hundred microsatellite loci, test for lake-stream differences in allele frequencies, and determine whether there is evidence for any genetic parallel evolution. This could be preliminary data for our vaguely-defined grant idea.

III.  The idyllic gathering in a guest house in the woods. Or rather, at a winery.
In December 2008 I was in San Francisco, looking after my 1½-year-old daughter while my wife attended the American Anthropological Association conference. At the end of the conference, Katie Peichel picked me up in a rental car and we drove up to Napa, to the Hendry Winery. Andrew was spending a sabbatical there, in the guest house on his uncle’s and brother’s winery. The three of us (plus Andrew’s postdoc Renaud Kaeuffer) spent three days there. We brainstormed the first day, expanded details the second, and wrote outlines and some text the third day. We worked hard all day, pausing for a long jog around the vineyard each morning. Around dinner time, we would receive a delivery of spare bottles of wine, left over from tastings earlier in the day. The best science of course happened in the evening after those deliveries. You might say, we were on a Mission.

Renaud Kaeuffer, Andrew Hendry, Dan Bolnick, Katie Peichel (left to right), in Napa in 2008
 By the end of the three days, we had the skeleton of a grant. I should also note that this vineyard meeting led to two other papers as well ***see footnotes.

Yesterday, I dug up some old files from those meetings: a five-page file I wrote out a couple weeks before our Napa meeting, to start the conversation, and a document I wrote during the meeting, summarizing our brain-storming effort. These contained the following seed of our subsequent work:

Is evolution repeatable and deterministic, or idiosyncratic and stochastic? This question, elegantly framed by Gould’s ‘tape of life’ quote, has remained a major puzzle for biologists. At its most expansive, the question is of course unanswerable, because we are faced with a single replicate of life's history on earth. However, we can begin to address Gould's challenge by shifting to smaller temporal and spatial scales, for which replication is feasible.

The key questions we identified were:
1)    To what extent is phenotypic divergence (between lake and stream stickleback) parallel between replicated lake-stream pairs (in different watersheds).
2)    To what extent is the genetic basis of this divergence parallel?
a.      To be addressed by:
                                               i.     QTL mapping
                                              ii.     GWAS across replicated clines
                                            iii.     Common garden assays of heritability
3)    To what extent is selection parallel across lake/stream pairs?
4)    How do mate choice and habitat choice contribute to lake-stream divergence?

After the meeting, I went home with a new outline and proceeded to write.  By January we had a proposal. I budgeted $560,000 for field experiments and genotyping (SNP array at the time) of many wild-caught fish. Katie budgeted $940,000 for doing a bunch of replicate QTL maps. Andrew would share in my budget for field work and a common-garden heritability study at McGill. We were pretty excited by the ideas, but felt that the budget was a bit big. We couldn’t see a way to get it smaller without compromising the science. After some worrying, we justified the budget as appropriate for the large-scale that we planned to do. We submitted the proposal in January 2009 (this was in the days before pre-proposals).

IV. In which the hero of our tale (the grant proposal) dies. Several times.
It was rejected.  I have the reviews on hand (1 Excellent,  2 Excellent/Very Good, 2 Very Good, 1 Very Good/Good). Basically, they all wanted us to do even more: more sampling, more lake-stream pairs, more environmental and phenotypic measures, and more genetic data. And it was too expensive. Do more, for less.

We resubmitted in January 2010. The budget was about the same. We proposed to do more, for less money. We got 2Excellent,  1 Excellent/Very Good, 4 Very Good, and 1 Good. Concerns this time mostly centered on our plan to do genomics across replicate clines, whether we were adequately measuring predation, and whether the cages for our field fitness experiment would alter the environment appreciably. All fair concerns, so we made some changes. In conversations with the program officer, it was also clear that two of us PIs were part of the trouble. I had both a Packard Foundation fellowship and a Howard Hughes Medical Institute Early Career Scientist position. Although my last NSF grant had expired three years earlier, they saw me as over-funded already. I ended up going 5 years without any federal grant funding, relying on private foundations instead.  And Katie had NIH grants which to NSF meant she was considered ‘rich’ as well. In fact, as a faculty member at the Fred Hutchinson Cancer Research Institute she was expected to bring in a lot of grants to pay her own salary, and was barely scraping by.

We resubmitted in January 2011. My budget went up to $722,000 (some things were more expensive by then, and we added some features to appease reviewers), Katie’s was flat at $940,000.  The reviewers still didn’t like the clinal genomics plan, and still were worried that field enclosures would modify flow regimes. For the first time, the panel specifically recommended that we split the proposal up, carving off some features for another grant. Said one reviewer, “Quite simply, this proposal is far too expansive and ambitious”. And again, private conversations made it clear that NSF saw us as not needing more money.  

Katie thought it wasn’t worth trying again. I insisted.  We resubmitted in July 2011, roughly the same budget. On this fourth try, our scores came back:  (5 Excellent, 1 Excellent/VeryGood).  Only one other time have I had such uniformly positive reviews (6 Excellents).  Reviewers wrote things like: “This is a tremendous proposal. The PIs are absolutely correct that no one has done anything like it.”.  “this is an outstanding proposal on all fronts 
and in my view it is not worth picking on reamaining minor 
details that may have been overlooked in order to try 
being negative. 
Related to the inlecctual [sic] merit, both the main objective and related specific objectives of this proposal are at the very forefront of the most pressing question in our field”. “tour de force”;  “This proposal will likely be the most extensive quantitative characterization of morphological parallelism/non-parallelism to date”.  Declined.  (incidentally, my 6-E CAREER proposal was also declined). Why? We had other funding. Now, I’m a pretty left-wing guy, and believe strongly in fair distribution of resources. But I also believe that a proposal with 5 or 6 ‘Excellents’ deserves funding. We were pretty demoralized, and decided that until we were research paupers, we probably couldn’t do this project. So we basically decided to dig a hole and bury it. Cue the violins. 


Our grant



V.  The zombie grant
Grant proposals, like zombies, sometimes emerge from the cold hard ground when the main characters of our story least expect it. Our fourth submission, which got such great reviews, was rejected in December of 2011. And we decided to leave it dead. But, on April 18th 2012 I got a phone call out of the blue from NSF. Two things happened. They found some money stuffed into someone’s old shoe, or that had fallen behind some couch cushions. Something like that. Apparently this is not uncommon. Also, they realized that Katie, despite her NIH grant, was in dire need of funding because of how FHCRC pays its faculty. These two factors brought the grant-corpse back to partial life. We were getting funded. Not our whole $1.7 million 4-year request, mind you, but just over half that. This is a zombie story, after all. Not a complete resurrection. But we weren’t about to protest at a $900,000 budget for a 3-year project. $450,000 for Katie, and the balance mostly to my lab but some as a sub-contract to Andrew. And just like zombies are missing limbs, jaws, etc, our partly revived project was missing some important parts because we had to slice out nearly half the budget. No more field measurements of selection (which reviewers were both critical of and especially excited by). No more GWAS surveys along replicated lake-stream clines (I did that anyway at a smaller scale with HHMI money; the first of several papers from that is Weber et al 2017 Evolution). Scaled-back common-garden rearing. Fewer QTL populations.
The funding started July 1 2012 (just after our usual field season, sadly). That first summer of funding, my soon-to-be graduate student Brian Lohman spent three weeks scouting out lake-stream pairs all over Vancouver Island, giving us a set of 16 study sites. In January 2013 we had a small working group meeting at my house in Austin, with Katie and Andrew, Rowan Barrett, Dieta Hansen, and an incoming postdoc Yoel Stuart, who was just finishing his PhD with Jonathan Losos at Harvard. Yoel would be the glue the bound the project together and the engine that made everything happen. We talked, ate great tacos, drank scotch, read childrens’ books, played Cards Against Humanity (Yoel won every time), and went for a climbing excursion to Reimer’s Ranch.


Andrew lectures Dan on some of the finer points of parallel evolution. 

Andrew climbing in Dan’s backyard bouldering gym.
Andrew and Katie climbing in parallel.  Get it? Parallel?    


The Austin 2012 meeting ended in a climbing trip at Reimer’s Ranch



 Yoel and I assembled a team of field helpers, and planned the field season. As field crew, he’d have PhD student Brian Lohman, K-12 teachers Tania Tasneem and Andrew Doggett, and undergraduates Rebecca Izen and Cole Thompson. Andrew and Katie contributed people as well (PhD student Dieta Hansen, Undergraduates Elena Motivans and Mingsha Zhou), and Rowan Barrett joined in for some field work. From Seattle, we were joined by Katie’s team including Matt Dubin, Susannah  Halbrook, and high school teacher Carole Tanner. There was also a team of 12 people from my lab doing a different project (another NSF grant that was also a back-from-the-dead funding situation in 2012). At our peak we had 28 people in the field at once. I was basically just a glorified travel agent that year.



The 2013 field crews (missing a few!). Left to right, approximately: Dan Bolnick, Andrew Hendry, Cole Thompson, Katie Peichel, Dieta Hansen, Yoel Stuart, Matt Dubin, Kelsey Jiang, Connor French, Hollis Woodard, Alicia sp., Brian Lohman, Amy Doan, Rebecca Izen, Chase sp., Racine Rangel,  Kim Hendrix (HS teacher), Travis Ingram, Carole Tanner (HS teacher), Susannah Halbrook,  Gina Conte, Rowan Barrett,  Ruger is the dog. Yes, as in the gun. He handles security at Roberts Lake Resort. The mountain lions respect him.    
 In the summer of 2013 Yoel and his team collected environmental data and stickleback specimens from 16 lakes, 16 adjacent streams, and 4 estuaries, yielding roughly 80 fish from each of 36 locations. This was a lot of hard work, schlepping traps and YSI monitors through thick forest and deep muck. It was not without incident. Cole Thompson, a University of Texas Undergraduate, punctured his foot when bush-wacking through the brush beside a remote lake, when a branch on a fallen tree penetrated the sole of his waders. Aside from that day in the emergency room, much fun was had.



Yoel's in a boat, y'all

That log isn't totally stable
Rebecca Izen processing stickleback in the truck bed to stay dry

Andrew tries out Dan's boat. It is fun.

Dieta Hansen catching fish

First day, training newcomers

It rains a lot there

Cluexewe Estuary campground is spectacular



Videos of field work:






            For the next two years, Yoel and a group of really dedicated undergraduates worked on collecting detailed data from samples of fish from each of the 36 sites, including >70,000 SNPs (via ddRAD, with help from Jesse Weber) on 24 fish per site,  86 morphological characters (fin shape, body shape, gill rakers, armor and spine traits, eye traits, jaw biomechanical traits, brain morphology), diet data, parasite data. They processed mud and water samples for environmental invertebrate abundances.
            Data curation was also a massive undertaking. The resulting dataset spanned site-level data (eg., elevation, surface area), microhabitat data taken at each trap (substrate, flow rates), morphology, diet, parasites, and genetics. Thor Veen joined the team when he contributed an excellent GitHub data structure and helped with analyses. Analysis itself took a long time. For instance, we recruited Mark Ravinet to contribute Approximate Bayesian Simulations to estimate population genetic history parameters (divergence time, migration rates, colonization order) for all 16 lake-stream pairs. That computationally intensive process took months.
Finally, we had results, and a paper. The usual rounds of edits, then we sent it off. Rejected (Nature).  Rejected (Science).  And finally to Nature Ecology & Evolution, where we went through three rounds of review including a remarkable >100-day wait for the second reviews after our first revision. It was finally accepted at Nature E&E, and came out today. Co-authors include two K-12 science teachers, and multiple undergraduates. Some of the basic science insights are summarized in a nice blog post, by Yoel Stuart, on Nature Ecology and Evolution's website.

A few weeks ago, I made corrections on the proofs to our paper. I’ve always enjoyed proofs. We set the paper aside for weeks or months for the journal to process, then get to see our work as if with entirely new eyes. And I was really proud of this work. It pulls together many ideas, and an absolutely massive amount of data, to ask some pretty cool questions. The basic punch-line is, there’s less parallel evolution in stickleback than you thought, and we can explain why, at least in part. If you want to know more than that, well… go read the paper (email me [danbolnick <> utexas <.> edu] if you can't access it). Look: it took us 9 years of dealing with zombies to deliver this to you, don’t tell me you can’t take half an hour to look it over. To put it in perspective, it is shorter than this blog post.

The project also spun off other papers as well. There’s an incomplete list at the end of this post (***footnote). Analysis and writing of the QTL work is ongoing, and there’s a lot more that we are pulling out of the genomic and phenotypic data. Look for more papers to come.


Looking back

To conclude, I want to reflect on some lessons learned.

First, science is really slow. From idea (early summer 2008) to paper (late spring 2017), was enough time for my infant daughter to grow big enough to read The Hobbit and to play Fur Elise on the piano (nicely, at that). Almost half that time was waiting for money. This bothers me**.

Second, I tend to do a shock-and-awe approach to grant writing: put in an ambitious (but in my mind genuinely feasible) agenda with many interlocking parts. That does not always serve me well, and reviewers wanted us to do something more modest in the end. They told us to cut, and we cut. This ultimately paid off, not just because we got funded. Remember, our requests ended up being $1,700,000, but we had to cut $800,000 off that to get our grant.  As a result, we sadly deleted what we thought was the most exciting component of the study, the field experiments to measure selection in multiple replicate lake-stream pairs. A missing arm of the grant-zombie, meant to test whether parallel phenotypic divergence really reflects parallel selection. Well, zombie arms have a habit of clambering out of the ground on their own, later on. My great postdoc Yoel Stuart turned that cut-out-idea into a new grant proposal. As our first grant came to an end, he got another $1,000,000 funded on his first try (somebody hire the guy!), to do what we had previously eliminated. Actually, it wasn’t funded exactly on the first try; this, too, was a back-from-the-dead proposal. That’s ongoing.

Third lesson: hire a great postdoc for your projects. Yoel Stuart handled a complex web of field, molecular, and morphological data collection and analysis, and wrote what I think is a genuinely good paper. As a corollary, hire Yoel for your faculty.
Dr. Yoel Stuart
Finally, persistence does pay off. Think about those reviews: we went from okay-ish reviews with a mix of good to Excellent, to a really strong set of reviews on our fourth submission. This project ended up being better, more focused, and probably more feasible, thanks to those anonymous reviewers (I’m looking at you, Brian Langerhaans) who worked hard to give us feedback through multiple rounds of submissions. Keep this in mind when you get that rejection from NSF this spring. Try, try again. “But often faltering feet Come surest to the goal;”


 -Dan Bolnick, May 5, 2017

Yoel contemplating evolution



Footnotes:
**
I’ve had several graduate students come up with great project ideas that we’ve turned into grant proposals. Twice, these ideas have been funded. But with lag-times like this, there’s almost no way that a grad student idea can be turned into a full NSF proposal, and from there to funding and work and papers, within the time-span of a PhD. That has huge policy implications to how we train students. If we have to support students as RAs (rather than teaching assistants), then students will typically have to work on my ideas, conceived years before they entered our graduate program, instead of their own ideas. That’s a problem, because my experience is that my students are more innovative than I am. We need rapid response funding for good student ideas that won’t take 4 years to get funded. NSF DDIG grants are a good attempt at that, but have such a limited budget that our students’ imaginations must be kept in check.


***
A selection of other papers from this project:

UT Austin undergrad Newaz Ahmed has a paper just out in Ecology & Evolution on brain morphology evolution in lake-stream stickleback.  Basically, no parallel evolution there. http://onlinelibrary.wiley.com/doi/10.1002/ece3.2918/abstract

UT undergrad Rebecca Izen worked on within-stream phenotypic variation

UT graduate student Kelsey Jiang published several papers on lake versus stream divergence in swimming behavior in flowing water:

UT graduate student Brian Lohman published papers on multivariate lake-stream clines:

UT postdoc Jesse Weber published a paper on clinal lake-stream divergence. This is basically a smaller version of what we had to cut out for budgeting reasons when our grant was cut in half. We did it with HHMI funds instead, and at least one more paper is coming from that dataset. For now, here’s the paper, and a digest about the paper:


Brian also has a paper in review and on BioRXiv about transcriptomic response to novel environments in lake and stream stickleback:

McGill PhD student Dieta Hansen got out a paper on the apparent lack of allochrony-driven reproductive isolation between lake and stream stickleback:

McGill PhD student Krista Oke has a paper on the role of phenotypic plasticity in lake-stream parallel evolution:

Krista also has a fantastic new paper out about parallel evolution in fish more generally:

I have a paper on MHC-parasite associations in three lake-stream pairs:

McGill postdoc Renaud Kaeuffer had a paper that emerged from our Hendry Winery meeting:
with a follow-up paper by Andrew:

The Winery meeting also led to this review paper:

I have a paper in press in Nature on the lack of (detectable) divergent selection in lake-stream stickleback
(embargoed)

UT undergrad  Cole Thompson has a revision in re-review at Evolution on evolution of complex biomechanical traits in the lake-stream pairs.

I’m sure I’ve missed a few papers that have been touched by this project, and more will be out in the coming year. We have several people actively working on more analyses and papers. Keep your eyes peeled!







2 comments:

  1. The counterbalance to “this costs too much” ought to be how many good papers are likely to result from the work. If you divide the size of the grant by the number of good papers that came out of it, suddenly it doesn't look so expensive any more! Also important is the number of people that ended up being supported by the grant. Big grants like this create a whole ecosystem around them; if it's a productive ecosystem making important and unique contributions, then it's worth the money.

    ReplyDelete
  2. Wish I was more confident that your experience is typical, Dan. But I wonder if the experience of another friend of mine is more typical. Getting invited for a full proposal on the first try and getting very good evals but not funded. Revising conscientiously, getting invited for a full proposal again, and not getting funded again, with reviews that suggested undoing some of the changes the first set of reviewers wanted. Trying a third time after further revision, and this time not even getting invited for a full proposal.

    Ok, in truth I'd be reluctant to generalize from any individual's experience with one grant. And your message that "everybody gets rejected, often" is well-taken. I'm just less sure about the "persistence pays" lesson. Presumably, sometimes it pays and sometimes it doesn't, and it seems like it can be hard to judge when it will pay and when it won't.

    ReplyDelete

Check your (taxonomic) biases at the door

Many of us like to believe that we are conceptually-oriented researchers; our particular study organism(s) are just means to an end, the en...