Resources and notes from my readings on science, growth and "progress" (or rather applied progress studies)
Science of Science
Articles investigating the inner workings and the problems afflicting modern science and academia:
- How Life Science Actually Work - Alexey Guzey (2020)
Examples of case studies that succeeded in accelerating science:
- Goal: Exceed the fundamental limits of existing propulsion by further advancing physics to discover the breakthroughs that could revolutionize spaceflight and enable interstellar voyages.
- Operating principles:
- Reliable. Define success as gaining reliable knowledge rather than claiming breakthroughs (puts emphasis where needed).
- Immediate. Focus on immediate "make-or-break" issues
- Iterated. Gain knowledge via short-term incremental tasks
- Diversified. Support divergent research not just hot topics
- Measured. Track applicability and progress with the cycles
- Impartial. Research is judged via peer assessments, using the reliability of results not the feasibility of the concept as metric
- Empirical. Experiments over pure theory
- Published. Today we would say: Open source the results (pro and cons)
- Incentives and Creativity: Evidence from the academic life sciences (Azoulay, Graff Zivin, Manso 2011) shows that Howard Hughes Medical Institute (HHMI) investigators (who are funded for a longer term and in a more open ended way) outperform those of the National Institutes of Health (NIH) that have shorter review cycles and more concrete grant proposals.
- However, the effect sample reported is huge, perhaps too good to be true, and the extent to which this model can scale is debatable, as the author himself also says that it is not clear how well this model can scale. HHMI is very much focused on a narrow elite.
- This supports the thesis of fund people not projects
- Fund projects: A scientist proposes to do a specific, well defined project. There is an expected deliverable and there may be milestones involved that have to be met. Rather then being open ended and highly uncertain, it is roughly known to be achievable. The project should be ended if it's not meeting its goals.
- Fund people: A scientist is funded generously and have all their requirements for assistants and equipment met without paperwork or further reviews. They have zero accountability, and what they decide to research or publish has no bearing on the continuation of funding. In effect they have life tenure. They don't perish if they don't publish.
- It's clear that there are advantages and disadvantages to both approaches; they are the explore/exploit tradeoff with another name. The first one is similar to the Focused Research Organization concept.
- With DP1 and HHMI, researchers do not submit a grant proposal for something specific they want to do, they submit their resume and a broad summary of what they are going to be working on.
- The reason why there may be personal factors involved may be that "how good of a researcher someone is" is hard to observe from their published work and related citations.
- The Azoulay paper tries to do is compare HHMI investigators with a control sample from NIH-funded researchers that were awarded early career prizes. Their control sample is additionally tweaked to make it resemble more like the kind of profile that would get into the HHMI. And finds:
- Overall publication output increases by 39% relative to non-HHMIs
- Highly cited research jumps by 96% relative to non-HHMIs
- Relative to themselves prior to the award, HHMis publish more
- HHMIs' fraction of low-cited works increases
- HHMIs tend to use more novel keywords relative to control
- HHMIs tend to use more novel keywords relative to themselves prior to the award
- HHMIs' work is cited by a more diverse set of journals, relative to non-HHMIs
- HHMIs' work is cited by a more diverse set of journals, relative to themselves prior to the award
- Policy-makers might consider adopting a portfolio-based mindset that seeks a diversity of approaches, while accepting that high failure rates for individual projects is in fact part of an effective grant-making program
- Proves that more rigourous screeining of project = better outcomes
An evergreen resource to understand the current speed of science might be Wikipedia's constantly up-to-date year in science
- "Across a broad range of case studies at various levels of (dis)aggregation, we find that ideas — and in particular the exponential growth they imply — are getting harder and harder to find."
- The manifesto of progress studies.