Evolution and Innovation
How should individuals think about creating technology?
In his book, The Big Picture, scientist Sean Carroll describes how different explanations are appropriate at different scales of thinking. For instance, describing a single water molecule in terms of its viscosity or surface tension is not really useful. Terms like covalent bonds and atomic weights are more relevant for a single molecule. In Carroll’s terms, the concept of viscosity is outside its domain of applicability when used to describe a single molecule. However, viscosity and surface tension are useful terms for describing an entire glass of water. Each set of descriptions creates a story that accurately describes reality, but only if used within the appropriate scale of thinking.
He writes, “The important takeaway here is that stories can invoke utterly different ideas, and yet accurately describe the same underlying stuff....Organisms can be alive even if their consitutient atoms are not.”
This essay borrows from Carroll’s thinking and applies it to another phenomena which also can be explored at different scales, not water, but innovation. In the same way that water can be described as a single molecule, or an entire glass full, innovation can be described at two levels, the macroscopic and microscopic.
Exploring innovation at the macroscopic scale means looking at the cumulative technological progress of an economy, or the innovation spurred by an investment fund. Exploring innovation at the microscopic scale means looking at innovation potential of the individual technologist. At the risk of belaboring the analogy, the individual technologist is the constituent ‘molecule’ of the entire ‘glass full’ of innovation, the economy or fund.
My motivations for writing about innovation at two scales is mostly one of curiosity, I think it is an interesting framing for understanding the process of creating technology. Will the explanations for innovation at a macroscopic scale make sense when tried at the microscopic? Or, like the case of water, are the terms and concepts completely different.
Macroscopic Innovation is Evolutionary
At the macroscopic scale, technology progresses via a Darwinian-esque process of trial and error. Darwinian evolution states that random genetic variations within organisms grant some of those organisms reproductive advantages, which are then passed down to the offspring. This process started when life started, and produced all of the diversity of living species via a process called natural selection.
In his book How Innovation Works, Matt Ridley writes that innovation, “like evolution, is a process of constantly discovering ways of rearranging the world into forms that are unlikely to arise by chance - and that happen to be useful.” Technological progress at this scale is a simple equation, progress increases in proportion to the number of attempts to find useful ‘ways of rearranging the world’.
Jeff Bezos once described this same equation clearly when referring to innovation at Amazon, “If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce.”
Evolution is Undirected Failure
In biological evolution, the vast majority of genetic variations that are passed from one generation to the next don't work, meaning they don’t give their host any reproductive advantage. But, a very small number do, and those genetic variations eventually propagate across the entire species as their holders become more successful at reproducing.
Similarly, at the scale of an entire economy, most ideas, products, and businesses don’t happen to be useful, and thus don't produce innovation— they fail. A very small number of products, ideas and businesses do happen to be useful, are ‘selected’ by the market, and so push technology forward.
At the scale of an early stage investment fund, the same concept holds true, the returns of an early stage venture fund follow a power law distribution. Most investments barely register on the fund’s returns, and a disproportionately small number of investments account for almost all of the fund’s return. In both cases, the predominant outcome is failure, in the same way that most genetic changes fail to produce advantages for their holders.
Biological evolution also famously retired the belief that there was a ‘grand designer’ (i.e. God) of the living world. According to evolution, all of the living organisms alive today emerged by way of chance, and were shaped by the environment they inhabited, and competition for resources with themselves, and other organisms.
At the macroscopic scale, technological evolution is similar, it follows no grand vision, ideas and products emerge and are shaped by competitors and the market. There is no central planner, no grand designer, only the forces of the market which guide resources toward productive technologies.
And so, at the macroscopic scale, the story that best describes how innovation works is Darwinian evolution. Success for an economy or a fund depends on encouraging a large number of bets (lots of ‘at bats’), with the expectation that many of those bets will fail, but a consistent minority of them will succeed wildly. Innovation, when viewed at this scale is the result of random trial and error, just like organisms are the result of random genetic variations. This is why it is so difficult for investment funds to outperform indices (although obviously not impossible), and why government is bad at selecting and investing in specific sectors to spur innovation.
But does that same evolutionary story make sense when applied to the individual? Are individuals that create technology following a Darwinian process?
Microscopic Scale Innovation is ____?
The most compelling exploration of this question comes from a set of notes recorded by Blake Masters from Peter Thiel’s Stanford class on startups, later compiled into the book Zero to One. The book is a widely popular guide for technologists and aspiring entrepreneurs, and so it is the perfect place to find an answer on what it takes create new technology at the microscopic, individual scale.
If innovation at the individual level is the same as it is at the macro level, then evolution should apply here too. Thiel and Masters describe an evolutionary way of thinking about technological progress as indefinite optimism. They write, “The Darwinist metaphor plays a central role in thinking about indefinite optimism." The following is an excerpt describing how an individual who believes that technology evolves by chance might think about creating technology.
“If you believe that the future is fundamentally indeterminate, you would stress diversification. This is true whether you’re optimistic or pessimistic. And indeed, chasing optionality seems to be what most everybody does. People go to junior high and then high school. They do all sorts of activities and join lots of clubs along the way. They basically spend 10 years building a diverse resume. They are preparing for a completely unknowable future. Whatever winds up happening, the diversely prepared can find something in their resume to build on.”
The indefinite optimist tries to acquire a skillset that will work in a large set of possible future states of the world, because they believe the future is largely unpredictable. Thinking the future is unknowable is very Darwinian, as evolution doesn’t ‘know’ anything in advance, the genetic variations of evolution are random, and the success of any single variation is entirely up to chance.
But, Thiel and Master’s argue, this indeterminate, probability-driven strategy completely eliminates the possibility that skill has a role to play in successfully creating new technology. If an individual’s probability of creating successful technology is a function of chance, then the most important activity is to try as many things as possible, to fail at as many things as possible, and wait for the probabilities to work in your favor. (Recall that this is essentially the exact explanation for innovation at the macroscopic scale.)
A better approach for creating technology, according to Thiel and Masters, is to be a definite optimist. Definite optimists believe that the future is to some extent knowable, that certain skills are more valuable than others, and they have a vision for a technology that improves the state of the world. They favor a model of technological progress by design, not by chance.
Thus, the evolutionary story for creating technology does not seem applicable at the scale of the individual.
So, what is a more appropriate story to describe innovation at the scale of an individual? What story best describes a definite optimist?
Microscopic Level Innovation is Directed Evolution
Thomas Edison describes the tremendous struggles he overcame when creating the lightbulb, “I have not failed. I've just found ten thousand ways that won't work.”
His story is evolutionary in a sense, as his ten thousands failures are very similar to the vast number of genetic 'failures' in Darwinian evolution, but there is a key difference. Darwinian, biological evolution is not goal seeking, because evolution is not intelligent. In the jargon of science, biological evolution is not teleological, meaning evolution has no purpose, no intent, it just happens.
Edison was failing repeatedly, but he was failing toward a specific goal. His efforts might be described as goal-directed, or, teleological, trial and error. The distinction between goal-directed evolution and non goal-directed evolution is the difference between definite and indefinite optimism. Definite optimists believe in the existence of a specific outcome, of a specific goal. Indefinite optimists don't. It is worth noting that definite optimists aren't always correct, but that's less important than their ability to believe in a future state of the world, and to work toward proving they are right.
There exists a great example of this goal directed evolution within the world of science. In science, which much resembles innovation at the macroscopic level, millions of experiments are conducted in hopes of observing accurate explanations about the world. Progress at the broadest possible scale is again more or less a function of the number of experiments conducted, and failure is the most common outcome (this should sound familiar). But crucially, scientists actually tend to be very narrowly focused within specific domains. It’s the rare scientist indeed who, in hopes of improving their contributions to science, decides to blur their focus, and hedge their bets across many different domains. Scientists tend to remain fanatically focused on single areas.
In his book The Knowledge Machine, Michael Strevens writes that science progresses due to the single-mindedness of stubborn scientists that are fully dedicated to a single domain. And this is necessary for science. It’s necessary because ‘doing’ science is actually very difficult. Experiments are laborious, the rigor required is exhausting. Observations are extremely hard to discern, and valid conclusions often hinge on data with many digits past the decimal point. The devil indeed is in the details.
He cites the example of Andrew Schally, who shared the Nobel prize in 1977 for his work describing the structure of an important brain hormone TRH. He was fanatically motivated toward the goal of discovering the structure of this hormone, believing that it’s structure would provide the world with useful knowledge. Strevens writes, “Schally estimated that in the course of his efforts to find the structure...he had to process the hypothalami of 160,000 pigs to obtain less than a thousandth of a gram of the sought after substance.” This is clearly an example of a definite optimist, indefinite thinkers do not persist with a task so discouraging.
In science at the individual scale, failure is the norm, discouragement is the expectation. Scientists must fail, and fail again, but must continue to fight to acquire knowledge in the area within which they think. If every scientist thought indeterminately, and when the going became difficult, decided to pivot to an entirely unrelated field, (thinking that all theories are equivalently likely) it is unlikely science would progress much at all, because there would be no directed progress.
This description of scientific progress maps well to the progress of technology.
Individuals seeking to create technology are best served by developing a specific view on how the world works (a hypothesis you might say), and then pursuing technologies that correspond with that view. Persistence toward the goal is the key. Individuals, unlike economies, do have agency, they can direct themselves toward a goal, and march steadily on the difficult path of trial and error.
And so, the story that best describes innovation at the scale of the individual is not Darwinian evolution, but directed evolution. This is innovation by persistence, and by design, not by chance.
There is hardly a more appropriate person to conclude this essay than Steve Jobs, who summarizes innovation at the individual level well (emphasis mine).
“I'm convinced that about half of what separates the successful entrepreneurs from the non-successful ones is pure perseverance.... Unless you have a lot of passion about this, you're not going to survive. You're going to give it up. So you've got to have an idea, or a problem or a wrong that you want to right that you're passionate about; otherwise, you're not going to have the perseverance to stick it through."