Where Good Ideas Come from: The Natural History of Innovation
by Steven Johnson
Goodreads link

We have this popular notion of the inventor as a lone genius laboring in the wee hours of the night in his garage until finally Eureka! - a flash of insight and a new technology is born, to be patented and sold for profit.

But is that really how innovation works? Where do good ideas come from, and what kinds of environments best support innovation?

That’s core question that Steven Johnson tries to answer in this book.

The Adjacent Possible

The Adjacent Possible is a framework for thinking about the space of innovations that are possible with at a particular point of time.

Most new good ideas come from taking other good ideas and recombining them in novel ways to create something new. To create the printing press, Gutenberg combined technology from winemaking presses, movable type from blacksmithing, paper, ink etc.

Think of the status quo in a specific area like a room with four doors. The adjacent possible are the rooms immediately next to this room. Each time one door is opened, the adjacent possible expands - opening one door leads to three more doors that can be opened next.

There is such a thing as being too early - one great example being Charles Babbage’s Analytical Engine. It leaped ahead of the adjacent possible, and that’s not always a good thing - it wasn’t actually feasible to build his engine in his time. His core idea wouldn’t actually be built until the vacuum tube was created many decades later.

The key to making breakthroughs is to increase the pile of things and ideas and tools that are available to be recombined in novel ways.

Liquid Networks

Good ideas tend to arise in places that are the right balance of order and chaos.

Chaos pushes ideas against each other and lets new interesting properties and ideas arise. On the other hand, stability is needed to ensure that those ideas have a chance to develop and survive.

This reminds me of the hub-and-spoke concept Cam Newport talks about in his book Deep Work. The ideal office layout, he says, is like a hub and spoke - a hub where random encounters between disparate individuals happen, and spokes where everyone can get their deep work done in peace.

The Slow Hunch

For the vast majority of great ideas, there is no single Eureka! moment where the idea materializes in one’s mind. Interestingly, the sudden-flash inspiration is a story inventors often create in retrospect when asked to explain where the idea came from. This was the case with Charles Darwin’s ideas on natural selection, for example.

Rather, most great ideas come as slow hunches that are developed over an extended period of time.

To do so, ideas need a nurturing culture that allows them to develop, to meet other hunches, to morph and change and ultimately blossom (or wither and die). Tim Berners-Lee tells his story of how he came up with the idea for the world-wide web over a matter of decades - tracing it all the way back to an old copy of a compendium of advice on household tasks marvelously-titled “Enquire Within Upon Everything”.

One good way to nurture hunches is to write them down. This became very popular during the enlightenment in the form the “commonplace book” - a journal where one would write down ideas, hunches, passages of books and so on. The book would not be strongly structured in order to encourage at-first unrelated hunches to live side by side, such that they may interact and evolve - but it should have an index, so that passages about a particular topic could readily be found.

And, of course, probably the best way to grow hunches is to bounce them off others and their hunches.

Serendipity

Another key component that fosters the creation of new ideas is serendipity - the fortuitous and unexpected discovery of something one wasn’t looking for that turns out to be beneficial.

Johnson notes that “The work of dreams turns out to be a particularly chaotic, yet productive way to explore the adjacent possible.” When dreaming, neurons fire in unusual patterns, creating novel thoughts and ideas and connections. Interestingly, the waking mind also has regular periods of such chaos.

As Johnson says, “Serendipity is built out of happy accidents… but what makes them happy is the fact that the discovery you’ve made is meaningful to you.” In other words, you need fertile ground - a lot of hunches lying around, waiting to interact and blossom.

Hunches often come together when the mind wanders - think leisurely walks or long showers or baths.

Arrangements to encourage and benefit from serendipity can be made on an individual, organizational, and even societal level. Google’s 20% time is one example of an organizational-level mechanism to encourage serendipity.

Error

Scientists often assume that results that don’t meet their expectations are somehow inaccurate, the result of experimental error. Outsiders are more likely to find meaning in them that might yield valuable insights. That’s why talking about your work, even (or especially) when it’s not going well is so valuable.

It might also partially explain why people tend to make their most important contributions to a field when they are new to it.

There is research that noise (as opposed to signal) spurs creativity. The history of good ideas is filled with errors upon errors that eventually yield a key valuable insight.

Evolution has tuned in to this - random errors in copying DNA produce mostly irrelevant and harmful changes, but every now and then yield a valuable improvement - and those improvements over time far outweigh the impact of the errors.

Exaptation

Exaptation is a term from biology, referring to the use of a trait for a purpose totally different from the purpose for which it evolved. For example, feathers evolved for temperature regulation, but eventually came to be used for flight in some species.

Where error, mutation, and serendipity unlock new doors in the adjacent possible; exaptation helps us explore the new possibilities beyond those doors.

This is common in the realm of ideas and innovation as well. One key ingredient to being able to take advantage of exaptation is to have connections to a broad range of fields and individuals from a diverse set of intellectual disciplines. One way to ensure this is to have a broad range of hobbies in different fields - what Johnson calls “slow multitasking”, where one task occupies the front of the mind for hours or days at a time - but the others are well within reach. Each hobby has its own adjacent possible - but sometimes they overlap, and a great idea is born.

As Johnson says, “chance favors the connected mind.”

Platforms

Platforms are another key. Build an open platform, and you will be shocked by the creativity and wide ideas that others will build on top of it.

In the nomenclature of the adjacent possible, platforms let others create new doors to more adjacent possibles.

The internet and the many ecosystems on it are a great example, of course, but there are much more tangible ones, too - from Miles Davis breaking the rules (and yet building on so many platforms!) in A Kind of Blue, to atolls bristling with life created by coral reefs.

One interesting thing to note here is that Johnson extols the openness of the web and the platforms built on top of it - but there has actually been significant backsliding since the book was published in 2010. The open Twitter APIs he praises, for example, are now far more restricted. Today it feels like walled gardens and paywalls are becoming the norm, alas.

The Fourth Quadrant

In the book’s final chapter, Johnson a look at influential ideas and breaks them down into four quadrants based on whether they were developed in a closed or collaborative environment, and whether their creators were driven by the profit motive or not.

He finds that good ideas from all four quadrants, but that in the past two centuries there has been a shift towards what he calls the fourth quadrant - collaboratively-developed, non-market-driven ideas.

The internet and open source software are great examples of projects in the fourth quadrant - but so is GPS, punch cards, radar and so on.

It seems, in fact, that ideas are fundamentally different from property - that one cannot truly own them. Just like using your candle to light another does not diminish yours, so too sharing an idea takes nothing from you.

This has some interesting implications. For example, in the US copyright protections were originally set to last 14 years when the law was passed in 1790 (with a possible extension of another 14 years). Copyright-holders have lobbied hard, though, and so now the term in the US “life of the author + 70 years”.

There’s no good reason why copyright should last long. Economist Rufus Pollock estimated the optimal copyright term to be about 15 years, based on overall benefit to society. Other leading economists also argue that current copyright terms are too long.

Closing Thoughts

Here are some personal habits that I am trying out to spur my creativity (some of them adapted from this book, but more often slowly developed over time):

• Writing down ideas and hunches as they come. Right now, I use Google Keep to jot down ideas quickly on the go, and have a journal (a Google Doc) where I elaborate further. There is a lot of room for improvement here. I’d like to try out Zettelkasten, but haven’t quite found a solution I feel great about. If you have one, let me know.
• Taking notes on (nonfiction) books read - just like these! I don’t end up polishing all of them to be suitable for publication here, but the act of summarizing something in my own words is helpful for improved recall.
• Having a wide variety of hobbies and side projects.
• I’ve added a bookmark to Wikipedia’s Random Article redirect, and take a look at it on most days. I think I originally got this idea from Cal Newport’s Deep Work.
• Leaving space for my mind to wander. That means walks, showers, even drives - without the radio and without music. Meditating also helps greatly.

5/5 - Great anecdotes of breakthroughs and higher-level analysis with useful and actionable insights drawn. A fun read.