I have identified three means by which new ideas are created.
I. Random Incremental Creativity
The creator starts this process with an existing “product” that the creator wishes to improve upon. In this case, the creator has no idea of any rules or principles that might suggest how to improve the product; the creator is therefore required to make random attempts at improvement. A simple way to understand the method is to imagine a person standing in terrain, desirous of moving upward.
In this example, we would imagine the person to be blind, so he must try out random steps to figure out the best step to take.
This is the method that nature uses in evolution of living creatures. This method is ideal for such problems, because no creative methodology can know in advance what kind of new environment a living creature might be facing. Could it be a colder climate? A new predator who can run faster? A competing animal who more efficiently gathers food? Without having any way to know in advance what problem to solve, nature must use the most general method: random incremental creativity.
For much of history, people used random incremental creativity. Without any way to understand the inner processes taking place in the real world, people just had to rely on trial and error. I’ve been able to figure out a simple, clear example of this process at work in the Chinese invention of gunpowder.
Now, if you read the official Chinese histories, they will tell you that gunpowder was invented by a Chinese scholar seeking a medication that would bestow immortality — or, at least, very long life. If you know anything about simple chemistry, you will immediately recognize just how absurd this notion is. You see, gunpowder consists of three basic components: charcoal, sulfur, and saltpeter. Have you ever tried to eat charcoal? It makes cardboard seem tasty. Charcoal simply soaks up whatever chemicals are in your mouth, leaving you with the ultimate in un-taste. Nobody would imagine that charcoal could somehow confer immortality.
It gets worse. The second component, sulfur, turns into sulfuric acid when it mixes with the water in the saliva in your mouth. Nobody who has ever experienced a strong acid in their mouth could imagine it to be conducive to immortality.
I have never tasted saltpeter, but it has been used in a variety of medications throughout history. However, in large quantities it is deadly. I doubt that a seeker of immortality would be tempted to try saltpeter.
No, gunpowder was not invented by a Chinese scholar seeking a medication for immortality. This story was concocted by the people who wrote the history books: Chinese scholars. The notion that something important could have been created by a slobbering commoner was repugnant to these scholars, so they cooked up a story more nurturative of their egos.
The real story behind the invention of gunpowder makes a lot more sense. In my hypothesis, gunpowder was invented by Chinese potters {gasp!}
Right up until fairly recently, Chinese pottery was the best in the world. With a large population and its system of canals, and prestige associated with owning fine pottery, the market for quality pottery was always strong, leading to a large and highly motivated pottery industry. Chinese potters quickly learned that the easiest way to make fine pottery was to get higher and higher temperatures in their pottery furnaces. Everybody already knew from experience in the metal industry that replacing wood with charcoal, and pumping in lots of air drove temperatures way up. Another easy lesson was that a really hot furnace required really thick walls. The easiest way to accomplish this was to dig into the side of a river bank and carve the furnace out of the thick silt there:
Chinese potters were able to achieve excellent results with these methods. But they were always driving for higher temperatures to get thinner products: porcelain, or what we call “China”. To accomplish this, they needed even higher temperatures. To get those higher temperatures, they needed to get the charcoal to burn faster so as to release its energy very quickly, thereby reaching high temperatures.
Now, these people didn’t know anything about chemistry, so they had to rely on trial and error: random incremental creativity. What if we pee on the charcoal? Nope, that didn’t help. Maybe mixing in dead frogs? Nope, that didn’t work either. After lots of failures, they discovered that sulfur did seem to help a little. But the real breakthrough came when they added some saltpeter to the mix.
Given the rarity of saltpeter and sulfur, you can imagine how long it must have taken them to hit upon that combination. But it definitely accelerated the combusion of the charcoal, increasing temperatures inside the furnace.
Next, they experimented with differing ratios of sulfur and saltpeter to charcoal. That’s when they learned the value of asking yourself, “What happens if I succeed beyond my wildest expectations?” They stumbled upon the optimal mix of charcoal, sulfur, and saltpeter — and it exploded.
Blowing up your pottery is not profitable. But the more entrepreneurial potters came up with productive uses of this new, too-fast-burning stuff. First came firecrackers. Then they realized that, by using just a tad less sulfur and saltpeter, they could get it to burn fast enough to emit a powerful stream of gas. That’s how they invented rockets. Then they staged some gunpowder on the top of a rocket, so that it flew into the air and exploded: fireworks!
Then somebody realized that you could aim the rocket at enemies and blow them up. Well, actually, that didn’t work very well, because the rockets weren’t that accurate, and the explosion really wasn’t that big. But it sure scared the hell out of attacking barbarians.
Finally, somebody invented the gun, only it wasn’t much of a gun. They made a strong cone of metal, put some gunpowder into it, packed it with arrows, aimed the cone at the enemy, and lit the gunpowder. The explosion sent fragments of arrows flying in the general direction of the enemy. It may not have injured many barbarians, but it sure was impressive.
That’s about as far as the Chinese took gunpowder. They didn’t fight that many wars, so they never needed to develop gunpowder much further. But when the war-obsessed Europeans found out about gunpowder, they continued the random incremental process, improving the purity of the ingredients, building stronger guns with tighter clearances, and improving the technology all the way to this:
But this was as far as they could go with random incremental creativity. There just weren’t any further improvements that could be wrung out of gunpowder technology with trial and error techniques.
II. Informed Incremental Creativity
This creative process begins in exactly the same fashion as the first: the creator already has an existing creation and wishes to improve upon it. But in this creative method, the creator has knowledge of the underlying principles at work, and is able to apply those principles to guide their work. Those principles do not provide an omniscient perception of the entire problem; they serve only to provide improvements upon the immediate problem. Think of it this way: informed incremental creativity is used when the creator can see the area around his feet, but nothing more. Rather like this:
The creator can see his immediate area, but has no idea what lies beyond.
This has been the primary form of technical creativity since the Scientific Revolution. Engineers apply their knowledge of science to improve existing technologies. This is how the West dramaticall improved explosives starting around 1870. Chemists by that time had figured out that the crucial limiting factor on combusion was the availability of oxygen, and that saltpeter (chemical formula KNO3) released its oxygen at high temperatures, providing oxygen for the combustion of the charcoal. This realization led them to search for other chemicals that could provide more oxygen more quickly. I won’t walk you through the complex chemistry, but within a few decades, chemists had invented high explosives, which were dramatically more powerful than low explosives (gunpowder). Gunpowder burns by the transfer of heat from the burning charcoal to adjacent charcoal, increasing its temperature to a level sufficient for that adjacent chacoal to start burning. This process is, in terms of chemistry, rather slow; it takes tens of milliseconds for the heat to be transmitted all the way through all the gunpowder. But in high explosives, the material is ignited by the shock wave initiated by the very first combustion — which happens much faster than with gunpowder, making the explosion more powerful. Chemists also figured out how to increase the energy content of their high explosives, making them even more powerful. Thus, while the cannons of the American Civil War had a range of, at most, 4,000 yards, the cannons of World War I had a range of over 10,000 yards, and the “Paris gun” used by the Germans had a range of 75 miles.
Another good example comes from integrated circuits. The critical technical constraint on ICs is the size of the smallest transistor that can be etched onto silicon. Here’s a graph of the industry’s progress:
Note that the vertical axis is expressed in log form. The actual decrease in channel size runs from 20,000 nanometers in 1968 to 3 nanometers in 2022. This improvement is certainly dramatic, but it was accomplished entirely by steady incremental improvements in the equipment used to manufacture ICs.
III. Goal-directed Creativity
This creative process is radically different from the first two methods; it is rarely used because it is most difficult. Creators using this method do not start with an existing product; instead, the creator desires to create something entirely new, something that has never been made before. Here’s a picture analogous to the previous pictures for random incremental creativity and informed incremental creativity:
This kind of creativity is usually opportunistic to some extent. Some new idea, technology, or theoretical development opens the creator’s eyes, permitting them to see farther than ever before. This creator does not look down at his feet; he looks far into the distance, and sees a mountaintop that can now be reached — at least theoretically. The creator doesn’t know how to get there. He doesn’t see the immediate step to take; perhaps traveling to the left or right will ultimately lead to a better path. All he really knows is that this final goal is suddenly achievable.
The first method of creativity, random incremental creativity, requires no intelligence; the creator makes random choices. The second method of creativity, informed incremental creativity, requires both knowledge and intelligence. The creator must have mastered the principles underlying the existing product, and devise an improvement based on those principles.
But the third method of creativity, goal-directed creativity, requires vastly more knowledge and intelligence. The creator must know everything that might bear on the problem. The creator must completely understand the new development that opened the door to the mountaintop. The creator must think in the most abstract terms in order to perceive the many possible paths to the goal.
Here’s an example to illustrate just how difficult this task is. By the 1920s, physicists had learned that stupendous amounts of energy lay bound in the nuclei of heavy atoms like uranium. It was obvious that, if a means to tap that energy could be devised, the benefits to humanity would be glorious. But nobody could see how that energy could be tapped. It wasn’t until 1934 that Leo Szilard figured out a way to unleash that energy. The solution was a chain reaction, but, while Szilard’s design could get the chain reaction started, his design had no way to control or limit the amount of energy released. It was a design for a bomb. A few years later, physicists figured out how to limit the nuclear reactions, thereby inventing the nuclear reactor. Szilard patented the design of the atom bomb, but then sold the license to the patent to the government for $1. Physicists are such lousy businessmen!
Leo Szilard
Step Two
Achieving the inspiration is only the first step in the process; the second step is to work out how to actually make it work. Creators often fail this step; all too often the grand idea simply cannot be worked out. For example, there were many attempts at building flying machines, all but one of which failed. Today we can watch hilarious videos of such attempts.
Success often takes gigantic efforts. The Manhattan Project, whose goal was to build the first atomic bomb, employed 130,000 people and cost, in today’s money, about $27 billion.
I struggled with my own effort at goal-directed creativity. After over ten years familiarizing myself with the concepts of computer game design, I set an ambitious goal: interactive storytelling. In 1991 I decided that computers had grown powerful enough to permit the creation of interactive storytelling technology, so I went to work. Sad to admit, I set off in the wrong direction, believing that stories required the use of language, and it was therefore necessary for me to build some sort of dramatically functional language system. I’m proud of Deikto, the technology I created that allowed the user to interact linguistically with the computer, with the semantic algorithms built directly into the vocabulary of the system. It was highly creative and very powerful — but it wasn’t good enough for solving the problem of interactive storytelling. It was only after I abandoned that strategy, throwing away about 15 years of work, that I was able to develop a completely different strategy. That new strategy solved the problem; I used it to build the first genuine work of truly interactive art, Le Morte D’Arthur.
I emphasize how rare and difficult this third form of creativity is. We have few examples of this. In the field of computer hardware, we had Babbage’s computer of the 1840s; the early computer designs of the 1940s; the transistor; the integrated circuit; magnetic tape storage; the light emitting diode; LED displays; optical fibers. I don’t whether to count the Internet as a hardware design or a software design. The entire computer revolution has seen a staggering amount of innovation, but only these seven elements were accomplished with goal-directed creativity. Everything else was achieved with informed incremental creativity.