Serendipity and strategy in rapid innovation T. M. A. Fink∗†, M. Reevesz, R. Palmaz and R. S. Farry yLondon Institute for Mathematical Sciences, Mayfair, London W1K 2XF, UK ∗Centre National de la Recherche Scientifique, Paris, France zBCG Henderson Institute, The Boston Consulting Group, New York, USA Abstract. Innovation is to organizations what evolution is to organ- process. isms: it is how organisations adapt to changes in the environment Serendipity. On the other hand, a serendipitous approach is and improve. Yet despite steady advances in our understanding of seen in firms like Apple, which is notoriously opposed to mak- evolution, what drives innovation remains elusive. On the one hand, ing innovation choices based on incremental consumer demands, organizations invest heavily in systematic strategies to accelerate in- novation. On the other, historical analysis and individual experience and Tesla, which has invested for years in their vision of long- suggest that serendipity plays a significant role in the discovery pro- distance electric cars [13]. In science, many of the most impor- cess. To unify these two perspectives, we analyzed the mathematics of tant discoveries have serendipitous origins, in contrast to their innovation as a search process for viable designs across a universe of published step-by-step write-ups, such as penicillin, heparin, component building blocks. We then tested our insights using histor- X-rays and nitrous oxide [9]. The role of vision and intuition ical data from language, gastronomy and technology. By measuring the number of makeable designs as we acquire more components, tend to be under-reported: a study of 33 major discoveries in we observed that the relative usefulness of different components is biochemistry \in which serendipity played a crucial role" con- not fixed, but cross each other over time. When these crossovers are cluded that \when it comes to `chance' factors, few scientists unanticipated, they appear to be the result of serendipity. But when `tell it like it was'" [14, 15]. we can predict crossovers ahead of time, they offer an opportunity Key results. To unify these two perspectives and understand to strategically increase the growth of our product space. Thus we what drives innovation, in this Article we do four things. First, find that the serendipitous and strategic visions of innovation can be viewed as different manifestations of the same thing: the changing we study data from three sectors: language, gastronomy and importance of components over time. technology (Fig. 1). We measure how the number of makeable products (words, recipes and software products) grows as we Innovation is how governments, institutions and firms adapt to acquire new components (letters, ingredients and development changes in the environment and improve [1]. Organizations that tools). We observe that the relative usefulness of components innovate are more likely to prosper and stand the test of time; is not fixed, but cross each other in time. Second, to explain those that fail to do so fall behind their competitors and suc- these crossovers, we prove a conservation law for the innova- cumb to market and environmental change [2, 3]. But despite tion process over time (eq. (1)). The conserved quantity is a the importance of innovation, what drives innovation remains combination of the usefulness of components and the complex- elusive [1, 4]. There is a perennial tension between a strategic ity of products. We use it to forecast crossovers in the future approach, which views innovation as a rational process which based on information we already have about the products we can be measured and prescribed [5{8]; and a belief in serendip- can make. Third, we identify a spectrum of innovation strate- ity and the intuition of extraordinary individuals [9{11]. gies dependent on how far into the future we forecast: from Strategy. The strategic approach is seen in firms like P&G and short term gain to long term growth. A short-sighted strategy Unilever, which use process manuals and consumer research to maximizes what a new component can do for us now, whereas maintain a reliable innovation factory [12], and Zara, which sys- a far-sighted strategy maximizes what it could do for us later. tematically scales new products up and down based on real-time We apply both strategies to our three sectors and find that they sales data. In scientific discovery, \traditional scientific training differ from each other to the extent that each sector contains and thinking favor logic and predictability over chance" [9]. If crossovers (Fig. 4). Fourth, we resolve the tension between the discoveries are actually made in the way that scientific publi- strategic and serendipitous interpretations of innovation. Both cations suggest, the path to invention is a step-wise, rational 104 R cayenne ) Rails ) ) cocoa 104 jQuery UI Language F Gastronomy 100 Technology 1000 lime 50 X Sauce Labs 1000 usefulness usefulness ( usefulness ( ( 100 10 100 5 10 10 1 0.5 Makeable words Makeable recipes 1 1 Makeable software 0 2 4 6 8 10 12 14 16 18 20 22 24 26 0 127 254 381 0 331 662 993 Acquired letters Acquired ingredients Acquired development tools 1st R 1st cayenne 1st Rails 2nd F 2nd cocoa 2nd jQuery UI Rank 3rd X Rank 3rd lime Rank 3rd Sauce Labs FIG. 1: Products, components and usefulness. (Top) We studied products and components from three sectors. In language, the products are 79,258 English words and the components are the 26 letters. In gastronomy, the products are 56,498 recipes from the databases allrecipes.com, epicurious.com, and menupan.com [16] and the components are 381 ingredients. In technology, the products are 1158 software products catalogued by stackshare.io and the components are 993 development tools used to make them. (Bottom) The usefulness of a component is the number of products we can make that contain it. We find that the relative usefulness of a component depends on how many other components have already been acquired. For each sector, we show the usefulness of three typical components: averaged at each stage over all possible choices of the other acquired components and|for gastronomy|for a particular random order of component acquisition (points). 2 can be viewed as different manifestations of the changing im- that you believe will help you reach this goal. Let's now sup- portance of components over time. When component crossovers pose each player approaches this differently. Your approach is are unexpected, they appear to be the result of serendipity. But to follow your gut, arbitrarily selecting bricks that look intrigu- when we can forecast crossovers in advance, they provide an ing. Alice uses what we call a short-sighted strategy, carefully opportunity to strategically increase the growth of our product picking Lego men and their firefighting hats to immediately space. make simple toys. Meanwhile, Bob chooses pieces such as axels, Lego game. Let's illustrate these ideas using Lego bricks. wheels, and small base plates that he noticed are common in Think back to your childhood days. You're in a room with two more complex models, even though he is not able to use them friends Bob and Alice, playing with a big box of Lego bricks| straightaway to produce new toys. We call this a far-sighted say, a fire station set. All three of you have the same goal: to strategy. build as many new toys as possible. As you continue to play, Who wins. At the end of the day, who will have innovated each of you searches through the box and chooses those bricks the most? That is, who will have built the most new toys? We find that, in the beginning, Alice will lead the way, surging ahead with her impatient strategy. But as the game progresses, E A fate will appear to shift. Bob's early moves will begin to look I R serendipitous when he is able to assemble a complex fire truck N from his choice of initially useless axels and wheels. It will seem T 6 O that he was lucky, but we will soon see that he effectively cre- S L ated his own serendipity. What about you? Picking components C on a hunch, you will have built the fewest toys. Your friends had U D an information-enabled strategy, while you relied on chance. M 13 Language P Spectrum of strategies. What can we learn from this? If in- H G novation is a search process, then your component choices to- Y day matter greatly in terms of the options they will open up B F to you tomorrow. Do you pick components that quickly form V 20 K simple products and give you a return now, or do you choose W those components that give you a higher future option value? Z X By understanding innovation as a search for designs across a J Q universe of components, we made a surprising discovery. Infor- 26 mation about the unfolding process of innovation can be used egg wheat to form an advantageous innovation strategy. But there is no butter onion garlic one superior strategy. As we shall see, the optimal strategy de- milk vegetable_oil pends on time|how far along the innovation process we have cream tomato olive_oil advanced|and the sector|some sectors contain more oppor- 95 black_pepper pepper tunities for strategic advantage than others. vanilla cayenne vinegar cane_molasses bell_pepper cinnamon 1 Results parsley chicken Components and products. Just like the Lego toys are made up Gastronomy 190 lemon_juice beef cocoa of distinct kinds of bricks, we take products to be made up corn bread of distinct components. A component can be an object, like a scallion mustard ginger touch screen, but it can also be a skill, like using Python, or a basil celery 286 carrot routine, like customer registration.