Pattern Overload (And Why We Love It)

hprscript: Multi-Pattern Search Without Running grep N Times Why I built it Like many other programmers, I use AI agents for code development. A lot of the work is supervising what the agent does - reading its plans, checking its diffs, and sanity-checking the code it touches. That means a lot of grep commands, run one after another, often against the same tree. At some point I started thinking this was wasteful. Each grep invocation re-reads the files, re-compiles its pattern, and pays its own startup cost. And when an AI agent is the one driving, every separate search also re-sends the surrounding context to the model - the same conversation, the same instructions, the same tool schemas - just to ask one more question of the same codebase. Ten searches mean ten round trips, ten context replays, ten walks over the data. I knew there was a library that could match many patterns at once: Intel's Hyperscan . It compiles N regexes into a single DFA and matches them all ...

Innovation vs. Creativity (or Tactics vs. Strategy)

Nokia had the future in its drawer

My first mobile phone was a Nokia. Around 2000 almost everyone's was: Nokia sold roughly a third of all handsets in the world, and at the height of the dot-com boom its market value briefly touched about €300 billion, making it the most valuable company in Europe. In 2013 Microsoft agreed to buy the entire phone business for €5.44 billion (Silicon Canals). How does the undisputed leader of an industry lose almost everything in a little over a decade?

The usual explanation is that Nokia "missed the smartphone revolution". That is not quite true. Nokia sold its first smartphone, the Nokia 9000 Communicator, in 1996, and in 2010 it still held about a third of the smartphone market; by 2013 its share was under 3%. Nokia did not miss the smartphone. It missed a much bigger shift: a change in how a company and its community interact.

Nokia understood too late that a smartphone is not a product but a platform, a place where a whole community of outside developers brings an enormously wide range of functionality that no single company could ever design. Apple's App Store opened in July 2008 with 500 apps; Nokia's Ovi Store opened only in May 2009. By 2011 Ovi offered about 30,000 apps, against roughly 400,000 for Android and 550,000 for the iPhone (ReadWrite). In February 2011 Nokia's new CEO, Stephen Elop, admitted it in his famous "burning platform" memo: the battle of devices, he wrote, had become "a war of ecosystems" (Engadget).

Why did the best phone maker in the world miss this? Because of its greatest strength. Nokia wanted to control every single aspect of its product, and that is exactly how it made the best phones. But control has a price. Absolute control does not allow any change, not even good changes, and a platform lives on changes that nobody at the centre planned.

And Nokia did not fail for lack of ideas. It had the ideas, the money and the engineers. More than seven years before the iPhone, Nokia designers showed a phone with a colour touchscreen above a single button, and in the late 1990s they secretly built a touchscreen tablet with a wireless connection (WSJ, via Gizmodo). In 2004 an internet-ready touchscreen prototype was killed because management feared it would flop, and the same year a design for an online app store was rejected (VentureBeat), four years before Apple opened its own.

So Nokia had the future in its drawer, kept the drawer closed, and lost the market it was best placed to win. Customers who moved to the new kind of product did not come back, and the classic feature phone shrank to a niche. Dmitry Chernavsky's model of the "struggle of informations", which I described in an earlier post, calls this a case of antagonistic informations: two products compete for the same customers, and one pushes the other almost out of existence.

The real question, then, is not "why didn't Nokia have the idea?" It is "why couldn't a company that had the idea turn it into a product?"

Nokia is far from the only example. Kodak invented the digital camera: in December 1975 its engineer Steve Sasson built an 8-pound prototype that recorded black-and-white pictures on a cassette tape. Executives were unimpressed, since people had been happy with prints for a hundred years (PetaPixel); Sasson remembers the verdict as "that's cute, but don't tell anyone about it." Kodak was not blind. It invested billions in digital imaging and in 2001 even bought Ofoto, an online photo-sharing site, but used it mainly to sell prints. In 2012, the year Kodak filed for bankruptcy, it sold the site for $23.8 million (Wikipedia), the same spring Facebook paid $1 billion for Instagram.

Why do big companies, with all their resources, keep missing the big changes?

Two different processes

Companies spend heavily on "innovation and creativity" as if they were one activity. They are two different processes, so different that one organisation rarely does both well.

First, what I mean by each word:

  • Innovation is a continuous process of improving a product that already exists.
  • Creativity is the act of creating a product that does not exist yet.

A word of warning about terms. Management literature often uses these words differently: "creativity" for generating any new idea, and "innovation" for turning ideas into products. What I call innovation is close to what others call incremental innovation, or what James March called exploitation. What I call creativity is close to radical innovation, or March's exploration. I keep my own words because they point at the thing I care about: whether the new idea has an existing context or not.

Both innovation and creativity create new information, but in very different contexts. For innovation the context is well defined: it is the existing product, and new information improves it. For creativity the context is so unstructured, mostly a scatter of unrelated facts, that it hardly deserves the name.

That is the key difference. Innovation can be evaluated in the context of the existing product: the value of a new idea can be examined and estimated before anyone invests in it. Creativity can hardly be evaluated at all. That is why creativity brings enormously higher risk, and why most managers are uncomfortable with it.

Uncertainty and information

The difference between the two contexts is the amount of uncertainty, and information theory gives a precise way to think about it. Claude Shannon showed that information is what reduces uncertainty: before a message we face many possible outcomes, after it fewer. A situation with two possible outcomes holds little uncertainty; one with a million holds a great deal. In information theory this uncertainty can even be measured, by what is called informational entropy, but for this post the everyday word is enough. I lean on information theory deliberately, because in my view it allows a more scientific approach to how technical systems evolve than vague words like "radical" or "disruptive".

Seen this way, both innovation and creativity create information by reducing uncertainty, but they start from very different places. The context of innovation has low uncertainty: few open questions. The context of creativity has very high uncertainty: almost everything is still open. And since information is measured by how much uncertainty it removes, a single creative decision can carry more information than many steps of improvement.

Here is a real example. Around 1921, a 14-year-old Idaho farm boy named Philo Farnsworth was plowing a potato field. Looking at the parallel furrows, he realised that an electron beam could draw a picture the same way, one line at a time (Linda Hall Library). That was the moment of creativity: the idea of electronic television. Then came years of work to make it real, with the first patent filed in 1927. That was innovation: new ideas, but inside the context of an idea that already existed.

Two lessons follow. Creativity is a moment; innovation is a process. And the moment is unpredictable, but it is not random. Creativity does have a context, only an enormously unstructured one: mostly unrelated facts that hardly deserve to be called a context. For Farnsworth they were magazine articles about television and the rows of a potato field, and nothing linked them until the moment of insight did. As Pasteur said, chance favours only the prepared mind. Nobody can schedule the moment, but people and organisations can collect the pieces and be prepared for it. Keep this in mind for the section on strategy.

Every product starts at a creativity point and is then improved by innovation, so the uncertainty of its context falls over time: very high at birth, very low in maturity. Students of industrial innovation describe the same pattern without the word: Abernathy and Utterback showed that a new industry begins with many competing designs, converges on a dominant design, and then settles into incremental improvement. The TRIZ law of increasing ideality explains why the fall is inevitable. Higher ideality, more useful function for less cost and harm, requires higher complexity; higher complexity brings more interdependencies; more interdependencies reduce the changes that are still possible without breaking something; and fewer possible changes is exactly what lower uncertainty means. A product near the top of its ideality curve is almost impossible to change in any fundamental way. That is the state Nokia's phones had reached by 2007.

[embed: node/9a1330fc-6d73]

The picture also explains Nokia. By 2007 the feature phone sat at the bottom of its curve: low uncertainty, polished by thousands of small improvements. The touchscreen smartphone was the top of a new curve. Nokia was excellent at moving down a curve and poor at jumping to the next one. Kodak adds a twist: digital imaging was built on semiconductors, far from the precision chemistry Kodak knew (Willy Shih, MIT Sloan Management Review), so the new curve made not only the old product irrelevant but the old knowledge too.

One last note on theory. Shannon's theory measures how information changes uncertainty, but it does not explain where new information comes from. For that I rely on the Dynamic Theory of Information of the Russian physicist Dmitry Chernavsky (overview), which separates receiving existing information from generating new information. Both creativity and innovation are acts of generation; they differ in how much uncertainty they start from.

Why creativity is worth more, and why managers avoid it

Creativity is where the big value is, and that is exactly why a well-run company tends to suppress it.

The most profitable markets are new ones, and the largest share usually goes to whoever defines the new product. In information terms this is natural: the creative point is where the most uncertainty is removed, so it is where the most information is created.

But "first" needs care. Being first to sell is not the same as winning. Golder and Tellis studied about 500 brands in 50 product categories and found that market pioneers failed 47% of the time and averaged only about 10% market share (Steve Blank's summary; original paper).

The iPhone is the best example I know. It was not the first phone with a touchscreen, not by a long way. IBM's Simon had one in 1994, Ericsson's R380 in 2000, and LG's Prada, with the first capacitive touchscreen, reached the market a few months before the iPhone in 2007 (Android Authority). Nokia, BlackBerry, Palm and Windows Mobile devices had sold smartphones for years. Each of these products was growing along its own S-curve, as I described in my earlier post. So why did the iPhone win?

Chernavsky's model of the struggle of informations answers this. Those devices were antagonistic informations: nobody carries two handheld devices, so every customer who chose one rejected the others. In such a struggle the model allows only one information to survive, and the decisive advantage goes to whoever reduces uncertainty fastest. The iPhone's advantage was not the touchscreen but the App Store. From July 2008 a whole community of outside developers reduced the uncertainty of the new product far faster than any single company could, and the other platforms could not keep up. Apple did not win by being first. It won by being first to bring the new product's uncertainty down to a level ordinary customers would accept, that is, by defining the dominant design. The value lies in the creative act plus the speed at which its uncertainty is reduced.

The same model explains why the struggle ended with two survivors rather than one. It allows two antagonistic informations to coexist when the diffusion between them is very low, when they do not mix. That is the case of iPhone and Android. Their incompatibility is the barrier: an app bought on one, a backup made on one, a family's shared services on one, none of it transfers. Each ecosystem is a closed territory, and so two have survived side by side for more than fifteen years. Android holds about 70% of the world's phones and iOS about 30% (StatCounter, June 2026), and neither has shown any sign of swallowing the other.

One more property of the model deserves attention: among antagonistic informations, even a small advantage ends in a clean sweep. Facebook and Myspace show it. Myspace was the largest social network in the world from 2005 to 2008, with 115 million monthly visitors at its peak in April 2008, and News Corporation had paid $580 million for it in 2005 (Wikipedia). Facebook was not a lot better. It was a social network much like Myspace, aimed at first only at college students. But a social network is an antagonistic information in its purest form: your friends are on one site or the other, so a slight edge in users becomes a slightly bigger edge the next month, and the model's winner-takes-all dynamics do the rest. Facebook passed Myspace in worldwide visitors in April 2008; in 2011 News Corporation sold Myspace for about $35 million, one-sixteenth of what it had paid. And the small edge was a familiar one: Facebook opened its platform to outside developers, while Myspace built everything in-house.

Now the problem. The value of creative information cannot be estimated in advance. A business case needs numbers, and a creative idea has none yet. Every approval process in a big company asks "how much will it bring, and how sure are we?" For a highly uncertain idea the honest answer is "we don't know", so the idea dies. It is killed not by a bad manager but by a normal, sensible process. Nokia's 2004 touchscreen prototype is a textbook case: according to a former manager, it was expensive, nobody yet understood what a touchscreen could do, and so management shut it down. Kodak faced the same arithmetic for thirty years: every digital camera it sold meant a customer who would stop buying film, its most profitable product.

Hence the paradox. Everyone agrees that creativity is the most valuable thing a company can have, and the ordinary machinery of a well-run company is built to remove it.

Tactics vs. strategy

Innovation is tactics and creativity is strategy: the first can be done directly, the second can only be made likely.

Tactics is a set of actions taken to solve a known problem directly. Strategy, as I use the word, is creating an environment in which the problem gets solved indirectly, often in a way nobody planned. I am not alone in this view: the economist John Kay calls it obliquity, and Henry Mintzberg writes about emergent strategy. Complexity theory has a precise name for the mechanism: self-organisation.

Innovation can be done tactically. The context is known, the problem can be stated, and there are methods to solve it. The best known methodology is TRIZ, which turns an improvement problem into a contradiction and resolves it with tested principles. (I hope that one day TRIZ will become a full science.)

Creativity cannot be done tactically. Nobody can be told "have a breakthrough idea by Thursday." What can be managed is the environment. Complexity theory tells us that self-organisation appears in systems that meet a few conditions. Translated into the language of a company, they are:

  1. Openness. A steady inflow of new information from outside: customers, other industries, science. Farnsworth's magazines were his inflow.
  2. Fluctuations. Room for small deviations from the plan: prototypes, side projects, experiments that are allowed to fail.
  3. Amplification. A way for a promising fluctuation to grow and attract people and money, instead of being averaged away at the next budget review.
  4. Tension. A system at rest does not reorganise. There must be a real need or threat that the existing product cannot answer.

Nokia had tension: its top managers were well aware of the competitive threat. It had fluctuations too, as its prototypes show. What it lacked was openness and amplification. It kept tight control of every part of its phones, so the community outside could not add its own ideas, and the system inside damped every fluctuation that threatened the core business. The smartphone platform is strategy in exactly the sense of this section: Apple and Google did not design most of what their phones can do; they created an environment in which a huge community of developers did it for them. Nokia kept doing tactics, designing every feature itself, and no amount of TRIZ applied to the feature phone could have fixed that.

Corporation vs. startup

The bigger the company, the less uncertainty it can carry, so the modern economy moved high-uncertainty work out of big companies and into startups.

A big company is a complicated system with many interdependencies. Uncertainty in one part spreads to the others: a risky new product touches the brand, the supply chain, the sales channels and the quarterly results. Even a small uncertainty can have serious consequences, so a big company damps fluctuations, and it does so rationally. What looks like stiffness is often self-protection. Kodak's film business was tied to factories, chemical supply chains and thousands of photofinishing labs; a move to digital threatened all of them at once. Between 2003 and 2012 alone it closed 13 plants and some 130 labs.

Nokia shows how this works from the inside. Timo Vuori and Quy Huy interviewed 76 Nokia managers, engineers and outside experts and found an organisation gripped by shared fear (INSEAD Knowledge). Top managers feared competitors and shareholders; middle managers feared their superiors and peers, so they kept bad news to themselves. As a result, top management developed an overly optimistic picture of how quickly Nokia could match the iPhone. In my terms, the channels that should have amplified fluctuations filtered information out instead.

The economy found a way around this. A startup takes an idea with very high uncertainty and works it down: a product that works, customers who pay, a business model that holds. When the uncertainty has fallen to a level a corporation can accept, the corporation buys the startup. In information terms, the corporation is not buying an idea; it is buying reduced uncertainty, information that has already been created and tested. The acquisition price includes a large premium for that service.

The pattern is older than venture capital. Philo Farnsworth, with a small lab and a few investors, was in effect a startup. RCA, the giant of American radio, first offered \$100,000 for his company; when he refused, it fought him over patents for years. In 1939 RCA agreed to license his patents, paying a $1 million down payment, the first time it paid royalties instead of collecting them (Television Academy). After the war RCA dominated the television market, while Farnsworth's company ended up in the hands of ITT. The creator did not capture the market; the big company that scaled the idea did.

Can a big company be creative? A TRIZ answer

Yes, if it treats its problem as a TRIZ contradiction and separates the opposite requirements instead of compromising between them.

The first edition of this post ended with a promise: a big company can learn to be creative if it formulates the contradiction properly and resolves it with TRIZ. Here is the contradiction:

A big company must have high uncertainty, so that new products can appear, and it must have low uncertainty, so that its existing, interdependent business is not put at risk.

In TRIZ this is a physical contradiction: one parameter must take two opposite values. TRIZ does not resolve such contradictions by compromise, because "a little uncertainty everywhere" usually gives the worst of both worlds. It has several ways to resolve them: the forty inventive principles, the standard solutions and the laws of evolution, among others. For a physical contradiction the most natural one is separation, keeping both opposite values but in different places, times, conditions or levels of the system. Here are the four separations applied to our contradiction.

 Separation in space. Keep high uncertainty in one place and low uncertainty everywhere else. Apple did exactly this with the iPhone. The project, code-named Purple, was given to a small, secret team that worked on a locked-down floor of a building on Apple's campus and reported directly to Steve Jobs (Computerworld). Apple was already a big company in 2005, yet it created a protected island of high uncertainty inside itself. Lockheed's Skunk Works and the "ambidextrous organisation" described by O'Reilly and Tushman follow the same principle.

Separation in time. Allow high uncertainty in one phase and demand low uncertainty in the next. In its early phase a project is judged by what it has learned, not by revenue; only after its uncertainty has fallen does it face the normal business case. The classic mistake is to apply the second phase's rules to the first, which is what happened to Nokia's 2004 prototype.

Separation between the system and its parts. The company as a whole stays stable while its small parts fluctuate. Many small, cheap experiments, each of which may fail without hurting the whole, give the system its fluctuations without its risk. Evolution works the same way: a single mutation is risky, the population is not.

Separation upon condition. High uncertainty is allowed only under explicit conditions: a capped budget, a fixed time box, clear criteria for stopping or scaling up. The conditions make the risk bounded, and bounded risk is something a big company can accept.

The market's own solution, startups followed by acquisitions, is separation in space and in time moved outside the company's boundary. Corporate venture funds and acquisitions are a legitimate answer to the contradiction, not an admission of failure. They are, however, expensive: the company pays someone else for the uncertainty reduction it could not do itself.

Kodak shows the trap in this solution. In 2001 it bought Ofoto, a young photo-sharing company: separation in space, achieved by acquisition. Then it made Ofoto serve the old business, as a channel for selling prints. Separation must cover the rules by which the new unit is judged, not only its place on the organisation chart.

One condition is not supplied by TRIZ. Information must be able to travel upward. Separation protects new ideas from the core business, but if people are afraid to report bad news, as they were at Nokia, no structure will help. A creative island needs a direct and honest line to the top.

Back to Nokia

Nokia did not lack creativity; it lacked a strategy for creativity.

Its people had the creative moments: the touchscreen phone, the tablet, the app store. What Nokia did not have was an environment in which those highly uncertain ideas could survive long enough for their uncertainty to fall. The organisation was built for innovation, for moving efficiently down a known curve. It did that superbly, until the curve itself stopped mattering. Kodak, from its 1975 digital camera to its photo-sharing site, followed the same path.

Apple in 2005 was also a big company. The difference was not resources: Nokia was the larger company and the bigger spender on research. The difference was that Apple separated the high-uncertainty work from its core business and protected it, while Nokia judged every new idea by the rules of the old one, just as Kodak judged digital photography by how many prints it would sell.

The lesson for managers is simple to state and hard to practise. Do not ask your innovation process to produce creativity. Innovation is tactics: you can plan it, measure it and improve it with methods like TRIZ. Creativity is strategy: you cannot order it, but you can build the conditions in which it happens and, above all, you can stop killing it.

Further reading

First published November 2023; revised October 2026.


Comments

Popular posts from this blog

Big Software Contradiction

"Life", "Death" and "The Winner Takes It All"