04
Change One Thing Without Breaking Everything
illustrationLet us go back to the website for the nail salon and imagine it has been running for six months. It has three thousand clients and twelve manicurists, reminders go out by text message the day before an appointment, and once a month the accountant exports a spreadsheet of bookings and payments. One day the salon’s administrator asks for a small improvement: let a client enter several phone numbers, because some of them book appointments for their daughters and want to receive the reminders themselves.
The agent manages it in about three minutes. An “add another phone” button appears in the form, room for a second and third number appears in the database, the administrator checks that everything saves and opens, and happily closes the task. Everything works as requested.
About two weeks later it turns out that reminders have stopped reaching roughly a third of the clients. A week after that one of the manicurists calls a client on her second number, which turns out to be her husband’s, and learns more about the client’s family life than the client would have liked. At the end of the month the accountant’s export will not open: the program the accountant once set up to process the spreadsheet expected a single number in the “phone” column and has no idea what to do with a list.
None of these breakages was visible at the moment the administrator checked the result, and every one of them was caused by the same change, which looked flawless on the screen.
Developers call a change to code a diff, from the English word difference: a list of the differences between how the program looked before the edit and how it looks after, rather like track changes in a word processor, where deletions are shown in red and additions in green. When you ask a model to change something, it produces such a list of differences and tries to make it a good one: tidy, clear, solving the task it was given.
By its nature a language model is very good at the task “make change X.” A real engineering task is almost always phrased differently: make change X in such a way that the whole system remains intact afterwards. Almost the entire profession lies between these two phrasings, and the nail salon shows where.
A client’s phone number is stored in far more than one place in the program, even if on the screen it looks like a single field. First, it is recorded in the database, and the way the database is organised, which cells it has and what goes in each, is called the data schema. The schema used to have one cell for the phone number; now it needs a list, and that is already a different schema. Next you have to decide what to do with the three thousand clients whose number is already stored in the old cell, and stored any old way: some with a leading 8, some with +7, and some with a note in the same cell saying “call after six.” Moving old data into the new schema is called a migration, and it is one of the most nerve-racking operations in development, because a mistake in it cannot be undone simply by changing the code back: the data has already been moved.
Then comes the logic of the business itself, which developers call the domain. Whom should the reminder go to if there are three numbers: all of them, the first, the one marked as the main one? Who decides which is the main one if the client has not said? A model nobody asked will answer these questions silently and somehow, and in our case the reminder service kept looking for the phone number in the old cell, which for some of the clients turned out to be empty after the migration.
The client’s number is needed outside the website too: the text-messaging service, the accountant’s export and perhaps something else the administrator has long forgotten about all ask the website for a client’s phone number through a so-called application programming interface, a set of agreements about how one program asks another for data and in what form it receives the answer. The answer used to be a single number and is now a list, and every program that does not know this breaks in its own way.
Then there is the question of permissions. A manicurist used to see a client’s number because she sometimes needs to warn her about a rescheduling. Should a manicurist see all three numbers, including the husband’s? Probably not, but nobody wrote down a rule about it, because until today the second number did not exist.
And the client’s number is also shown in the manicurist’s calendar, on the client’s card, in the administrator’s dashboard, in the booking confirmation email and on the receipt, and each of these places has to be taught either to show a list or to pick one number from it.
What remains are the checks. The program had automatic tests that checked that the phone number was saved and that the reminder went out, and they were written for a single phone number. If they failed after the change, that is good news, because it means the system reported a problem. If the agent, seeing the failed tests, adjusted them so that they passed again, the news is bad, and there will be a separate chapter about that.
The result is a chain of seven links: data schema, migration, logic, connections to other programs, permissions, screens, checks. The administrator’s small request touched all seven, and the change the agent made touched two. Any new capability in a program, even the most modest, changes a connected network in which every link is tied to its neighbours, and to change one link without looking at the others is to leave a breakage that will show up later and somewhere else.
The problem itself is as old as programming. In 1978, at a conference in Chicago, three researchers, Stephen Yau, James Collofello and T. MacGregor, presented a paper on what they called the ripple effect: a change in one part of a program spreads outwards through the rest like rings from a stone thrown into water, and the more complex the program, the farther the rings travel. They even devised a way to estimate in advance how widely the ripple from a particular edit would spread. Ever since, the ability to predict where the ripple will go has been considered one of the main marks of an experienced engineer. A beginner sees the place he was asked to change. An experienced engineer sees what else will shift along with it.
There is a later and harsher formulation of the problem. The Google engineer Hyrum Wright observed that with a sufficient number of users of a program, all of its observable behaviours will be depended on by somebody, regardless of what was promised. Nobody promised to produce the accountant’s export in a “one phone per column” format; it just turned out that way, but the accountant relied on it, and the change broke an agreement that officially did not exist. Programmers call this Hyrum’s law and remember it every time a harmless edit breaks something in the next department.
illustrationThe most expensive illustration of what one small change in a large system looks like came on the first of August 2012, at the American firm Knight Capital, then one of the largest brokers on the stock market. The New York Stock Exchange was launching a new program for retail investors, and Knight wrote new code for it. The company’s trading system had long contained an old piece of code called Power Peg, which had stopped being used back in 2003 but was never removed. It had once contained a counter that tracked how many shares had already been bought for an order and stopped sending instructions as soon as the order was filled. In 2005 that counter was moved to another part of the program, and it occurred to nobody to check whether Power Peg still worked afterwards, since nobody used it anyway. The new code, as the US Securities and Exchange Commission later established, reused the switch that used to turn on that old piece. In itself this is a sensible move: the switch is there, it is free, why not use it.
The new code had to be installed on eight servers. A technician installed it on seven. Nobody noticed, because there was no check that would compare the eight servers with one another. On the morning of the first of August, when trading opened, the switch was turned on, and the new code started running on seven servers, while on the eighth the old Power Peg woke up, received real client orders and began executing them, but now without the counter that had once told it to stop. To fill 212 client orders, the system sent more than four million instructions to the market in forty-five minutes and bought and sold a total of 397 million shares. By the time it was stopped, the company had lost more than 460 million dollars, it had problems with the net capital required by law, and less than a year later it merged with another trading firm and ceased to exist as an independent company.
There is not a single stupid decision in this story if you look at each one separately. The new code, by all accounts, was written correctly. Reusing a free switch is ordinary practice. The technician made the kind of mistake all people make. The disaster arose at the joints: between the change of 2005 and the code nobody checked after it, between the new code and the old code someone forgot to delete, between the switch and its previous meaning, between eight servers that were supposed to be identical. Each link looked sound on its own, and together they brought the company down.
Knight Capital went under in forty-five minutes, but more often the network tears slowly, and then it looks less like an accident and more like a chronic illness. In June 2026 I was examining the structure of a software company in which about one and a half million dollars had been invested by then. The owners had one complaint, and they phrased it almost word for word the same way: every new feature breaks an old one. Over several years the interface and the server side had turned into a zoo in which every animal had its own habits, and nobody could say any longer where the source of truth for the data was: the same figure lived in three places and was slightly different in each. I found no single bad decision there. There were hundreds of sensible changes, each made without asking what would shift next door, and for a million and a half dollars the company had bought itself a system it was afraid to touch.
Language models are especially vulnerable at this point, and the reason is quite technical. A model sees what it has been shown: the files that came into view, the description of the task, the result of the latest check. Everything else does not exist for it. If the reminder service lives in another file the model never got to, it does not know that the service reads the phone number from the old cell. If the accountant’s export is produced by a separate program on the accountant’s own computer, the model will never find out about it. Good agents can search the whole project, and they search better every year, but they will search for what they have been told to, and they will stop when the task looks solved.
Besides, a model has a very clear idea of success: the requested change has been made and the visible checks have passed. When an agent writes “done,” it honestly reports that its list of differences is finished and knows nothing about whether the system is.
From this follows a distinction I consider one of the most important in this book. The model optimises the current change and tries to make it as good as possible. The architect, meaning the person responsible for how the system as a whole is built, minimises disorder across the whole system and asks first of all what else will shift if this is shifted. A model can be taught to ask this question, and further on I will explain how, but for that someone has to know that the question exists at all.
Most people who sit down today to assemble a program with AI for the first time do not know the question exists, and it is not their fault: in the past, to learn about it you had to write code for several years and regularly get your knuckles rapped.