Until It Works for People, Nothing Happened

illustration

On the evening of the thirty-first of January 2017 an engineer at GitLab, one of the world’s largest code-hosting services, was dealing with a problem in a database. The database had two servers, the primary and the secondary, and the secondary had fallen behind the primary and stopped copying data from it. To restart the copying, the secondary server had to be wiped. It was late at night in the engineer’s time zone, he was tired, he had been fighting the problem for several hours on end, and at some point he ran a command to delete data in a window that was connected to the wrong server. A second or two later he noticed the mistake and stopped the command. In that second about three hundred gigabytes of production data vanished from the primary server.

The company had several methods of backup, and each had been set up. When the team began checking them, it turned out that none of them worked the way everyone believed. Daily copies to cloud storage were not being made, because the backup program was built for an older version of the database, and it reported its errors by email, which nobody received because of another setting. The other mechanisms either were not enabled for this server or produced copies from which a quick recovery was impossible. What saved the situation was luck: six hours before the accident one of the engineers had taken a snapshot of the database for his own testing needs. Everything was restored from it, while the changes made in those six hours, projects, comments, about seven hundred new accounts, were lost. The recovery was streamed live on YouTube, and thousands of people spent several hours watching data being copied.

Before the accident everything looked exemplary. Backups were configured, the jobs were on the schedule, the documentation said copies were being made. At any meeting the question “do we have backups?” would have been answered “yes,” and the answer would have been sincere. Yet there were no copies, because nobody had ever once tried to restore from them. A backup that nobody has ever restored from exists in roughly the same sense as the software-shaped object of the second chapter.

Something similar happened to me on a far more modest scale, and for that reason, I think, it is more revealing. On the twenty-eighth of March 2026 I was rolling out a new version of one of my projects, initerent. The deployment script was supposed to do four things in order: move the database to the new schema, fill it with initial data, migrate the old documents, and only after that stop the application and bring it up again. The script tripped on its very first line because of a small error in shell syntax, and after that, because of wrongly set configuration and an empty variable holding the account name for the image registry, the migration, the seeding and the document transfer began failing one after another. And then, although not a single preparatory operation had succeeded, the script reached the last step and dutifully stopped the application in order to recreate it on the new version.