The Test as a Reward Function

An automated test in programming is a very simple thing. It is a small program that runs a piece of a larger program with data known in advance and checks that the result known in advance comes out. If a manicure costs fifteen hundred and the client has a ten percent discount, the pricing program must return thirteen hundred and fifty, and the test checks that: it calls the calculation, compares the answer with the number and reports “passed” or “failed.” A good project accumulates thousands of such checks, and they are run with every change.

For people, tests were for decades a kind of insurance. A programmer wrote code and then, if he had the time and the conscience, wrote tests that confirmed he had made no mistake and went on confirming it with every subsequent edit. Almost nobody liked writing them, because a test adds nothing to a program that can be shown to the client, and when deadlines were tight, tests were put off until later, and later never came. In 2002 the American programmer Kent Beck published a book on test-driven development in which he proposed doing it the other way round: first write a test describing what you want from the code, make sure it fails, and then write code until the test passes. The method became famous, it was taught in courses and praised at conferences, and most programmers, as far as I can tell, never actually used it.

With the arrival of agents tests have acquired a new meaning, and today, in my view, it matters more than the old one.

---

illustration

To explain it, we need to look for a minute at how machines are trained at all. One of the main methods is called reinforcement learning, and it works much like training a dog. The dog does something, and if it is what the trainer wanted, it gets a treat. Over time it starts doing more often what earns a treat and less often what does not. For machines the treat is replaced by a number called the reward, and the rule by which that number is calculated is called the reward function. The model tries, gets a number, changes so that next time the number is larger, and tries again.