Documentation Beyond Tests

Recently, I encountered an interesting, if twisted, interpretation of the "Tests as Documentation" principle. Reviewing some more-complicated-than-usual code, I asked the responsible colleague, if there was some documentation and, if not, that it would a be good idea to add some. He quickly responded that, of course, he'd provided good test coverage for the code at hand and since we worked under the "tests as documentation" principle it was all there. Indeed, he did write neat, compact tests, who will do their job, and still, in my opinion, he missed the point.

Coming from agile development methodologies, the idea behind this phrase is to minimize unnecessary and inflexible documentation in favor of tests. The latter are maintained anyways and ideally describe the code's behavior precisely. From my point of view, a more specific meaning depends on the kind of test we are talking about and what is documented by it.

Acceptance / customer tests belong to a class of checks, that verify on an abstract level that the system meets its business requirements. Here, the detailed implementation is irrelevant: The question is /if/ and not /how/ the business value is achieved. At this level, the documentation basically is a list of features that are hopefully of importance to someone. In case of larger systems, where there are tons of otherwise forgotton features, this is valuable knowledge.

In the area of unit tests the focus lies on documenting (and fixing) the behavior of a piece of code. By just looking at the tests, it should immediately follow what the code requires as input and what it will produce as output. This is even more so in the case of white-box tests that make the interface of the component precise. As a consequence, the test should be /readable/, in the sense that it ought to come as close to natural language as is feasible. This kind of low-level description is what my collegue meant, and he was right in the sense, that his tests did deliver this.

So, these two kinds of tests fix, define and therefore document behavior (and structures) of a system. Unfortunately, none of them considers /why/ something has been done in a particular way (or at all). Of course, this is by design - the tests do their jobs nicely. My point is, that there is knowledge, that - while absolutely essential - is not and cannot be provided by these tests alone.

Interestingly, on the business side this idea is well understood. If a company does something, there needs to be a good reason to do it and to do it this way. Each of the features that the acceptance tests verify, will be either part of a customer contract or a strategic plan from management. Requirements engineering has found nice solutions for this, e.g. user stories. Because, in the end, cost-effectiveness and accountability are of the essence.

At the level of code, this notion has not yet surfaced equally. If I see some complicated piece of code, I don't only want to know what it does, and if it does it. I want to know why we need it, why the developer chose this particular implementation, and why it must be so complicated.

I believe that the tests above are not capable nor intended to answer these important questions and plead in favor of good ol', natural language, classical documentation.