Documentation Beyond Tests

Recently, I encountered an interesting, if twisted, interpretation of the "Tests as Documentation" principle. Reviewing some more-complicated-than-usual code, I asked the responsible colleague, if there was some documentation and, if not, that it would a be good idea to add some. He quickly responded that, of course, he'd provided good test coverage for the code at hand and since we worked under the "tests as documentation" principle it was all there. Indeed, he did write neat, compact tests, who will do their job, and still, in my opinion, he missed the point.

Coming from agile development methodologies, the idea behind this phrase is to minimize unnecessary and inflexible documentation in favor of tests. The latter are maintained anyways and ideally describe the code's behavior precisely. From my point of view, a more specific meaning depends on the kind of test we are talking about and what is documented by it.

Acceptance / customer tests belong to a class of checks, that verify on an abstract level that the system meets its business requirements. Here, the detailed implementation is irrelevant: The question is /if/ and not /how/ the business value is achieved. At this level, the documentation basically is a list of features that are hopefully of importance to someone. In case of larger systems, where there are tons of otherwise forgotton features, this is valuable knowledge.

In the area of unit tests the focus lies on documenting (and fixing) the behavior of a piece of code. By just looking at the tests, it should immediately follow what the code requires as input and what it will produce as output. This is even more so in the case of white-box tests that make the interface of the component precise. As a consequence, the test should be /readable/, in the sense that it ought to come as close to natural language as is feasible. This kind of low-level description is what my collegue meant, and he was right in the sense, that his tests did deliver this.

So, these two kinds of tests fix, define and therefore document behavior (and structures) of a system. Unfortunately, none of them considers /why/ something has been done in a particular way (or at all). Of course, this is by design - the tests do their jobs nicely. My point is, that there is knowledge, that - while absolutely essential - is not and cannot be provided by these tests alone.

Interestingly, on the business side this idea is well understood. If a company does something, there needs to be a good reason to do it and to do it this way. Each of the features that the acceptance tests verify, will be either part of a customer contract or a strategic plan from management. Requirements engineering has found nice solutions for this, e.g. user stories. Because, in the end, cost-effectiveness and accountability are of the essence.

At the level of code, this notion has not yet surfaced equally. If I see some complicated piece of code, I don't only want to know what it does, and if it does it. I want to know why we need it, why the developer chose this particular implementation, and why it must be so complicated.

I believe that the tests above are not capable nor intended to answer these important questions and plead in favor of good ol', natural language, classical documentation.

Rapid Prototyping and Software Engineering?

Today, somehow, I stumbled upon 3d printing, a form of Rapid Prototyping. This reminded me of a talk, delivered by Bre Pettis at 25C3 last year. Again, I began to wonder where this might not only be cool - which it undoubtfully is - but where it also could be useful. (This is not so obvious.) So, as expected, I thought about where to use rapid prototyping at work, i.e. in software engineering. And I think I've identified a spot where it might come in handy. (There are likely to be many more, and I'll post about them, if they come to mind.)

Some time ago, I had an idea to better understand where bad code eats up manpower in terms of support. I figured, given an architecture in adequate detail, it might be useful to print it on a sheet of paper and put that on a pin board. Then, using for example colored pins, we could get an impression of where problems lie. For example, one might want to remember performance bottle necks, occurrences of bugs or black holes in the code, regions that no one understands nor wants to touch.

I've been too busy studying lately to actually put this forward, but still think its a good idea. Easy to use, trivial to evaluate. The only drawback is that it really doesn't look all that nice. (There is worse, but still ...) And here comes 3d printing and saves the day.

Take a look at the following FMC diagram, depicting an architecture of a (still) fictitious browsergame.

Now imagine this as a 3d model, as architects either show them on screen or build them in miniature. The blocks (agents) would rise, the nested ones even further above, the bubbles (storages) would actually be round. Well, the channels might just pose a problem, but anyways! Using materials like polystyrene or plaster, we would be provided with a model of our system that we could make use of as discussed before. With pins, we will be able to mark hot spots and so on. Only this time, it will look cool.

Unfortunately, the 2d solution would work just as fine. So, we're back at square one, having found ourselves a neat solution (to a problem, actually) - but is it actually useful?

If I find out, I'll do another post.

Documentation for IRIS

Lately, IRIS drew my attention. Written in Java, it is "an extensible reasoning engine for expressive rule-based languages." Features include safe and unsafe datalog, magic sets and both locally stratified and well-founded semantics.

The code is well written and documented (most of the times anyways) and there is a paper on how IRIS works. Still, there are no architecture-level documents, introducing the software on such level. (Which is understandable - it primarily serves as a research prototype.)

Since I take interest in reasoning and also don't mind learning new stuff about Java in general, I have begun documenting. The goal is to produce an introduction to fill the gaps from the software-engineering point of view. Hopefully, it will then be of help to people intending to learn about the implementation of IRIS or of reasoners in general. Even better, somebody, for example me, might learn something about modeling and documenting.

Currently, the following is planned and already done in parts:

• high-level overview, visualized by UML and FMC diagrams
• detailed presentation of some interesting algorithms
• introduction into the project's structure
  
• the java code of IRIS, originating from this effort, hopefully here and there with some more comments and clearer at times

The sources (docbook, images, ...) will be available for download, too. Hopefully, in some weeks this will be finished for good.