#ifdef

There’s been a long running conversation about the relationship between static typing and unit testing. As early as 2003, Bob Martin found that unit tests reduce dependence on type safety. Around the same time Bruce Eckel argued that compilation is simply one test for correctness; and that code needs to pass all the tests that define correctness of your program. More recently, Stuart Halloway claims ‘In 5 years, we’ll view compilation as the weakest form of unit testing‘. Are you spotting a trend here?

This all bubbled up again for me recently on an embedded project I’m working on to optimize security policy lookups on a multicore security platform. The task was to optimize performance of the vendor’s C reference implementation. Policy insertion used a simple, but unscalable single linked list having O(n) performance. Of course, the code comes without tests of any kind, so how does one know that the optimization changes don’t break basic functionality?

Obviously, the best thing to do in this situation is to write tests for existing behavior so that as the API is refactored and reimplemented, the tests find any new logic errors. I got payback even before the optimization work got started. The linked list insert (implemented thousands of times by thousands of developers) had a bug in it. Policies that were supposed to be stored in priority order after insert were off by one.

Definition: Reference System n. A software implementation provided by hardware vendors to demonstrate programming of a target device; not intended for deployment in end user products. (2) A software implementation provided by hardware vendors that is for the most part, directly inserted into the systems of end users.

This really isn’t a rant about reference code, it’s about defensive testing and the fact that writing tests are not just for dynamic languages. The Java community is mostly on board with this, but it seems to have gotten slower adoption by the C variant world. So c’mon embedded developers. Let’s stop pretending that the type safety testing by our compilers is good enough.

Here’s one option. I really like check as a unit test tool for C. It has a niced xUnit flavor and ought to be in everyone’s toolbox.

START_TEST(it_should_insert_in priority_order)
{
    Policy* policy5 = alloc_policy_with_priority(5);
    Policy* policy6 = alloc_policy_with_priority(6);
    insert_policy(policy5);
    insert_policy(policy6);
    fail_if(policy_find(policy5) == NULL);
    fail_if(policy_find(policy6) == NULL);
    fail_unless(policy_first() == policy5);
}
END_TEST

And if you happen to be implementing reference code, a set of tests for your API would be a welcome addition to your product; not a program to exercise the hardware, but tests that verify and document the behavior of your APIs. It will reduce the bulk of documentation that needs to be provided to end users and may allow you to remove those weasel words, “not intended for deployment”.