What's a unit test?

At it's core, a unit test is simply a function that alerts you to a problem with your code's runtime behavior. This can be as simple as:

bool testAdd() {
  if(add(2,2) != 4) return false;
  if(add(-2, -2) != -4) return false;
}

You just need something that, when run, will loudly tell you that the functionality under test is no longer working.

Why unit test?

There's a billion articles on the internet to convince you that any form of testing is important, and unit tests have their place in that. However, to make this post complete, there need to be some stated goals for what I consider the most important parts of a testing framework. In my opinion the major reasons to bother with unit testing in particular are:

Protection during refactoring

When you refactor your code, you're at risk of changing the behavior of the code. Unit tests can ensure that you did not actually change the behavior.

Ensure bugs are fixed

Unit tests provide a very convient way to find bugs. When a new bug is discovered, or reported by a user, it's useful to create a unit test that explictly triggers the relevant bug. In so doing:

• You can make it easier to find the bug, as you have a function you can keep re-running in the debugger until you locate the problem.
• You also have proof that you've fixed that specific reproduction case. The test will pass if the fix is correct.
• Lastly, you have protection from the bug being introduced again as new features are added.

Document the code's intended behavior

There's a train of thought called "Test Driven Design" where in a developer would choose to write all the tests first, then write the implementation. While I can't completely stand behind this, it does have its place. IMHO, it's equally relevant to write the tests after writing the code.

That said, in the end, the tests can be structured such that it's very clear the intent of a given class. Some tests are going to be there to catch all the "edge cases" where you want the code to return errors. While others are going to represent the desired outcomes. All of those desired outcome tests document for yourself and others, what you wanted the class to do.

Readable test cases

Test cases should be short and to the point. The harder the test case is to grok, the harder it is to ensure correctness. This not only means keeping the content simple, but keeping the format consistent and the operations easy to understand.

For example:

REGISTER_TEST_CASE(suite, testname) {
  // Setup
  const int a = 2;
  const int b = 2;
  
  // Test
  const int ret = add(a, b);
  
  // Validate
  AssertThat(ret, isEqualTo(4));
}

Basic Rules

1. Setup everything required for the test within the test-case

Tests become very hard to understand if they pull all their values from somewhere magicly outside the test.

Don't do this:

REGISTER_TEST_CASE(banktests, testblance) {
  const int balance = bank.getBalance(CHECKING, user);
  
  AssertThat(balance, isEqualTo(400));
}

• What's the setup of bank or user? Was it relevant?
• Where did the value 400 come from?

2. Setup only what is required for the test within the test-case

Don't go too far in the other direction either. Too much setup within each test case makes it difficult to distinguish what's being tested. When the same setup steps are repeated across many test cases it becomes very hard to see minor differences in each test case.

Don't do this:

REGISTER_TEST_CASE(banktests, testblance) {
  Bank impl = new Bank(100);
  User user = impl.newUser(1);
  impl.createCheckingAccount(user);
  impl.deposit(CHECKING, user, 400);
  
  const int balance = impl.getBalance(CHECKING, user);
  
  AssertThat(balance, isEqualTo(400));
}

• Is the setup of bank actually relevant?
• Is the setup of user relevant?

3. Clearly indicate magic values from test values

Lastly, when combined with the above two rules, it's important to write out any magic values so that it's clear they've been setup somewhere else.

A good re-write of the above tests would look like:

REGISTER_TEST_CASE(banktests, testblance) {
  impl.clearBalance(CHECKING, DEFAULT_USER);
  const int expectedBalance = 400;
  impl.deposit(CHECKING, DEFAULT_USER, expectedBalance);
  
  const int actualBalance = bank.getBalance(DEFAULT_USER);
  
  AssertThat(actualBalance, isEqualTo(expectedBalance));
}

• It's clear that DEFAULT_USER is setup somewhere else, and that setup is not relevant.
• No extranious functions besides deposit are called, because we have a default user setup elsewhere.
• The value under test expectedBalance is clearly paired off with the expected value actualBalance.
• The formulation of the balance variables also avoids the arbitrary propagation of magic numbers like 400 throughout the test.

Writing a Test Framework

First off, there's plenty of test frameworks out there for C++, including Google Test and CPP Unit. These are fully-featured test suites, and should be your goto for writing tests. However, what follows is a breakdown of the parts that make up such a framework, and how you could write your own framework.

Test Anything Protocol (TAP)

When you start writing tests, you're quickly going to run into two problems. First, you have a billion tests for each class. Second, you have a billion classes to test. When these two things are true, that's a lot of places your code can fail. Thus, the goal of any test framework should be to validate as many tests as possible and clearly indicate where the failures were. That is to say, one failing test function should not prevent the other billion from running.

The Test Anything Protocol is one, very common, way to output such a result in a machine-readable format. If your test framework can output in TAP format, other tools can easily parse the results to display information about passing and failing tests. Such an output may look like:

1..5
ok 1
ok 2 - # SKIP no network connection
not ok 3 - Result not equal
  adder_tests.cpp:17
  ---
  2 + 2
  got: 0
  expect: 4
  ---
ok 4
ok 5

Thus, the first order of buisness for a testing framework is to re-open std::cout std::cerr std::clog so that normal program output can't trash our pretty TAP formating. Even in C++ the redirects can be handled with the C function freopen Thus ending up with a main test loop like:

FILE *oldOut = stdout;
FILE *oldErr = stderr;

freopen("testlog.STDOUT.txt", "w", stdout);
freopen("testlog.STDERR.txt", "w", stderr);
freopen("testlog.STDLOG.txt", "w", stdlog);

for ( ... tests ... ) {
  switch ( ... test status ... ) {
    case OK:
    fprintf(oldOut, "ok %d - %s\n", test_number, test_message);
    break;

    case ERROR:
    fprintf(oldOut, "not ok %d - %s\n", test_number, test_message);
    break;
  }
}

That said, in the normal case, you'd be using a logging framework, which implies that you'd instead be telling that framework to dump to a file.

std::shared_ptr pLogSink = std::make_shared< core::logging::LoggingFileSink >(
    core::types::BitSet< LL >() | LL::Error | LL::Warning | LL::Info
        | LL::Trace,
    "testlog.txt");
CHECK(core::logging::RegisterSink(pLogSink));

Test Cases

In order to define test cases, some use of C++ RAII, and Macros will be required. Firstly, you don't want to have test cases defined that you forgot to register to be run. Secondly, you want to make the test cases use the minimal amount of code that the user has to memorize. Lastly, you very likely want access to some information available only within __FILE__ __LINE__ __FUNCTION__ macros.

Thus you'll need a class that registers itself at construction time:

class StaticRegister {
  public:
  StaticRegister(
      const char *const file,
      const char *const suiteName,
      const char *const name,
      tTestFunc func) {
    registerTestCase(file, suiteName, name, func);
  }
};

From there, you need to define the structure of your test cases and test suites. A test case being a function that tests a single behavior while a test suite represents some shared state between several behavioral tests. For example, a test suite may be used to encapsulate an expensive database setup that will be used across several test cases.

A test suite requires its own regestration, so that the harness can run a setup() and teardown() methods at the appropriate time during the test process.

Each test case needs access to the information from the test suite, and some way to return the status of the test. The suite information is easily a parameter, however the status can require a lot of thought. If you plan to write a test suite in an environment that allows for throwing of exceptions, a viable choice is to use throw/catch to handle test failures. If you plan to write tests in an environment that disallows throwing of exceptions, then you need to handle either custom jump logic, or use a custom paremeter/return value to pass back the failure information.

From there you can setup a macro with everything required to call that register class and prepare your test case function.

#define REGISTER_TEST_CASE(f_suite_name, f_test_name)                    \
  void f_suite_name##f_test_name(fishy::libtest::detail::TestResult &);  \
  static fishy::libtest::detail::StaticRegister UNIQUE_MACRO_NAME(       \
      __FILE__, #f_suite_name, #f_test_name, f_suite_name##f_test_name); \
  void f_suite_name##f_test_name(                                        \
      fishy::libtest::detail::TestResult &fishy_libtest_detail_testresults)
  
// ...
  
REGISTER_TEST_CASE(foo, bar) {
 // ...
}

Assertions

Assertions are the parts of the test that are written to "assert" that the code did what you wanted it to do.

This is again a place where macros are super useful to dump information like __FILE__ __LINE__ of the relevant error, names of variables, and other information if the assertion fails.

This is also a place where designing a "fluent" api is useful for avoiding errors in the test code. Some test frameworks have functions of the form AssertEqual(expectation, actual) which is super easy to typo as AssertEqual(actual, expectation). This results in erronous test which inverts expectation and reality, making it harder to debug. A framework that provides an inversion of this like TEST(actual, isEqualTo(expectation)) makes it harder to typo.

Some test frameworks provide a minimal set of assertions, which makes the framework simple, but the tests complicated. Again, if you only have a function like TEST(actual, isEqualTo(expectation)) then you'll end up writing many expressions like TEST(foo_produces_container().size(), isEqualTo(5)) which ouputs a horribly not useful error like "expected 5, but was 7", forcing you to read the test to figure out what's mismatched. A more fluent set of assertions can provide a better test experience. If you instead had expressions like TEST(foo_produces_container(), hasSize(5)) the framework can output a more useful "container expecting 5 elements but had 7 elements". Similarly for other common "equality" checks it's useful to have a whole suite of assertions like:

TEST(actual, isEqualTo(expectation));
TEST(actual, isNotEqualTo(expectation));
TEST(actual, isTrue());
TEST(actual, isFalse());
TEST(actual, isNull());
TEST(actual, isNotNull());
TEST(actual, isEmpty());
TEST(actual, isNotEmpty());
TEST(actual, hasSize(expectation));

Github

A full implementation of the above described test tool is available on my github