Friday, April 3, 2009

Why So Many C++ Testing Frameworks Out There?

These days, it seems that everyone is rolling his own C++ testing framework, if he hasn't done so already. Wikipedia has a partial list of such frameworks. This is interesting, as in many other OOP languages there are only one or two major players. For example, most Java people seem to be happy with either JUnit or TestNG. Are C++ programmers the DIY kind?

Since we started to work on Google Test, and especially after we open-sourced it, many people have asked us "why are you doing it?" The short answer, is that we were frustrated by the existing frameworks for one reason or another. This doesn't mean that they are all poorly designed or implemented. Rather, many of them have great ideas and tricks that we learned from. However, as a whole they don't fit our need.

Unlike Java, which has the famous slogan "Write once, run anywhere," C++ code is being written in a much more diverse environment. Due to the complexity of the language and the need to do low-level tasks, compatibility between different C++ compilers and even different versions of the same compiler is poor. There is a C++ Standard, but it's not well supported by compiler vendors. For many tasks you have to rely on unportable extensions or platform-specific functionality. This makes it hard to write a reasonably complex system that can be built using many different compilers and works on many platforms.

To make things more complicated, most C++ compilers allow you to turn off some standard language features in return for better performance. Don't like using exceptions? You can turn it off. Think dynamic cast is bad? You can disable Run-Time Type Identification, the feature behind dynamic cast and run-time access to type information. If you do any of these, however, code using these features will stop to compile and is useless to you. Many testing frameworks rely on exceptions. They are automatically out of the question for us since we turn off exceptions in many projects.

Why don't people just write a portable framework, then? You may ask. Indeed, that's a top design goal for Google Test, as we need to use it on various platforms with different configurations. And authors of some other frameworks have tried this too. However, this comes with a cost. Cross-platform C++ development requires much more effort: you need to test your code with different operating systems, different compilers, different versions of them, and different compiler flags (combine these factors and the task soon gets daunting); some platforms may not let you do certain things and you have to find a workaround there and guard the code with conditional compilation; different versions of compilers have different bugs and you may have to revise your code to bypass them all; etc. In the end, it's hard unless you are happy with a bare-bone system.

So, I think a major reason that we have many C++ testing frameworks is that C++ is different in different environments and it's hard to be portable. John's framework may not suit Bill's environment, even if it solves John's problems perfectly.

Another reason is that some limitation of C++ itself makes it impossible to implement certain features really well. One notable example is that C++ is a statically-typed language and doesn't support reflection. Most Java testing frameworks use reflection to automatically discover tests you've written such that you don't have to register them one-by-one. This is a good thing as manually registering tests is tedious and you can easily write a test and forget to register it. Since C++ has no reflection, we have to do it differently. Unfortunately there is no single best option. Some framework requires you to register tests by hand, some use scripts to parse your source code to discover tests, and some use macros to automate the registration. We prefer the last approach and think it works for most people, but some people don't think so. Also, there are differently ways to devise the macros and they involve different trade-offs, so the result is not clearly cut.

Finally, I believe a good framework should have a good extension story. Let's face it: you cannot be all things to all people, no matter what. Instead of bloating the framework with rarely used features, we should provide good out-of-box solutions for maybe 95% of the use cases, and leave the rest to extensions. If I can easily extend a framework to solve a particular problem of mine, I will feel less motivated to write my own thing. Unfortunately, many framework authors don't seem to see the importance of extensibility. I think that mindset contributed to the plethora of frameworks we see today. In Google Test and Google Mock we make it easy to define your own assertions (or matchers in mocking frameworks' terms) that can be used exactly the same way as the built-in assertions, and we are working on publishing an event listener API such that people can write plug-ins. We hope people will use these features to extend Google Test/Mock for their own need and contribute back extensions that might be generally useful. We shall talk more about them later.

5 comments:

  1. If you turn off exceptions, how to you report/catch failures in "new" and constructors?

    ReplyDelete
  2. Ben, we use the exit code of the test program to indicate whether it has succeeded or failed. If new or a ctor fails, and exceptions are disabled, the test program will crash, resulting in a non-zero exit code, which the test runner interprets as a failure.

    ReplyDelete
  3. Hi,
    I am curious about the status of enhancements to googletest that will make memory leak testing much easier. These changes were alluded to earlier in the year, but I haven't seen anything lately. Any update?

    ReplyDelete
  4. David,
    The feature you mentioned is the test event listener API. It's not finished yet, but we are working on it and hope to include it as part of the next gtest release. It should happen in the next few weeks.

    ReplyDelete
  5. simple because everyone want his name to be there at the C++ Testing Tools

    ReplyDelete

Note: Only a member of this blog may post a comment.