These days, it seems that everyone is rolling his own C++ testing framework, if he hasn't done so already. Wikipedia has a
partial list of such frameworks. This is interesting, as in many other OOP languages there are only one or two major players. For example, most Java people seem to be happy with either
JUnit or
TestNG. Are C++ programmers the
DIY kind?
Since we started to work on
Google Test, and especially after we
open-sourced it, many people have asked us "why are you doing it?" The short answer, is that we were frustrated by the existing frameworks for one reason or another. This doesn't mean that they are all poorly designed or implemented. Rather, many of them have great ideas and tricks that we learned from. However, as a whole they don't fit our need.
Unlike Java, which has the famous slogan "Write once, run anywhere," C++ code is being written in a much more diverse environment. Due to the complexity of the language and the need to do low-level tasks, compatibility between different C++ compilers and even different versions of the same compiler is poor. There is a C++ Standard, but it's not well supported by compiler vendors. For many tasks you have to rely on unportable extensions or platform-specific functionality. This makes it hard to write a reasonably complex system that can be built using many different compilers and works on many platforms.
To make things more complicated, most C++ compilers allow you to turn off some standard language features in return for better performance. Don't like using exceptions? You can turn it off. Think dynamic cast is bad? You can disable
Run-Time Type Identification, the feature behind dynamic cast and run-time access to type information. If you do any of these, however, code using these features will stop to compile and is useless to you. Many testing frameworks rely on exceptions. They are automatically out of the question for us since we turn off exceptions in many projects.
Why don't people just write a portable framework, then? You may ask. Indeed, that's a top design goal for Google Test, as we need to use it on various platforms with different configurations. And authors of some other frameworks have tried this too. However, this comes with a cost. Cross-platform C++ development requires much more effort: you need to test your code with different operating systems, different compilers, different versions of them, and different compiler flags (combine these factors and the task soon gets daunting); some platforms may not let you do certain things and you have to find a workaround there and guard the code with conditional compilation; different versions of compilers have different bugs and you may have to revise your code to bypass them all; etc. In the end, it's hard unless you are happy with a bare-bone system.
So, I think a major reason that we have many C++ testing frameworks is that
C++ is different in different environments and it's hard to be portable. John's framework may not suit Bill's environment, even if it solves John's problems perfectly.
Another reason is that
some limitation of C++ itself makes it impossible to implement certain features really well. One notable example is that C++ is a statically-typed language and doesn't support reflection. Most Java testing frameworks use reflection to automatically discover tests you've written such that you don't have to register them one-by-one. This is a good thing as manually registering tests is tedious and you can easily write a test and forget to register it. Since C++ has no reflection, we have to do it differently. Unfortunately there is no single best option. Some framework requires you to register tests by hand, some use scripts to parse your source code to discover tests, and some use macros to automate the registration. We prefer the last approach and think it works for most people, but some people don't think so. Also, there are differently ways to devise the macros and they involve different trade-offs, so the result is not clearly cut.
Finally, I believe a good framework should have a good extension story. Let's face it: you cannot be all things to all people, no matter what. Instead of bloating the framework with rarely used features, we should provide good out-of-box solutions for maybe 95% of the use cases, and leave the rest to extensions. If I can easily extend a framework to solve a particular problem of mine, I will feel less motivated to write my own thing.
Unfortunately, many framework authors don't seem to see the importance of extensibility. I think that mindset contributed to the plethora of frameworks we see today. In Google Test and Google Mock we make it
easy to define your own assertions (or matchers in mocking frameworks' terms) that can be used exactly the same way as the built-in assertions, and we are working on publishing an event listener API such that people can write plug-ins. We hope people will use these features to extend Google Test/Mock for their own need and contribute back extensions that might be generally useful. We shall talk more about them later.
Posts
