Create Your Own Matchers with Ease

Earlier I showed how you can use Google Mock matchers to make both your test code and your test output readable.  But what if the matcher you need isn't provided by Google Mock?

Don't fret -- you can roll your own matchers easily, either by building on existing ones or writing them from scratch. For the former, Google Mock gives you several composite matchers.  For the latter, you can use some simple macros.

Writing a Composite Matcher

The simplest composite matcher is Not(matcher), which negates the given matcher, as you probably have guessed.

Here's an example: ContainsRegex(s) is a matcher verifying that a string contains regular expression s as a substring.  To verify that a string does not contain the regular expression, write:

EXPECT_THAT(GetUserResponse(), Not(ContainsRegex("Acho+!")));

which may produce a message like:

Value of: GetUserResponse()
Expected: doesn't contain regular expression "Acho+!"
  Actual: "Ah! Achooo!"

AnyOf(matcher_1, ..., matcher_k) is another composite matcher you can use.  It asserts that at least one of the k matchers matches the value.  Therefore

EXPECT_THAT(s, AnyOf(StartsWith("Wow"), ContainsRegex("Acho+!")));

may print:

Value of: s
Expected: (starts with "Wow") or (contains regular expression "Acho+!")
  Actual: "Aha!"

Note how the matcher descriptions are combined.  There is also AllOf(matcher_1, ..., matcher_k) for asserting that all k matchers match the value.

Writing a Matcher from Scratch

Defining matchers from scratch is easy too.  Just use the MATCHER() or MATCHER_Pn() macro.  Here are some examples:

To define a simple matcher to verify that a number is even, just write:

MATCHER(IsEven, "") { return (arg % 2) == 0; }

Ignore the "" in the argument list for now.  As you probably have guessed, the matcher tests whether the argument (referred to by arg) is divisible by 2 -- you can put any code you want inside the {} to verify the argument.  With that, you can write:

EXPECT_THAT(Foo(5), IsEven());

and it may print:

Value of: Foo(5)
Expected: is even
  Actual: 9

The matcher description "is even" is automatically calculated from the matcher's name.  As long as you pick a good name for your matcher, you get the readable description for free.

A matcher can have parameters (as in the example of ContainsRegex("...")).  Such parameterized matchers can be defined using MATCHER_P or MATCHER_Pn, where n is the number of parameters:

MATCHER_P(Contains, element, "") {
  return std::find(arg.begin(), arg.end(), element) != arg.end();
}
...
EXPECT_THAT(prime_set, Not(Contains(12)));
EXPECT_THAT(user_list, Contains(my_user_id));

may print:

Value of: prime_set
Expected: not (contains 12)
  Actual: { 2, 3, 5, 7, 12 }

Value of: user_list
Expected: contains "wan"
  Actual: { "joe", "jon", "nat" }

Note that Contains() is polymorphic: it was first used to match a set<int>, and then used to match a list<string>.  The description 'contains "wan"' is automatically generated from the matcher name and its parameter -- you don't need to worry about a thing.

Also note that you don't need to write the type of the value being matched and the parameter types: the macro automatically infers all of them for you too.

What if you aren't happy with the auto-generated description?  You can override it.  Just write a description pattern instead of "" inside the MATCHER*() macro:

MATCHER_P2(InClosedRange, low, high, "is in range [%(low)s, %(high)s]") {
  return low <= arg && arg <= high;
}
...
EXPECT_THAT(my_age, InClosedRange(adult_min, can_withdraw_401k));

may print:

Value of: my_age
Expected: is in range [18, 60]
  Actual: 71

Note how you can refer to the matcher parameters in the description pattern.

Interested?  You can learn more about the usage of MATCHER* here.

Readable Test Code and Readable Test Output - Get Both Using Matchers

Who doesn't like readable code? The benefit is obvious: bugs are harder to hide in it, and you can change it more easily when the requirement changes.

Test code is code too. Thus it also benefits from being readable. In fact, I'd argue that readability is even more important for test code, as you have to be able to trust it (remember that most tests don't have their own tests).

If you have used a mocking framework before, you are probably familiar with the concept of a matcher: you give it a value, and it tells you whether the value has a certain property -- if yes, we say that it matches the value.

For example, in Google Mock, ContainsRegex("Ahcho+!") is a matcher that matches any string that has the regular expression "Ahcho+!" in it. For instance, it matches "Ahchoo!" and "Ahchoooo! Sorry.", but not "Aha!".

What's this to do with test readability, anyway? It turns out that matchers lend themselves easily to a style of writing assertions that resembles natural languages (and thus is easy for human to understand). For example, Google Mock supports the following assertion syntax, borrowed from the Hamcrest project:

EXPECT_THAT(value, matcher);

The assertion succeeds if the given value matches the given matcher. Therefore,

EXPECT_THAT(my_string, ContainsRegex("ab*"));

verifies that my_string contains regular expression "ab*".

Now, pretend the punctuations aren't in the above C++ statement and read it:

"Expect that my string contains regex ab*".

Does that sound like English?

That's not all. When an EXPECT_THAT assertion fails, it prints an informative message that includes the expression being validated, its value, and the property we expect it to have -- thanks to a matcher's ability to describe itself in human-friendly language. Therefore, not only is the test code readable, the test output it generates is readable too. For example,

EXPECT_THAT(GetUserList(), Contains(admin_id));

might produce:

Value of: GetUserList()
Expected: contains "root"
Actual: { "peter", "paul", "mary" }

This message contains relevant information to help you diagnose the problem, often without having to use a debugger. To get the same effect without using a matcher, you'd have to write something like:

std::vector<std::string> users = GetUserList();
EXPECT_TRUE(VectorContains(users, admin_id))
<< " GetUserList() returns " << users
<< " and admin_id is " << admin_id;

where << is the syntax Google Test uses for appending custom messages to an assertion failure. Clearly this is more tedious to write and harder to read than the one-liner using EXPECT_THAT.

Google Mock provides dozens of matchers for you to use out-of-box. Here's a list of them and their usage.

So matchers are nice. But what if you cannot find a matcher for the property you want to test? You can either combine existing matchers, or define your own from scratch. Both are quite easy to do in Google Mock. I'll show you how in another post. Stay tuned.

What's New in Google Mock 1.1.0

Three months after we released Google C++ Mocking Framework 1.0.0, we were ready to unveil a new version. Google Mock 1.1.0 is available for download now. It contains several new features we are excited about.

The previous version of Google Mock requires Google Test (as a reminder, a mocking framework is not a testing framework - it works with the latter to help you write better tests). We were aware that such requirement posed difficulty for some people, who cannot (yet) switch away from the testing framework they have already been using. This version of Google Mock can be used with a non-Google-Test testing framework. It still works with Google Test out-of-box, but you can easily configure it to use a different testing framework now.

Two other features make Google Mock much easier to extend than before. If you have used any mocking framework, you know that a matcher is good for validating a value and printing informative diagnostics when the validation fails. Google Mock provides many built-in matchers for the common scenarios, but there will always be a time when you need to write your own matcher. There were several ways to do that in the past, but some new macros just made the task an order of magnitude easier. I will cover this more in another post.

The second extensibility improvement is for actions, which you use to specify what to do when a mock object's member function gets called. Similarly, despite the rich set of built-in actions, sometimes a user will need to define his own. The ACTION and ACTION_P* macros make it trivial.

There are some minor improvements too, like new built-in matchers and actions, and better diagnostics, but I'll stop here for now.

What's New in Google Test 1.3.0

A couple of weeks ago we released version 1.3.0 of Google C++ Testing Framework (or Google Test for short). So what's new compared with the previous version, 1.2.1?

This is a major feature release. The most noteworthy things we added are:

• We have brought death tests to more operating systems. In addition to Linux, you can write death tests on Windows, Cygwin, and Mac OS X now. In case you don't know yet, a death test makes sure that your code does crash under certain conditions. Odd this may sound, it is useful for verifying that you have the right assertions in your production code to catch anomalies.
  
• If you cannot switch from your current testing framework to Google Test yet, we now allow you to use Google Test's assertions (including death tests) with your existing framework. You can then try out Google Test in small steps and decide whether you want to switch later.
• Google Test allows you to mark a test function as "disabled." Disabled tests will be compiled but skipped at run time (handy when you want to fix the bug later). Now, you can use the --gtest_also_run_disabled_tests command-line flag to run such tests without editing and recompiling the source code.
  
• The --help flag is added for printing the usage of common Google Test command-line flags, saving you a trip to the manual.
• You can read and change the state of Google Test's flags in tests now. This gives you more control on the behavior of Google Test.
• To make it easy to use Google Test for a quick job, we provide a script to pack all Google Test source code into two files: gtest.h and gtest-all.cc. Just copy the two files to any machine you need to run tests on, and start hacking!

Check it out. We'd like to hear what you think of it.

Why So Many C++ Testing Frameworks Out There?

These days, it seems that everyone is rolling his own C++ testing framework, if he hasn't done so already. Wikipedia has a partial list of such frameworks. This is interesting, as in many other OOP languages there are only one or two major players. For example, most Java people seem to be happy with either JUnit or TestNG. Are C++ programmers the DIY kind?

Since we started to work on Google Test, and especially after we open-sourced it, many people have asked us "why are you doing it?" The short answer, is that we were frustrated by the existing frameworks for one reason or another. This doesn't mean that they are all poorly designed or implemented. Rather, many of them have great ideas and tricks that we learned from. However, as a whole they don't fit our need.

Unlike Java, which has the famous slogan "Write once, run anywhere," C++ code is being written in a much more diverse environment. Due to the complexity of the language and the need to do low-level tasks, compatibility between different C++ compilers and even different versions of the same compiler is poor. There is a C++ Standard, but it's not well supported by compiler vendors. For many tasks you have to rely on unportable extensions or platform-specific functionality. This makes it hard to write a reasonably complex system that can be built using many different compilers and works on many platforms.

To make things more complicated, most C++ compilers allow you to turn off some standard language features in return for better performance. Don't like using exceptions? You can turn it off. Think dynamic cast is bad? You can disable Run-Time Type Identification, the feature behind dynamic cast and run-time access to type information. If you do any of these, however, code using these features will stop to compile and is useless to you. Many testing frameworks rely on exceptions. They are automatically out of the question for us since we turn off exceptions in many projects.

Why don't people just write a portable framework, then? You may ask. Indeed, that's a top design goal for Google Test, as we need to use it on various platforms with different configurations. And authors of some other frameworks have tried this too. However, this comes with a cost. Cross-platform C++ development requires much more effort: you need to test your code with different operating systems, different compilers, different versions of them, and different compiler flags (combine these factors and the task soon gets daunting); some platforms may not let you do certain things and you have to find a workaround there and guard the code with conditional compilation; different versions of compilers have different bugs and you may have to revise your code to bypass them all; etc. In the end, it's hard unless you are happy with a bare-bone system.

So, I think a major reason that we have many C++ testing frameworks is that C++ is different in different environments and it's hard to be portable. John's framework may not suit Bill's environment, even if it solves John's problems perfectly.

Another reason is that some limitation of C++ itself makes it impossible to implement certain features really well. One notable example is that C++ is a statically-typed language and doesn't support reflection. Most Java testing frameworks use reflection to automatically discover tests you've written such that you don't have to register them one-by-one. This is a good thing as manually registering tests is tedious and you can easily write a test and forget to register it. Since C++ has no reflection, we have to do it differently. Unfortunately there is no single best option. Some framework requires you to register tests by hand, some use scripts to parse your source code to discover tests, and some use macros to automate the registration. We prefer the last approach and think it works for most people, but some people don't think so. Also, there are differently ways to devise the macros and they involve different trade-offs, so the result is not clearly cut.

Finally, I believe a good framework should have a good extension story. Let's face it: you cannot be all things to all people, no matter what. Instead of bloating the framework with rarely used features, we should provide good out-of-box solutions for maybe 95% of the use cases, and leave the rest to extensions. If I can easily extend a framework to solve a particular problem of mine, I will feel less motivated to write my own thing. Unfortunately, many framework authors don't seem to see the importance of extensibility. I think that mindset contributed to the plethora of frameworks we see today. In Google Test and Google Mock we make it easy to define your own assertions (or matchers in mocking frameworks' terms) that can be used exactly the same way as the built-in assertions, and we are working on publishing an event listener API such that people can write plug-ins. We hope people will use these features to extend Google Test/Mock for their own need and contribute back extensions that might be generally useful. We shall talk more about them later.