Musings on C++ Testing Tools

Another Testing-on-the-Toilet on Matchers

2009-10-05T09:40:00.000-07:00

Today we published another Testing-on-the-Toilet episode about matchers on the Google Testing Blog. It shows how easy it is to define custom matchers that enhance your tests' readability. Check it out!

Testing-on-the-Toilet Featuring Google Mock Matchers

2009-09-30T16:32:00.000-07:00

As some of you know, Google publishes a "testing on the toilet" series on its testing blog from time to time. These are basically posts available in nicely-formatted PDF that you can print and hang up in the toilet at your workplace. Each episode talks about a particular testing technique or trick.

This week, a "testing on the toilet" episode about matchers in Google Mock was published. You can read it and download the PDF here.

Google Mock Internals: Converting Matchers Safely

2009-06-16T11:49:00.001-07:00

After spending the last 4 weeks on writing code, I'm finally ready to get back to blogging. In my last post, I introduced the basic typing rules for Google Mock's matchers. At the end of the post, the reader is left with this question:

When we use a composite matcher Not(m) in a context where a Matcher<T> is expected, what type should we require m to have?

As we have seen, it is too restrictive to demand m being a Matcher<T>, as it prevents you from using Not(an_int_matcher) as a Matcher<short>, even if you can convert the short argument to int before giving it to an_int_matcher.

To design the typing rule here in a non-restrictive yet safe way, we need to distinguish between two kinds of conversions between monomorphic matchers: safe conversions and unsafe conversions:

As an example of a safe conversion, you can use a Matcher<int> (let's call it m) as a Matcher<const int&>: you just pass the argument (typed as const int&) to m, which takes an int. No information will be lost in the process. The converse is not true: if you use a Matcher<const int&> as a Matcher<int>, you may get a wrong result as the Matcher<const int&> may be interested in the address of the argument (as in the example of Ref(x)).

But what dos it exactly mean for a conversion to be safe?

In general, if type A can be implicitly converted to type B, we can safely convert a Matcher to a Matcher<A> (i.e. Matcher is contravariant): just keep a copy of the original Matcher, convert the argument from type A to B, and then pass it to the underlying Matcher. There are two exceptions to this rule:

When B is a reference and A is not, the conversion is unsafe as the underlying Matcher may be interested in the argument's address, which is not preserved in the conversion from A to B.
Lossy conversions between built-in numeric types are implicit in C++. That is, as far as the compiler is concerned, you can cast a double to an int implicitly. Therefore, for conversions between built-in numeric types, we impose the additional constraint that they must not be lossy (e.g. converting from int to long is fine, but vice versa is not).

If it's still unclear to you why #2 is necessary, consider this example:

Matcher<int> equals_5 = Eq(5);
Matcher<double> double_equals_5 = SafeMatcherCast<double>(equals_5);

If you use double_equals_5 to match 5.2, you'll get "yes", even though 5.2 isn't equal to 5.

Before we end this post, let recap the basic typing rules for matchers:

Matchers are statically typed.
A monomorphic matcher that matches an int value has type Matcher<int>.
Matcher<int> and Matcher<const int&> are different types: a matcher of the latter type can see the address of its argument, while a matcher of the former type can't.
A polymorphic matcher m has a type that can be implicitly converted to Matcher<T> if m can be used to match a T value.
A Matcher can be safely converted to a Matcher<A> as long as A can be implicitly converted to B, the conversion is not from a non-reference to a reference, and it's not a lossy numeric conversion.

Google Mock Internals: How Not to Compare Apples to Oranges

2009-05-19T00:22:00.000-07:00

Earlier, we wrote about how matchers can help you validate values in tests and how you can easily define your own matchers. The latter post also shows that some matchers can match more than one type of values. We can them polymorphic matchers. Naturally, matchers that accept only one type of values are called monomorphic matchers.

Since C++ is a statically-typed language, matchers in Google Mock are statically typed. Namely, a monomorphic matcher that validates a value of type T is implemented as an object of class Matcher<T>. This lets the compiler catch the user error of giving the matcher a value with the wrong type. Failing early is far better than returning a bogus answer!

In Google Mock, the context where a matcher is used determines what type of value it receives, and hence determines the matcher's type. For example, if your mock object has a method Foo() that takes an int argument, you can write

EXPECT_CALL(mock, Foo(an_int_matcher));

to verify that Foo() will be called with an argument that matches an_int_matcher. Here, an_int_matcher is an expression of type Matcher<int>, and the compiler enforces that.

So far it has been simple. It gets more interesting when the argument is passed by reference, in which case the argument type becomes const T& instead of T (let's ignore the possibility of T& for now) . What type of matcher should we use to match such an argument?

The answer is Matcher<const T&>. This type is considered not equivalent to Matcher<T>, so you cannot use the two interchangeably. Such distinction is necessary for catching user errors (again). For example, the standard Ref(x) matcher matches a reference value that references x. Its type is Matcher<const T&> where T is x's type. If you say

EXPECT_CALL(mock, Foo(Ref(an_int_variable)));

where Foo() takes an int argument, you will get a compiler error - a good thing since Ref(an_int_variable) doesn't make sense for an int value.

Yet often we don't care about if a value is passed by value or by reference. After all, you cannot tell them apart by looking at the call site. This, is when a polymorphic matcher is useful.

The type of a polymorphic matcher is neither Matcher<T> nor Matcher<const T&>. Instead, it's a type that can be implicitly converted to Matcher<A>, where A is the argument type. A can be either a reference or a non-reference.

A frequently-used polymorphic matcher is Eq(x), which matches any value equal to x. For instance, Eq(5) can be used as Matcher<int>, Matcher<const int&>, Matcher<short>, etc, depending on the context.

In EXPECT_THAT(value, matcher), the context doesn't make it clear whether matcher should b a Matcher<T> or a Matcher<const T&>, where T is the type of value. To be safe, we treat matcher as a Matcher<const T&> here. The reason is two-fold: T may not be copyable (in which case Matcher<T> won't compile as it requires passing T by value), and matcher may be interested in the address of value (as in Ref(x)).

It gets even more interesting when we consider the typing rule for composite matchers like Not(m). Suppose we are using Not(m) where a Matcher<T> is expected. Since we need to pass the value (with type T) to the underlying matcher m, an obvious idea is to require that m has type Matcher<T> as well. While this is safe, it is also overly restrictive. In particular, it doesn't allow us to use Not(an_int_matcher) as a Matcher<short>, even though we could convert the short argument to int and then feed it to an_int_matcher.

In the next post, we will reveal how we define composite matchers in a safe yet non-restrictive way, and how we solve the problem of safely converting between matcher types in general.

Create Your Own Matchers with Ease

2009-04-17T22:44:00.001-07:00

Earlier I showed how you can use Google Mock matchers to make both your test code and your test output readable. But what if the matcher you need isn't provided by Google Mock?

Don't fret -- you can roll your own matchers easily, either by building on existing ones or writing them from scratch. For the former, Google Mock gives you several composite matchers. For the latter, you can use some simple macros.

Writing a Composite Matcher

The simplest composite matcher is Not(matcher), which negates the given matcher, as you probably have guessed.

Here's an example: ContainsRegex(s) is a matcher verifying that a string contains regular expression s as a substring. To verify that a string does not contain the regular expression, write:

EXPECT_THAT(GetUserResponse(), Not(ContainsRegex("Acho+!")));

which may produce a message like:

Value of: GetUserResponse()
Expected: doesn't contain regular expression "Acho+!"
Actual: "Ah! Achooo!"

AnyOf(matcher_1, ..., matcher_k) is another composite matcher you can use. It asserts that at least one of the k matchers matches the value. Therefore

EXPECT_THAT(s, AnyOf(StartsWith("Wow"), ContainsRegex("Acho+!")));

may print:

Value of: s
Expected: (starts with "Wow") or (contains regular expression "Acho+!")
Actual: "Aha!"

Note how the matcher descriptions are combined. There is also AllOf(matcher_1, ..., matcher_k) for asserting that all k matchers match the value.

Writing a Matcher from Scratch

Defining matchers from scratch is easy too. Just use the MATCHER() or MATCHER_Pn() macro. Here are some examples:

To define a simple matcher to verify that a number is even, just write:

MATCHER(IsEven, "") { return (arg % 2) == 0; }

Ignore the "" in the argument list for now. As you probably have guessed, the matcher tests whether the argument (referred to by arg) is divisible by 2 -- you can put any code you want inside the {} to verify the argument. With that, you can write:

EXPECT_THAT(Foo(5), IsEven());

and it may print:

Value of: Foo(5)
Expected: is even
Actual: 9

The matcher description "is even" is automatically calculated from the matcher's name. As long as you pick a good name for your matcher, you get the readable description for free.

A matcher can have parameters (as in the example of ContainsRegex("...")). Such parameterized matchers can be defined using MATCHER_P or MATCHER_Pn, where n is the number of parameters:

MATCHER_P(Contains, element, "") {
return std::find(arg.begin(), arg.end(), element) != arg.end();
}
...
EXPECT_THAT(prime_set, Not(Contains(12)));
EXPECT_THAT(user_list, Contains(my_user_id));

may print:

Value of: prime_set
Expected: not (contains 12)
Actual: { 2, 3, 5, 7, 12 }

Value of: user_list
Expected: contains "wan"
Actual: { "joe", "jon", "nat" }

Note that Contains() is polymorphic: it was first used to match a set<int>, and then used to match a list<string>. The description 'contains "wan"' is automatically generated from the matcher name and its parameter -- you don't need to worry about a thing.

Also note that you don't need to write the type of the value being matched and the parameter types: the macro automatically infers all of them for you too.

What if you aren't happy with the auto-generated description? You can override it. Just write a description pattern instead of "" inside the MATCHER*() macro:

MATCHER_P2(InClosedRange, low, high, "is in range [%(low)s, %(high)s]") {
return low <= arg && arg <= high;
}
...
EXPECT_THAT(my_age, InClosedRange(adult_min, can_withdraw_401k));

may print:

Value of: my_age
Expected: is in range [18, 60]
Actual: 71

Note how you can refer to the matcher parameters in the description pattern.

Interested? You can learn more about the usage of MATCHER* here.

Readable Test Code and Readable Test Output - Get Both Using Matchers

2009-04-09T15:52:00.000-07:00

Who doesn't like readable code? The benefit is obvious: bugs are harder to hide in it, and you can change it more easily when the requirement changes.

Test code is code too. Thus it also benefits from being readable. In fact, I'd argue that readability is even more important for test code, as you have to be able to trust it (remember that most tests don't have their own tests).

If you have used a mocking framework before, you are probably familiar with the concept of a matcher: you give it a value, and it tells you whether the value has a certain property -- if yes, we say that it matches the value.

For example, in Google Mock, ContainsRegex("Ahcho+!") is a matcher that matches any string that has the regular expression "Ahcho+!" in it. For instance, it matches "Ahchoo!" and "Ahchoooo! Sorry.", but not "Aha!".

What's this to do with test readability, anyway? It turns out that matchers lend themselves easily to a style of writing assertions that resembles natural languages (and thus is easy for human to understand). For example, Google Mock supports the following assertion syntax, borrowed from the Hamcrest project:

EXPECT_THAT(value, matcher);

The assertion succeeds if the given value matches the given matcher. Therefore,

EXPECT_THAT(my_string, ContainsRegex("ab*"));

verifies that my_string contains regular expression "ab*".

Now, pretend the punctuations aren't in the above C++ statement and read it:

"Expect that my string contains regex ab*".

Does that sound like English?

That's not all. When an EXPECT_THAT assertion fails, it prints an informative message that includes the expression being validated, its value, and the property we expect it to have -- thanks to a matcher's ability to describe itself in human-friendly language. Therefore, not only is the test code readable, the test output it generates is readable too. For example,

EXPECT_THAT(GetUserList(), Contains(admin_id));

might produce:

Value of: GetUserList()
Expected: contains "root"
Actual: { "peter", "paul", "mary" }

This message contains relevant information to help you diagnose the problem, often without having to use a debugger. To get the same effect without using a matcher, you'd have to write something like:

std::vector<std::string> users = GetUserList();
EXPECT_TRUE(VectorContains(users, admin_id))
<< " GetUserList() returns " << users
<< " and admin_id is " << admin_id;

where << is the syntax Google Test uses for appending custom messages to an assertion failure. Clearly this is more tedious to write and harder to read than the one-liner using EXPECT_THAT.

Google Mock provides dozens of matchers for you to use out-of-box. Here's a list of them and their usage.

So matchers are nice. But what if you cannot find a matcher for the property you want to test? You can either combine existing matchers, or define your own from scratch. Both are quite easy to do in Google Mock. I'll show you how in another post. Stay tuned.

What's New in Google Mock 1.1.0

2009-04-03T15:45:00.000-07:00

Three months after we released Google C++ Mocking Framework 1.0.0, we were ready to unveil a new version. Google Mock 1.1.0 is available for download now. It contains several new features we are excited about.

The previous version of Google Mock requires Google Test (as a reminder, a mocking framework is not a testing framework - it works with the latter to help you write better tests). We were aware that such requirement posed difficulty for some people, who cannot (yet) switch away from the testing framework they have already been using. This version of Google Mock can be used with a non-Google-Test testing framework. It still works with Google Test out-of-box, but you can easily configure it to use a different testing framework now.

Two other features make Google Mock much easier to extend than before. If you have used any mocking framework, you know that a matcher is good for validating a value and printing informative diagnostics when the validation fails. Google Mock provides many built-in matchers for the common scenarios, but there will always be a time when you need to write your own matcher. There were several ways to do that in the past, but some new macros just made the task an order of magnitude easier. I will cover this more in another post.

The second extensibility improvement is for actions, which you use to specify what to do when a mock object's member function gets called. Similarly, despite the rich set of built-in actions, sometimes a user will need to define his own. The ACTION and ACTION_P* macros make it trivial.

There are some minor improvements too, like new built-in matchers and actions, and better diagnostics, but I'll stop here for now.

What's New in Google Test 1.3.0

2009-04-03T15:07:00.000-07:00

A couple of weeks ago we released version 1.3.0 of Google C++ Testing Framework (or Google Test for short). So what's new compared with the previous version, 1.2.1?

This is a major feature release. The most noteworthy things we added are:

We have brought death tests to more operating systems. In addition to Linux, you can write death tests on Windows, Cygwin, and Mac OS X now. In case you don't know yet, a death test makes sure that your code does crash under certain conditions. Odd this may sound, it is useful for verifying that you have the right assertions in your production code to catch anomalies.
If you cannot switch from your current testing framework to Google Test yet, we now allow you to use Google Test's assertions (including death tests) with your existing framework. You can then try out Google Test in small steps and decide whether you want to switch later.
Google Test allows you to mark a test function as "disabled." Disabled tests will be compiled but skipped at run time (handy when you want to fix the bug later). Now, you can use the --gtest_also_run_disabled_tests command-line flag to run such tests without editing and recompiling the source code.
The --help flag is added for printing the usage of common Google Test command-line flags, saving you a trip to the manual.
You can read and change the state of Google Test's flags in tests now. This gives you more control on the behavior of Google Test.
To make it easy to use Google Test for a quick job, we provide a script to pack all Google Test source code into two files: gtest.h and gtest-all.cc. Just copy the two files to any machine you need to run tests on, and start hacking!

Check it out. We'd like to hear what you think of it.

Why So Many C++ Testing Frameworks Out There?

2009-04-03T13:04:00.000-07:00

These days, it seems that everyone is rolling his own C++ testing framework, if he hasn't done so already. Wikipedia has a partial list of such frameworks. This is interesting, as in many other OOP languages there are only one or two major players. For example, most Java people seem to be happy with either JUnit or TestNG. Are C++ programmers the DIY kind?

Since we started to work on Google Test, and especially after we open-sourced it, many people have asked us "why are you doing it?" The short answer, is that we were frustrated by the existing frameworks for one reason or another. This doesn't mean that they are all poorly designed or implemented. Rather, many of them have great ideas and tricks that we learned from. However, as a whole they don't fit our need.

Unlike Java, which has the famous slogan "Write once, run anywhere," C++ code is being written in a much more diverse environment. Due to the complexity of the language and the need to do low-level tasks, compatibility between different C++ compilers and even different versions of the same compiler is poor. There is a C++ Standard, but it's not well supported by compiler vendors. For many tasks you have to rely on unportable extensions or platform-specific functionality. This makes it hard to write a reasonably complex system that can be built using many different compilers and works on many platforms.

To make things more complicated, most C++ compilers allow you to turn off some standard language features in return for better performance. Don't like using exceptions? You can turn it off. Think dynamic cast is bad? You can disable Run-Time Type Identification, the feature behind dynamic cast and run-time access to type information. If you do any of these, however, code using these features will stop to compile and is useless to you. Many testing frameworks rely on exceptions. They are automatically out of the question for us since we turn off exceptions in many projects.

Why don't people just write a portable framework, then? You may ask. Indeed, that's a top design goal for Google Test, as we need to use it on various platforms with different configurations. And authors of some other frameworks have tried this too. However, this comes with a cost. Cross-platform C++ development requires much more effort: you need to test your code with different operating systems, different compilers, different versions of them, and different compiler flags (combine these factors and the task soon gets daunting); some platforms may not let you do certain things and you have to find a workaround there and guard the code with conditional compilation; different versions of compilers have different bugs and you may have to revise your code to bypass them all; etc. In the end, it's hard unless you are happy with a bare-bone system.

So, I think a major reason that we have many C++ testing frameworks is that C++ is different in different environments and it's hard to be portable. John's framework may not suit Bill's environment, even if it solves John's problems perfectly.

Another reason is that some limitation of C++ itself makes it impossible to implement certain features really well. One notable example is that C++ is a statically-typed language and doesn't support reflection. Most Java testing frameworks use reflection to automatically discover tests you've written such that you don't have to register them one-by-one. This is a good thing as manually registering tests is tedious and you can easily write a test and forget to register it. Since C++ has no reflection, we have to do it differently. Unfortunately there is no single best option. Some framework requires you to register tests by hand, some use scripts to parse your source code to discover tests, and some use macros to automate the registration. We prefer the last approach and think it works for most people, but some people don't think so. Also, there are differently ways to devise the macros and they involve different trade-offs, so the result is not clearly cut.

Finally, I believe a good framework should have a good extension story. Let's face it: you cannot be all things to all people, no matter what. Instead of bloating the framework with rarely used features, we should provide good out-of-box solutions for maybe 95% of the use cases, and leave the rest to extensions. If I can easily extend a framework to solve a particular problem of mine, I will feel less motivated to write my own thing. Unfortunately, many framework authors don't seem to see the importance of extensibility. I think that mindset contributed to the plethora of frameworks we see today. In Google Test and Google Mock we make it easy to define your own assertions (or matchers in mocking frameworks' terms) that can be used exactly the same way as the built-in assertions, and we are working on publishing an event listener API such that people can write plug-ins. We hope people will use these features to extend Google Test/Mock for their own need and contribute back extensions that might be generally useful. We shall talk more about them later.