Another Testing-on-the-Toilet on Matchers

Today we published another Testing-on-the-Toilet episode about matchers on the Google Testing Blog. It shows how easy it is to define custom matchers that enhance your tests' readability. Check it out!

Testing-on-the-Toilet Featuring Google Mock Matchers

As some of you know, Google publishes a "testing on the toilet" series on its testing blog from time to time. These are basically posts available in nicely-formatted PDF that you can print and hang up in the toilet at your workplace. Each episode talks about a particular testing technique or trick.

This week, a "testing on the toilet" episode about matchers in Google Mock was published. You can read it and download the PDF here.

Google Mock Internals: Converting Matchers Safely

After spending the last 4 weeks on writing code, I'm finally ready to get back to blogging.  In my last post, I introduced the basic typing rules for Google Mock's matchers.  At the end of the post, the reader is left with this question:

When we use a composite matcher Not(m) in a context where a Matcher<T> is expected, what type should we require m to have?

As we have seen, it is too restrictive to demand m being a Matcher<T>, as it prevents you from using Not(an_int_matcher) as a Matcher<short>, even if you can convert the short argument to int before giving it to an_int_matcher.

To design the typing rule here in a non-restrictive yet safe way, we need to distinguish between two kinds of conversions between monomorphic matchers: safe conversions and unsafe conversions:

As an example of a safe conversion, you can use a Matcher<int> (let's call it m) as a Matcher<const int&>: you just pass the argument (typed as const int&) to m, which takes an int.  No information will be lost in the process.  The converse is not true: if you use a Matcher<const int&> as a Matcher<int>, you may get a wrong result as the Matcher<const int&> may be interested in the address of the argument (as in the example of Ref(x)).

But what dos it exactly mean for a conversion to be safe?

In general, if type A can be implicitly converted to type B, we can safely convert a Matcher<B> to a Matcher<A> (i.e. Matcher is contravariant): just keep a copy of the original Matcher<B>, convert the argument from type A to B, and then pass it to the underlying Matcher<B>.  There are two exceptions to this rule:

1. When B is a reference and A is not, the conversion is unsafe as the underlying Matcher<B> may be interested in the argument's address, which is not preserved in the conversion from A to B.
2. Lossy conversions between built-in numeric types are implicit in C++.  That is, as far as the compiler is concerned, you can cast a double to an int implicitly.  Therefore, for conversions between built-in numeric types, we impose the additional constraint that they must not be lossy (e.g. converting from int to long is fine, but vice versa is not).

If it's still unclear to you why #2 is necessary, consider this example:

Matcher<int> equals_5 = Eq(5);
Matcher<double> double_equals_5 = SafeMatcherCast<double>(equals_5);

If you use double_equals_5 to match 5.2, you'll get "yes", even though 5.2 isn't equal to 5.

Before we end this post, let recap the basic typing rules for matchers:

• Matchers are statically typed.
• A monomorphic matcher that matches an int value has type Matcher<int>.
• Matcher<int> and Matcher<const int&> are different types: a matcher of the latter type can see the address of its argument, while a matcher of the former type can't.
• A polymorphic matcher m has a type that can be implicitly converted to Matcher<T> if m can be used to match a T value.
• A Matcher<B> can be safely converted to a Matcher<A> as long as A can be implicitly converted to B, the conversion is not from a non-reference to a reference, and it's not a lossy numeric conversion.

Google Mock Internals: How Not to Compare Apples to Oranges

Earlier, we wrote about how matchers can help you validate values in tests and how you can easily define your own matchers.  The latter post also shows that some matchers can match more than one type of values.  We can them polymorphic matchers.  Naturally, matchers that accept only one type of values are called monomorphic matchers.

Since C++ is a statically-typed language, matchers in Google Mock are statically typed.  Namely, a monomorphic matcher that validates a value of type T is implemented as an object of class Matcher<T>.  This lets the compiler catch the user error of giving the matcher a value with the wrong type.  Failing early is far better than returning a bogus answer!

In Google Mock, the context where a matcher is used determines what type of value it receives, and hence determines the matcher's type. For example, if your mock object has a method Foo() that takes an int argument, you can write

EXPECT_CALL(mock, Foo(an_int_matcher));

to verify that Foo() will be called with an argument that matches an_int_matcher.  Here, an_int_matcher is an expression of type Matcher<int>, and the compiler enforces that.

So far it has been simple.  It gets more interesting when the argument is passed by reference, in which case the argument type becomes const T& instead of T (let's ignore the possibility of T& for now) .  What type of matcher should we use to match such an argument?

The answer is Matcher<const T&>.  This type is considered not equivalent to Matcher<T>, so you cannot use the two interchangeably. Such distinction is necessary for catching user errors (again).  For example, the standard Ref(x) matcher matches a reference value that references x.  Its type is Matcher<const T&> where T is x's type.  If you say

EXPECT_CALL(mock, Foo(Ref(an_int_variable)));

where Foo() takes an int argument, you will get a compiler error - a good thing since Ref(an_int_variable) doesn't make sense for an int value.

Yet often we don't care about if a value is passed by value or by reference.  After all, you cannot tell them apart by looking at the call site.  This, is when a polymorphic matcher is useful.

The type of a polymorphic matcher is neither Matcher<T> nor Matcher<const T&>.  Instead, it's a type that can be implicitly converted to Matcher<A>, where A is the argument type.  A can be either a reference or a non-reference.

A frequently-used polymorphic matcher is Eq(x), which matches any value equal to x.  For instance, Eq(5) can be used as Matcher<int>, Matcher<const int&>, Matcher<short>, etc, depending on the context.

In EXPECT_THAT(value, matcher), the context doesn't make it clear whether matcher should b a Matcher<T> or a Matcher<const T&>, where T is the type of value.  To be safe, we treat matcher as a Matcher<const T&> here.  The reason is two-fold: T may not be copyable (in which case Matcher<T> won't compile as it requires passing T by value), and matcher may be interested in the address of value (as in Ref(x)).

It gets even more interesting when we consider the typing rule for composite matchers like Not(m).  Suppose we are using Not(m) where a Matcher<T> is expected.  Since we need to pass the value (with type T) to the underlying matcher m, an obvious idea is to require that m has type Matcher<T> as well.  While this is safe, it is also overly restrictive.  In particular, it doesn't allow us to use Not(an_int_matcher) as a Matcher<short>, even though we could convert the short argument to int and then feed it to an_int_matcher.

In the next post, we will reveal how we define composite matchers in a safe yet non-restrictive way, and how we solve the problem of safely converting between matcher types in general.

Create Your Own Matchers with Ease

Earlier I showed how you can use Google Mock matchers to make both your test code and your test output readable.  But what if the matcher you need isn't provided by Google Mock?

Don't fret -- you can roll your own matchers easily, either by building on existing ones or writing them from scratch. For the former, Google Mock gives you several composite matchers.  For the latter, you can use some simple macros.

Writing a Composite Matcher

The simplest composite matcher is Not(matcher), which negates the given matcher, as you probably have guessed.

Here's an example: ContainsRegex(s) is a matcher verifying that a string contains regular expression s as a substring.  To verify that a string does not contain the regular expression, write:

EXPECT_THAT(GetUserResponse(), Not(ContainsRegex("Acho+!")));

which may produce a message like:

Value of: GetUserResponse()
Expected: doesn't contain regular expression "Acho+!"
  Actual: "Ah! Achooo!"

AnyOf(matcher_1, ..., matcher_k) is another composite matcher you can use.  It asserts that at least one of the k matchers matches the value.  Therefore

EXPECT_THAT(s, AnyOf(StartsWith("Wow"), ContainsRegex("Acho+!")));

may print:

Value of: s
Expected: (starts with "Wow") or (contains regular expression "Acho+!")
  Actual: "Aha!"

Note how the matcher descriptions are combined.  There is also AllOf(matcher_1, ..., matcher_k) for asserting that all k matchers match the value.

Writing a Matcher from Scratch

Defining matchers from scratch is easy too.  Just use the MATCHER() or MATCHER_Pn() macro.  Here are some examples:

To define a simple matcher to verify that a number is even, just write:

MATCHER(IsEven, "") { return (arg % 2) == 0; }

Ignore the "" in the argument list for now.  As you probably have guessed, the matcher tests whether the argument (referred to by arg) is divisible by 2 -- you can put any code you want inside the {} to verify the argument.  With that, you can write:

EXPECT_THAT(Foo(5), IsEven());

and it may print:

Value of: Foo(5)
Expected: is even
  Actual: 9

The matcher description "is even" is automatically calculated from the matcher's name.  As long as you pick a good name for your matcher, you get the readable description for free.

A matcher can have parameters (as in the example of ContainsRegex("...")).  Such parameterized matchers can be defined using MATCHER_P or MATCHER_Pn, where n is the number of parameters:

MATCHER_P(Contains, element, "") {
  return std::find(arg.begin(), arg.end(), element) != arg.end();
}
...
EXPECT_THAT(prime_set, Not(Contains(12)));
EXPECT_THAT(user_list, Contains(my_user_id));

may print:

Value of: prime_set
Expected: not (contains 12)
  Actual: { 2, 3, 5, 7, 12 }

Value of: user_list
Expected: contains "wan"
  Actual: { "joe", "jon", "nat" }

Note that Contains() is polymorphic: it was first used to match a set<int>, and then used to match a list<string>.  The description 'contains "wan"' is automatically generated from the matcher name and its parameter -- you don't need to worry about a thing.

Also note that you don't need to write the type of the value being matched and the parameter types: the macro automatically infers all of them for you too.

What if you aren't happy with the auto-generated description?  You can override it.  Just write a description pattern instead of "" inside the MATCHER*() macro:

MATCHER_P2(InClosedRange, low, high, "is in range [%(low)s, %(high)s]") {
  return low <= arg && arg <= high;
}
...
EXPECT_THAT(my_age, InClosedRange(adult_min, can_withdraw_401k));

may print:

Value of: my_age
Expected: is in range [18, 60]
  Actual: 71

Note how you can refer to the matcher parameters in the description pattern.

Interested?  You can learn more about the usage of MATCHER* here.

Readable Test Code and Readable Test Output - Get Both Using Matchers

Who doesn't like readable code? The benefit is obvious: bugs are harder to hide in it, and you can change it more easily when the requirement changes.

Test code is code too. Thus it also benefits from being readable. In fact, I'd argue that readability is even more important for test code, as you have to be able to trust it (remember that most tests don't have their own tests).

If you have used a mocking framework before, you are probably familiar with the concept of a matcher: you give it a value, and it tells you whether the value has a certain property -- if yes, we say that it matches the value.

For example, in Google Mock, ContainsRegex("Ahcho+!") is a matcher that matches any string that has the regular expression "Ahcho+!" in it. For instance, it matches "Ahchoo!" and "Ahchoooo! Sorry.", but not "Aha!".

What's this to do with test readability, anyway? It turns out that matchers lend themselves easily to a style of writing assertions that resembles natural languages (and thus is easy for human to understand). For example, Google Mock supports the following assertion syntax, borrowed from the Hamcrest project:

EXPECT_THAT(value, matcher);

The assertion succeeds if the given value matches the given matcher. Therefore,

EXPECT_THAT(my_string, ContainsRegex("ab*"));

verifies that my_string contains regular expression "ab*".

Now, pretend the punctuations aren't in the above C++ statement and read it:

"Expect that my string contains regex ab*".

Does that sound like English?

That's not all. When an EXPECT_THAT assertion fails, it prints an informative message that includes the expression being validated, its value, and the property we expect it to have -- thanks to a matcher's ability to describe itself in human-friendly language. Therefore, not only is the test code readable, the test output it generates is readable too. For example,

EXPECT_THAT(GetUserList(), Contains(admin_id));

might produce:

Value of: GetUserList()
Expected: contains "root"
Actual: { "peter", "paul", "mary" }

This message contains relevant information to help you diagnose the problem, often without having to use a debugger. To get the same effect without using a matcher, you'd have to write something like:

std::vector<std::string> users = GetUserList();
EXPECT_TRUE(VectorContains(users, admin_id))
<< " GetUserList() returns " << users
<< " and admin_id is " << admin_id;

where << is the syntax Google Test uses for appending custom messages to an assertion failure. Clearly this is more tedious to write and harder to read than the one-liner using EXPECT_THAT.

Google Mock provides dozens of matchers for you to use out-of-box. Here's a list of them and their usage.

So matchers are nice. But what if you cannot find a matcher for the property you want to test? You can either combine existing matchers, or define your own from scratch. Both are quite easy to do in Google Mock. I'll show you how in another post. Stay tuned.

What's New in Google Mock 1.1.0

Three months after we released Google C++ Mocking Framework 1.0.0, we were ready to unveil a new version. Google Mock 1.1.0 is available for download now. It contains several new features we are excited about.

The previous version of Google Mock requires Google Test (as a reminder, a mocking framework is not a testing framework - it works with the latter to help you write better tests). We were aware that such requirement posed difficulty for some people, who cannot (yet) switch away from the testing framework they have already been using. This version of Google Mock can be used with a non-Google-Test testing framework. It still works with Google Test out-of-box, but you can easily configure it to use a different testing framework now.

Two other features make Google Mock much easier to extend than before. If you have used any mocking framework, you know that a matcher is good for validating a value and printing informative diagnostics when the validation fails. Google Mock provides many built-in matchers for the common scenarios, but there will always be a time when you need to write your own matcher. There were several ways to do that in the past, but some new macros just made the task an order of magnitude easier. I will cover this more in another post.

The second extensibility improvement is for actions, which you use to specify what to do when a mock object's member function gets called. Similarly, despite the rich set of built-in actions, sometimes a user will need to define his own. The ACTION and ACTION_P* macros make it trivial.

There are some minor improvements too, like new built-in matchers and actions, and better diagnostics, but I'll stop here for now.