26 November 2017

Test All The Things

Is this thing on? Hello? Great, you can see me. This time is all about unit testing in Perl 6. Are you curious about what that t/ directory is for, and want to fill that empty space with some test files? You've come to the right post. If you don't know what I'm talking about, now might be a good idea to go look at a few Perl 6 modules and check out the testing directory.

Perl has a long tradition of extensive test suites for its modules, and Perl 6 continues that tradition. You can start by downloading a module from GitHub following the Perl 6 modules link and checking out its t/ directory.

Perl 6 comes with a built-in Test module, which looks a lot like Perl 5's Test::More module. I'm not going to go into great detail (in this post, at least) about what all the methods do, I'm going to focus on just two or three ideas that I came up with when writing my own test suites.

DRYing out your tests

Sometimes you get into the zone writing tests, and your tests start to bunch up.

is sprintf( "%s", "a" ), "a", "'a' roundtrips.";
is sprintf( "%s", "€" ), "€", "'€' roundtrips.";
is sprintf( "%s", "\x[263a]" ), "\x[263a]", "Smiley roundtrips.";

While something of a contrived example, it's pretty obvious that all of these tests should be bundled together. You certainly could put them in their own file, and in this case it might be a good idea because being able to roundtrip strings (make sure that the input is the same as the output) needs some pretty thorough testing.

Right now, though, as it stands, three lines hardly is worth the effort to think of a new name for the file, copy to the new location, delete the old content, add it to git and do a push. Let's come up with an easier way to group these.

Visually separating them with a block certainly works...

{
  is sprintf( "%s", "a" ), "a", "'a' roundtrips.";
  is sprintf( "%s", "€" ), "€", "'€' roundtrips.";
  is sprintf( "%s", "\x[263a]" ), "\x[263a]", "Smiley roundtrips.";
}

While it certainly looks better, it doesn't help the repetition any. "roundtrips" still repeats, and every time we come up with a new string that might break sprintf() we've got to add it in three places. Let's tackle that first, before going on to the final round.

It's certainly tempting to write a quick subroutine to do this, so let's dash off one.

sub roundtrip( $name ) {
  is sprintf( "%s", $name ), $name, "'$name' roundtrips."
}
{
  roundtrip( "a" );
  roundtrip( "€" );
  roundtrip( "\x[263a]" );
}

Yippee! We've eliminated almost all of the redundancy, and our test output still works!

ok 1 - 'a' roundtrips.
ok 2 - '€' roundtrips.
ok 3 - '☺' roundtrips.

There's a problem lurking here, though. A couple, actually. What happens when sprintf() breaks?

ok 1 - 'a' roundtrips.
not ok 2 - '€' roundtrips.
# Failed test ''€' roundtrips.'
# at t/10-sprintf.t line 4
ok 3 - '☺' roundtrips.

Pretending we're in a hurry and this is just one of a number of problems we have to debug this evening (just like in real life), open your editor and go to line 4 to quickly trace down the bug...

use Test;

sub roundtrip( $name ) {
  is sprintf( "%s", $name ), $name, "'$name' roundtrips." # line 4
}
{
  roundtrip( "a" );
  roundtrip( "€" );
  roundtrip( "\x[263a]" );
}

Hold the phone here, I expected to jump to the test that failed, and I'm on a test subroutine! This would get even more confusing if I had a test library, and roundtrip() wasn't even in my test file. It'd be even a bit confusing if I just saw the word 'roundtrip' repeated over and over, and just searched for that. Or even worse, imagine that this is a file with 200+ tests in it, and every tenth test is for a low Unicode character, so you've got to page through 20 different tests 'til you find the right one. There's got to be a better way.

Now you could certainly throw away your changes, roll back to the point where you refactored and call the time a waste. It's easy enough to salvage, though.

use Test;

sub roundtrip( $name ) {
  sprintf( "%s", $name )
}
{
  is roundtrip( "a" ), "a", "'a' roundtrips.";
  is roundtrip( "€" ), "€", "'€' roundtrips.";
  is roundtrip( "\x[263a]" ), "\x[263a]", "'\x[263a]' roundtrips.";
}

This solves the problem, so when an is() test fails, we'll get pointed directly at the line number, and can jump there in Atom, Emacs, Vim or whatever. But we've gotten our duplication back. Let's try to refactor our way out of this, while making sure that we don't put the is() back into the helper roundtrip() subroutine.

We'll start by noting that is( $string, $roundtripped ) (ignoring the $message parameter) is equivalent to ok( $string eq $roundtripped ). Change the test from is() to ok() first, then add eq $name inside roundtrip(), and get rid of the redundant argument.

use Test;

sub roundtrip( $name ) {
  sprintf( "%s", $name ) eq $name
}
{
  ok roundtrip( "a" ), "'a' roundtrips.";
  ok roundtrip( "€" ), "'€' roundtrips.";
  ok roundtrip( "\x[263a]" ), "'\x[263a]' roundtrips.";
}

That's pretty good, but there's a constant 'roundtrips.' string there. Also we've got this {..} block that's unused, so let's put that block to work by factoring out the 'roundtrips.' bit, using subtest {..}.

use Test;

sub roundtrip( $name ) {
  sprintf( "%s", $name ) eq $name
}
subtest 'roundtrips', {
  ok roundtrip( "a" ), "'a'";
  ok roundtrip( "€" ), "'€'";
  ok roundtrip( "\x[263a]" ), "'\x[263a]'";
};

Looking good, but we've lost a bit along the way. Earlier, when we ran our test suite, we'd get nicely labeled output. Now... not so much.

  ok 1 - 'a'
  ok 2 - '€'
  ok 3 - '☺'
ok 1 - roundtrips

In a simple, short file like this, scanning from the ok 1 - 'a' line, thinking "Okay, why are we testing 'a'?... Aha, roundtrip tests." is pretty quick, and the indentation tells us we're grouping a bunch of tests, but it would be really nice to be able to put that test message where it belongs, in the roundtrip message. So let's do just that.

use Test;

sub roundtrip( $name ) {
  sprintf( "%s", $name ) eq $name, "'$name' roundtrips."
}
subtest 'roundtrips', {
  ok roundtrip( "a" );
  ok roundtrip( "€" );
  ok roundtrip( "\x[263a]" );
};

Great, we've got just one test function that tests and gives us a message! Let's run it!

  ok 1 -
  ok 2 -
  ok 3 -
ok 1 - roundtrips

What's going on here? roundtrip() returns both the truthiness of our test and the message correctly, you can test that on the command line yourself. Yes, Virginia, Perl 6 subroutines can return more than one value - perl6 -e'sub test { "foo", "bar" }; say test();' will return [ foo bar ] as you'd expect.

So ok() is getting the list that roundtrip() returns, and should be unpacking that list and...hey, waitaminnit. roundtrip() returns a list, and ok() expects one argument and one optional argument... maybe that's what's going wrong here. So, how do we solve this?

Luckily for us, there's an easy way to unpack our list into two arguments. It's not quite destructuring (there's another way to do that) but it works for us. The | (pipe) symbol before a list "expands" that list inline into a bunch of individual arguments, so let's put that before the roundtrip() call and unpack the list.

use Test;

sub roundtrip( $name ) {
  sprintf( "%s", $name ) eq $name, "'$name' roundtrips."
}
subtest 'roundtrips', {
  ok |roundtrip( "a" );
  ok |roundtrip( "€" );
  ok |roundtrip( "\x[263a]" );
};

And rerunning our test suite now, our output is what we expect.

  ok 1 - 'a' roundtrips.
  ok 2 - '€' roundtrips.
  ok 3 - '☺' roundtrips.
ok 1 - roundtrips

This may be a bridge too far for some, and I respect your decision. Using the subtest() block may be all you need, because you can quickly search for a keyword in the string and bounce immediately to the start of the tests where one fails. You may even want to go to the lengths of using ok( roundtrip($_) ) for < a € ☺ > to get rid of the last duplicated call to roundtrip(), that's your choice. all I'm offering here is some ways to DRY out your test files.

Gentle Reader, if you've gotten this far, thank you. I do read all the comments that I get, at least eventually. I'm also @DrForr on Twitter, 'Jeff Goff' on Facebook and LinkedIn. Thank you for your time, and I hope it was worth your while.

3 comments:

  1. Wouldn't it be rather more elegant to just write a new testing function "roundtrip"? I'm imagining something like this:

    roundtrip { sprintf( "%s", $name ) }, [ "a", "€", "\x[263a]" ];

    ReplyDelete
  2. That's certainly an alternative. If 'roundtrip' were to work along the same lines as 'ok' and 'like', you would end up subclassing Test or monkeypatching the Test class in order to get access to the current test index and nesting depth, in order to make it play nicely with existing code.

    It's certainly possible, though a bit advanced. I was trying to create something that I could throw into a t/lib/ directory and treat as part of the proverbial furniture. Thinking more about it, I suppose the next logical step would be to turn the t/lib/Utils.pm6 file that I've created into a custom Test subclass. Let's call that next week's posting, shall we?

    Thanks again for your comments and thoughts, keep 'em coming.

    ReplyDelete
    Replies
    1. Following up here, I tried doing just that for an article upcoming. Test is a module and not a class, so it can't be directly subclassed. While you can get access to the list of methods it exports, re-exporting those to the caller seems to require CompUnit::Util which seems rather experimental. You're of course more than welcome to try, and I'd love to hear if you made any progress.

      Delete