Doctest.js: unit testing for Javascript made simple

Doctest/JS: unit testing for Javascript made simple

Author: Ian Bicking <ianb@colorstudy.com>

License & Download

This library is licensed under the GPL. You can download it from the git repository at http://github.com/ianb/doctestjs/ with git clone http://github.com/ianb/doctestjs.git or download a zip

Bugs may be reported on the issue tracker. Patches are best provided by forking the repository through github.com, and then submitting a pull request. If you are using doctest.js, or have written about it, consider noting this on the wiki.

Introduction

Doctest/JS is a port of a widely used testing module doctest from the Python world. The original doctest is by Tim Peters.

Doctest was originally written to test documentation, but its also an embodiment of a more general pattern of example-oriented testing. Tests are made up of code and the code's output, almost as though each statement is an implicit assertEqual. In fact there isn't really a need for assert* helpers because it is implicit in all your tests.

An example is in order. These examples are all self-complete, but typically you would include the .js file you were testing, and wouldn't define (many) functions inline in the code. But don't be shy! It will frequently make your tests more compact and thorough to define helper functions.

The runner is embedded directly in the page. Here's the runner for all the tests on this page (you'll notice some failures, which are deliberate):

Individual tests also have their own runner (you'll notice the boxes with the buttons), so you can debug just one test.

Here's an example, where we will define and test a factorial function. Note I've written the test to get a failure:

$ function factorial(n) {
>     if (typeof n != 'number') {
>         throw('Not a number: '+n);
>     }
>     if (n == 0) {
>         return 1;
>     } else {
>         return n * factorial(n-1);
>     }
> }
$ factorial(3)
6
$ factorial(4)
20
$ factorial('foo')
bar

Also if you print out a really long object you'll get a multi-line representation:

$ obj = [{key1: "one string", key2: "another string",
>         a_key: "something else", something: "more strings",
>         and_then: "one last string"}, {}];
$ writeln(obj);
$ // To really force the pretty printing:
> writeln(repr({key: [1, 2, 3]}, '', 1));

There's not much in the HTML that you don't see right there. If you did view source you'd see something like this:

<div class="test">Description...
<pre class="doctest">
$ code
> ... continuation line
expected output
</pre>
</div>

Note that you don't have to quote > as > -- it's a little-known HTML fact that only < really needs to be quoted.

Try hitting the button and see what happens. You might notice it reloads the page. This is because everytime you run a test you've probably changed code to fix that test (or changed code to stop it from failing), and the reload gets you that fresh new code.

The format is like an interactive interpreter. There isn't any one interpreter for Javascript, so the prompts have been modelled on shell prompts, $ starts a statement and > continues the statement. Statements can be as long as you would like, and can even have multiple parts (separated with ;), they are simply chunks that are run all at once.

The output is the return value of the statement. In our example factorial(3) returns 6. Output can also include anything that is explicitly written (using writeln(value)), and if there's an error then the error is also written out (as you can see with factorial('foo'), which is an error). writeln will write the repr of objects, except strings which are written literally. (repr('"hey"') == "\"hey\"")

Output: every Javascript expression or statement has a return value, and some of those values are quite tedious (like a function definition). So if you don't include any expected output for an example, then the return value is simply ignored.

Errors: errors are worth testing for, so an error isn't necessarily a test failure. An error just results in the text Error: <error message> being printed out, which you can match for if you like.

repr: the repr of the result of a function is printed out. This is an idea borrowed from Python: each object has a programmer's representation. This is a helpful representation of the object, beyond simply what obj.toString() might return. Arrays are displayed like [obj1, obj2, ...] for instance. Generic objects are displayed as

{attr:
value}

. Objects can customize their output by defining a repr() method.

... (ellipsis): sometimes there are portions of the output that are interesting, and portions that are boring. Or parts might be volatile -- different on every test run. You can ignore a portion of the test by using ... in the output. All the matching is strictly textual. Note even errors can be matched this way (maybe unintentionally).

?: (question mark) this is like ..., but it only matches one word (letters, numbers, _, and .). This is so you can do something like {attr: ?} and avoid matching {attr: value, attr2: value}. It doesn't match quotation marks, so you may need "?" (more usefully though, it does match numbers, like a timestamp).

Writing: there are two functions to write to the test output, write() and writeln() (like write() but adds a newline). You can use these inside callbacks or loops to show bits of progress. (console.log may also still work, but isn't matched against.)

Comments: you can include comments anywhere, if you just want a comment in your test, do:

$ // look ma, nothing's executed!

Exceptions and Logging

It's encouraged that you test exception cases in your code. You do this like:

$ function countTag(parent, tag) {
>     return parent.getElementsByTagName(tag).length;
> }
...
$ countTag(document.getElementById('container'), 'pre');
10
$ countTag({}, 'pre');
Error: parent.getElementsByTagName is not a function

Unfortunately this hides the traceback. Doctest will try to print the traceback to the logs; it's not as good a traceback, but it might be helpful anyway. If you install Firebug or use Chrome's developer console, the traceback will be in the console.

You can also use console.log() liberally. In addition to showing up in the console, this activity is captured on a per-statement basis, and displayed next to failing tests (generally in purple). This makes it easier to focus just on the log messages that tell you about what went wrong, ignoring all the messages about what went right.

Test Page Structure

While there are ways to test specific things using doctest, writing the HTML to hook up the runner gets tedious quickly. The easiest way to write a test is to look at autotemplate.html and copy it for your use.

Each test should be in an element

<div
class="test">

. You can give the tests ids, or doctest will just number them for you if not. You can include a description of the test to introduce and explain the test, then use

<pre
class="doctest">

to actually write your test in.

The rest of the page is yours to do with as you want. Specifically you can include other HTML, manipulate it, see the result, etc. Testing DOM operations is simple enough as a result.

You'll notice you can test just a single section delimited by <div class="test">. This is nice for honing in on a particular test, but sometimes you'll have code you want to run for every test (because it sets up helper functions, mocks something out, etc). If that is the case, use <pre class="doctest setup"> and it will always be run.

Asynchronous Calls and delays

Not everything can be tested with call-result, specifically things that require callbacks and asynchronous activity. Some DOM updates require asynchronous activity, as one example -- even if it's just a moment that you have to release control from Javascript, you still must release control from Javascript before the DOM will full reflect updates. XMLHttpRequests are another obvious example.

To let the test wait a while during an example, call the wait() function. This function can be called with a millisecond timeout value, like wait(1000), and wait() alone means a 0-second wait (which releases control momentarily). You can also wait for a condition, like wait(function () {return req.state == 1;});.

An example (with some DOM stuff to update):

Output will go here

$ function updateDom() {
>   var el = document.getElementById('some-output');
>   el.innerHTML = 'Test Output';
> }
...
$ updateDom();
$ wait();
$ document.getElementById('some-output').innerHTML;
"Test Output"

And another example of a callback:

$ finished = false;
$ function doSomethingSlowly(callback) {
>   setTimeout(function () {finished = true; callback();}, 1);
> }
$ doSomethingSlowly(function () {
>   writeln('Something was done');
> });
> wait(function () {return finished});
Something was done

When using wait(callback) there will still be a timeout. All timeouts are in milliseconds; the default timeout is 2000 (2 seconds). If that much time passes the wait will be considered a failure. You can pass a second argument with a timeout, or set doctest.defaultTimeout to change the value of that timeout.

The Spy object makes this particularly easy. You can generally test callback-oriented code very nicely like:

$ function doRequest(callback) {
>   // imagine we use XMLHttpRequest
>   setTimeout(function () {callback('data');}, 1);
> }
$ doRequest(Spy('doRequest.response', {wait: true}));
doRequest.response('data')

A Spy is kind of a mock object that tracks when it is called (and any time it is called it writes out the call info). You can also save it in a variable, inspect the arguments, etc, as described in the section later on.

Halting Further Tests

The tests are run in order, even if one fails. In some cases this is tedious, such as a test that requires the server to be present -- everything will of course fail, but nothing really means anything.

If you use throw Abort('reason') (or simply call Abort('reason')) then all further tests will be skipped. This can also be thrown in callbacks. Don't even instantiate this class if you don't want to stop the tests; simply instantiating it is enough (because it's hard to actually catch exceptions in callbacks).

If you only want to halt one section of tests (one <pre> element) you can use AbortSection(). This can be useful for skipping tests in some environments.

Spy/Mock

Also included is a simple mock object called Spy. This object can be called, and you can check if it is called and how.

You use it like:

$ function funcWithCallback(callback) {
>   setTimeout(function () {callback('foo')}, 100);
> }
$ mySpy = Spy('mySpy');
$ funcWithCallback(mySpy);
$ mySpy.wait();
mySpy("foo")

The args are kept in mySpy.args, the this object in mySpy.self.

Spy takes an options object as the second argument:

writes (bool): If true (which is is by default), then everytime the function is called it will write the function call (like mySpy.formatCall() does explicitly in the example).
returns: This is the value that is returned when the function is called. Defaults to null.
throwError: This is an error that is thrown when the function is called.
applies: This is a function that will be called. Setting this basically makes the function wrap this other function. If the second argument is a function (not an object of options) then it is assumed to be a value for applies.
wait: If true, then after creating it will immediately call this.wait();. You can use this for shortcuts.
ignoreThis: If the function is called with a this value that is "interesting" (not window for instance) then it will be printed. But sometimes that's just distracting, so using ignoreThis: true the value of this won't be written out.
wrapArgs: When printing the function call you might want to force wrapping; use wrapArgs: true to force this.

You may set global defaults for these options with doctest.defaultSpyOptions, for instance to never write calls ({writes: false}).

When using writes (as by default) then when you call spy.wait() the function call will be printed out immediately after the wait.

If you wish to be brief about it, you can do:

$ funcWithCallback(Spy('test')); Spy('test').wait();
$ // or:
$ funcWithCallback(Spy('test', {wait: true}));

Spies also have some attributes and methods of interest:

.name: The name you gave the spy
.called: Whether this spy has been called yet
.args: The list of arguments it was called with (null if not yet called)
.self: The value of this when called (if it was used as a method)
.argList, .selfList: These are like .args and .self, but are appended to for each call, giving a history of all the calls.
.func: This is a function with no attributes. You can use this if you want to pass in a mock object that doesn't have all these extra attributes and methods attached to it.
.formatCall(): This returns a string that represents how this function was last called; it's what gets printed (if you have writes on)
.method(name, options): This adds a method to the object. This is another spy object, assigned to an attribute of the parent spy object and with an appropriate name. So Spy('foo').method('bar') will give you a spy that is named foo.bar, and is available as Spy('foo.bar').
.methods(properties): This takes a bunch of methods, with options for values (or null if you don't care about options)
.wait(optional timeout): This waits for the spy to be called.

Object Diff

A helper is included to make a diff of objects. This can help to view and test state changes.

$ oldObject = {a: 10, b: 12};
$ newObject = {a: 11, c: 3};
$ doctest.writeDiff(oldObject, newObject)
+c: 3
-b: 12
a: 10 -> 11

The output always starts with any added attributes (which show their new value), then any deleted values (which show the old value), then finally any changed attributes. Attributes that aren't changed aren't shown.

There is also a function doctest.objectEqual if you want to test if any attributes have changed.

CoffeeScript

You can use CoffeeScript in your tests instead of Javascript. You should include your files as usual (e.g., using type="text/coffeescript" or precompiling to Javascript). To make the actual tests themselves use CoffeeScript, use this in the head:

<script>
  doctest.useCoffeeScript();
</script>

This replaces doctest.eval with a function that uses CoffeeScript.compile.

Hooking into the reporter

You may want to get access to the test results. The easiest way isn't to replace the reporter, but there is a way to access the results of the reporter via a reporter hook.

If there is an object named doctestReporterHook when the doctests are run, then that object will be used. If you have the tests running automatically on load, then you should be sure to get this object in place immediately (via <script>).

The object can implement several methods, all optional:

.init(reporter): This is called once at the beginning, and lets you bind the hook to the reporter.
.startElement(el): This is called at the beginning of each new <pre> element that is tested. This could be considered a TestCase if you are mapping this to xUnit terminology.
.reportSuccess(example, output): This is called for each line of testing (i.e., each line that starts with $). It gives the example (which is an instance of doctest.Example and the text that was expected (which matches, hence the sucess!)
.reportFailure(example, output): More interesting, this is a failure. The example output (example.output) did not match the actual output (output).
.finish(reporter): All tests have finished running. You can access a summary of the results with reporter.success and reporter.failure

To Do

Detail should be displayed (but not matched for) when there is an exception. Maybe similarly, detail should be shown when output doesn't match (like a line-by-line match to highlight the difference, at least when dealing with multi-line output). Some of this is in the console, but should be inline.
Implement CommonJS loading/module. Not sure how/if this works in a browser.
A clear, documented way to plug in different output comparison tools. Pure string literal comparison is just one not-so-great way. Also clarify how repr() works and can be extended.
The parser doesn't check for many errors, e.g., you can interleave "output" with expressions, depending on whether you use the > prompt.
Maybe instead of > and $, it should use >>> for the prompty, and ==> to indicate that "output" is starting, with no continuation lines. Also maybe a line starting with // should be ignored (as a comment on the tests themselves, instead of needing $ //).
Figure out how this can work without a browser and HTML being involved. Or... with env.js and a custom reporter? Or simply a text-based parser?
Implement a mode that works in a pure-JS file, that then fills in an HTML report. I imagine something like:
```
writeln('hey you');
/* =>
hey you
*/
```
Only writeln (or print?) would be supported. Output would only go in comments.
Experiment with driving an iframe using doctests, for site functional/integration testing.
Handle the unstable nature of anchor links to specific failures: these should either be stable (link everything upfront?) or have some fallback.
Do something so the screen doesn't scroll around annoyingly during tests as text is added to the page. Maybe a fixed-height scrolling reporter?
Document doctest.params. And organize this document better.

doctest.js by Ian Bicking