[ Jocelyn Ireson-Paine's home page | Free software | Publications ]

Autotest for SWI-Prolog

Autotest is a program for automatic testing of Prolog goals against their expected outcomes.

Calling Autotest

The software is here as a zip file. There is one source file, autotest.pl. The distribution also comes with this page, doc/autotest.html, and a set of sample tests, examples/autotest.tst - reproduced in the examples section of this page. To get started immediately, start SWI and run Autotest as in the session below:

Welcome to SWI-Prolog (Multi-threaded, Version 5.4.2)
...More SWI banner...
For help, use ?- help(Topic). or ?- apropos(Word).

1 ?- cd('d:/kb7/autotest').

Yes
2 ?- [autotest].
% autotest compiled 0.01 sec, 12,612 bytes

Yes
3 ?- '$run_test'( 'examples/autotest.tst' ).
Line 1 of autotest.tst
Goal 5=5 should have failed but succeeded
Goal 6=6 should have failed but succeeded
Goal 7\=7 should have succeeded but failed
...More Autotest output...
Final line of autotest.tst

Yes
4 ?- 
The sections following explain the format of input files and how to specify tests.

Autotest defines two top-level predicates: '$run'(F) and '$run_test'(F).

Steering files

Steering files contain zero or more Prolog terms. Each term can be:

If the term is 'log_to(_)', it resets the log files - see later. If it's log_all_tests, it alters whether successful tests are recorded. Otherwise, the atom specifies a filename, that of a so-called 'test file'. If any filename is 'stay', then the rest of the steering file is assumed to be the one and only test file.

Test files

Test files contain zero or more Prolog terms. Each term can be:

:: is an xfx, 1200 operator declared by Autotest.

If the term is 'log_to(_)' or 'log_all_tests(_)', it behaves like the same term in a steering file.

If the term is not of the form Goal :: Spec, then Autotest will write it to the log file(s). It does this by passing it to its '$writef' predicate. To simplify constructing messages, this interprets the following terms specially:

<> and ... are xfy, 600 operators.

Specifying tests

If the term is a structure, Goal::Spec, then Spec must be one of

The first four specify whether the Goal should succeed, or fail, or crash with some error. If the term is anything else, Autotest interprets it as another goal, which must succeed. Usually, it will test variables bound by Goal.

Examples are:

When testing a Goal::Spec term, Autotest calls the Goal (and then cuts it, to prevent backtracking into it). It notes whether the Goal succeeded, failed, or caused an error. If it caused an error, Autotest tries to determine the error term. Autotest then compares the Goal's effect with its Spec. If they differ, it writes a message stating what did happen, and what should have happened. In doing this:

Normally, Autotest will not log successful tests. You can make it do so by including the term 'log_all_tests(on)' in the steering or test file. This will cause all tests, whether or not they succeeded, to be logged. A subsequent 'log_all_tests(off)' will revert Autotest to logging only failed tests.

Trapping errors

The way errors are detected is implementation-dependent. This version of Autotest uses 'catch' to detect them, as expected in SWI-Prolog. To modify this, you need to alter the predicate '$call_giving_sfe'. This should return a structure 'e(ErrorTerm)' in its second argument; or the atom 'e' if you can't find out the error term.

Logging

When Autotest writes messages, it does so via the predicate '$writef_to_log', which sends to the current log files. Their initial value is specified by the assertion '$log_files' in the Autotest source, and can be changed as below.

'$writef_to_log' writes the same message to each file in turn. The files are closed when Autotest finishes.

If you give a 'log_to(FL)' in the steering file, FL must be one atom, or a list of atoms, specifying the names of the new log files. Autotest closes any log files which are not named in FL, and makes those in FL the new files. They are not opened until first written to.

Predicate names and modules

In order to avoid interfering with predicates under test, all predicates in Autotest have names starting with $ (dollar). There are three infix operators declared: ::, <>, and ... . Autotest does not use modules.

Open files

If an error occurs while Autotest is reading a file, then that file may not be closed properly. This means that the next call of '$run' or '$run_test' may start part of the way down the file.

Examples

Here is a sample steering file.

/*  autotest.steer: Steering file for Autotest.
*/

'autotest.tst'.
That file would cause the contents of autotest.tst to be tested.

Here is autotest.tst. It's available in the examples subdirectory.

/*  autotest.tst  */

/*  Sample input to autotest.pl.

    This is the SWI version of an example input file for
    Autotest. The expected output is shown below. To fully
    demonstrate the program, the tests are wrong, in
    that they specify incorrect outcomes for = and other 
    predicates.

Line 1 of autotest.tst
Goal 5 = 5 should have failed but suceeded
Goal 6 = 6 should have failed but suceeded
Goal 7 \= 7 should have suceeded but failed
Goal 8 \= 8 should have suceeded but failed
Goal 9 = 9 should have crashed but suceeded
Goal 10 = 10 should have crashed but suceeded
Goal 11 \= 11 should have crashed but failed
Goal 12 \= 12 should have crashed but failed
Goal 13 = 13 should have crashed with error 137 but suceeded
Goal 14 = 14 should have crashed with error 137 but suceeded
Goal 15 \= 15 should have crashed with error 137 but failed
Goal 16 \= 16 should have crashed with error 137 but failed
Goal 1 is 1 should have crashed with error error(type_error(evaluable, a/0), _) but succeeded
Goal 1 is 1 should have crashed with error error(fnoo_error(evaluable, a/0), _) but succeeded
Goal _ is a should have crashed with error error(type_error(evaluable, b/0), _) but crashed with error error(type_error(evaluable, a/0), _)
Goal 1 is 1 should have crashed with error error(fnoo_error(evaluable, a/0), _) but succeeded
Goal 1 is 1 should have crashed but succeeded
Goal 1=2 should have crashed but failed
Goal functor(f(1, 2), f, 2) should have passed tests f = g , 2 = 3 but failed
Final line of autotest.tst

*/

'Line 1 of' ...
'autotest.tst'<>nl.             /*  Display  */

1 = 1 :: s.                     /*  OK  */

2 = 2 :: succeeds.              /*  OK  */

3 \= 3 :: f.                    /*  OK  */

4 \= 4 :: fails.                /*  OK  */

5 = 5 :: f.                     /*  Message  */

6 = 6 :: fails.                 /*  Message  */

7 \= 7 :: s.                    /*  Message  */

8 \= 8 :: succeeds.             /*  Message  */

9 = 9 :: c.                     /*  Message  */

10 = 10 :: crashes.             /*  Message  */

11 \= 11 :: c.                  /*  Message  */

12 \= 12 :: crashes.            /*  Message  */

13 = 13 :: c(137).              /*  Message  */

14 = 14 :: crashes(137).        /*  Message  */

15 \= 15 :: c(137).             /*  Message  */

16 \= 16 :: crashes(137).       /*  Message  */

17 = X :: X = 17.               /*  OK  */

X is 1 :: crashes( error(type_error(evaluable, a/0), _ ) ).
                                /*  Message  */

X is 1 :: crashes( error(fnoo_error(evaluable, a/0), _ ) ).
                                /*  Message  */

X is a :: crashes( error(type_error(evaluable, a/0), _ ) ).
                                /*  OK  */

X is a :: crashes( error(type_error(evaluable, b/0), _ ) ).
                                /*  Message  */

X is 1 :: crashes( error(fnoo_error(evaluable, a/0), _ ) ).
                                /*  Message  */

X is 1 :: crashes.              /*  Message  */

1 = 2  :: crashes.              /*  Message  */

functor( f(1,2), F, A ) ::
    F = f, A = 2.               /*  OK  */

functor( f(1,2), F, A ) ::
    F = g, A = 3.               /*  Message  */

'Final line of'<>
' autotest.tst'<>nl.            /*  Display  */

Here is another steering file:

/*  demo.steer  */

'autotest.tst'.             % This is the first test file. We log to the default files.
log_to( 'fred' ).           % Log to fred.                        
'test2.pl'.                 % test2.pl is the next test file.      
log_to( [bert,joe,me] ).    % Log to bert., joe., me.              
'test3.tst'.                % test3.tst is the next test file.  
log_to( user ).             % Now log just to the terminal.        
stay.                       % Future test input comes from here.   
fail::s.
log_all_tests(on).          % Report future tests, even if OK.     
true::f.
'Final line of steering file.'<>nl.

Note on testing for correct instantiation

Suppose we want to test the predicate 'append'. The obvious way to do this is:

    append( [1,2,3,4], [5,6,7,8], [1,2,3,4,5,6,7,8] ) :: s.
Now, suppose that our 'append' works correctly if the third argument is instantiated at call, so
    ?- append( [1,2,3,4], [5,6,7,8], [1,2,3,4,5,6,7,8] ).
succeeds. Suppose further that it doesn't quite work correctly if the third argument is uninstantiated. Instead, it gives a wrong solution; but if you backtrack into it, it then gives the correct solution:
    ?- append( [1,2,3,4], [5,6,7,8], X ).
    X = []
    More ? (yes/no)
    yes
    X = [1,2,3,4,5,6,7,8]
(This is actually unlikely to happen with the way that most people write 'append', but it illustrates what can happen when testing other predicates).

If you autotest 'append' with the third argument instantiated as above, this bug will not be provoked. If you test it like this:

    append( [1,2,3,4], [5,6,7,8], X ), X = [1,2,3,4,5,6,7,8] :: s.
the bug will be provoked. However, although the first value for X will be wrong, the goal X=[1,2,3,4,5,6,7,8] will force backtracking into append; append will then provide a correct value for X; the whole test will succeed; so the bug won't manifest.

Get round this by using this test:

    append( [1,2,3,4], [5,6,7,8], X ) :: X = [1,2,3,4,5,6,7,8].
As soon as 'append' returns with its first (wrong) value for X, AutoTest will cut it, so 'append' can never return the second (correct) value for X. The post-test X=[1,2,3,4,5,6,7,8] will fail, and Autotest will report this, hence making the bug obvious.

[ Jocelyn Ireson-Paine's home page | Free software | Publications ]