tests/README

			    The Automake test suite


User interface
==============


Running the tests
-----------------

  To run all tests:

    make -k check

  You can use `-jN' for faster completion (it even helps on a
  uniprocessor system, due to unavoidable sleep delays, as
  noted below).

  To rerun only failed tests:

    make -k recheck

  To run only tests that are newer than their last results:

    make -k check RECHECK_LOGS=

  To run only selected tests:

    make -k check TESTS="foo.test bar.test"             (GNU make)
    env TESTS="foo.test bar.test" make -e -k check      (non-GNU make)

 To run the tests in cross-compilation mode:

    make -k check host_alias="$host_alias"              (GNU make)
    env host_alias="$host_alias" make -e -k check       (non-GNU make)

 Here `$host_alias' should be defined to a proper value different from
 configure-determined `$build_alias', and should refer to a set of
 cross-compilers you have available on your system; for example, if
 on Linux you have a set of MinGW-targeted cross-compilers named
 'i586-mingw32msvc-cc', 'i586-mingw32msvc-c++', etc., you could use:

   make -k check host_alias='i586-mingw32msvc'


Interpretation
--------------

  Successes:
    PASS  - success
    XFAIL - expected failure

  Failures:
    FAIL  - failure
    XPASS - unexpected success

  Other:
    SKIP  - skipped tests (third party tools not available)
    ERROR - some unexpected error condition


Getting details from failures
-----------------------------

  By default, verbose output of a test `foo.test' or `foo.tap' is
  retained in the log file `foo.log'.  A summary log is created in
  the file `test-suite.log'.

  You can limit the set of files using the TESTS variable, and enable
  detailed test output at the end of the test run with the VERBOSE
  variable:

    env VERBOSE=x TESTS='first.test second.test ...' make -e check

  You can also run the tests by hand, as explained in the next subsection.


About the tests
---------------

  There are two kinds of tests in the Automake testsuite (both implemented
  as shell scripts).  The scripts with the `.test' suffix are "simple"
  tests, their outcome completely determined by their exit status.  Those
  with the `.tap' suffix use the TAP protocol.  If you want to run a test
  by hand, you can do so directly if it is a simple test:

    ./nogzip.test

  (it will be verbose by default), while if it is a TAP test you can pass
  it to your preferred TAP runner, as in e.g.:

    prove --verbose --merge ./add-missing.tap

  The tests can also be run directly in a VPATH build, as with:

    /path/to/srcdir/tests/nogzip.test
    prove --verbose --merge /path/to/srcdir/tests/add-missing.tap


Supported shells
----------------

  By default, the tests are run by the $SHELL detected at configure
  time.  They also take care to re-execute themselves with that shell,
  unless told not to.  So, to run the tests with a different shell, say
  `/path/to/another/sh', the user must use:

    AM_TESTS_REEXEC=no /path/to/another/sh ./foo.test
    AM_TESTS_REEXEC=no prove -v -e /path/to/another/sh ./bar.tap

  to run a test directly, and:

    make check LOG_COMPILER=/path/to/sh         (GNU make)
    LOG_COMPILER=/path/to/sh make -e check      (non-GNU make)

  to run the test(s) through the makefile test driver.

  The test scripts are written with portability in mind, so that they
  should run with any decent Bourne-compatible shell.

  However, some care must be used with Zsh, since, when not directly
  started in Bourne-compatibility mode, it has some incompatibilities
  in the handling of `$0' which conflict with our usage.  Our testsuite
  can automatically work around these incompatibilities when a version
  4.3 or later of Zsh is used, but unfortunately not when an older
  version of Zsh is used.  Thus, if you want to run a test script, say
  foo.test, with Zsh 4.2, you *can't* simply do `zsh foo.test', but
  you *must* resort to:
    AM_TESTS_REEXEC=no zsh -o no_function_argzero foo.test

  Note that this problem does not occur if Zsh is executed through
  a symlink with a basename of `sh', since in that case Zsh starts
  in Bourne compatibility mode.  So you should be perfectly safe
  when /bin/sh is Zsh, even a it's version < 4.3.


Reporting failures
------------------

  Send verbose output, i.e., the contents of test-suite.log, of failing
  tests to <bug-automake@gnu.org>, along with the usual version numbers
  (which Automake, which Autoconf, which operating system, which make
  version, which shell, etc.)



Writing test cases
==================


Do
--

  If you plan to fix a bug, write the test case first.  This way you'll
  make sure the test catches the bug, and that it succeeds once you have
  fixed the bug.

  Add a copyright/license paragraph.

  Explain what the test does.

  Cite the PR number (if any), and the original reporter (if any), so
  we can find or ask for information if needed.

  If a test checks examples or idioms given in the documentation, make
  sure the documentation reference them appropriately in comments, as in:
    @c Keep in sync with autodist-config-headers.test.
    @example
    ...
    @end example

  Use `required=...' for required tools.  Do not explicitly require
  tools which can be taken for granted because they're listed in the
  GNU Coding Standards (for example, `gzip').

  Include ./defs in every test script (see existing tests for examples
  of how to do this).

  Use the `skip_' function to skip tests, with a meaningful message if
  possible.  Where convenient, use the `warn_' function to print generic
  warnings, the `fail_' function for test failures, and the `fatal_'
  function for hard errors.  In case a hard error is due to a failed
  set-up of a test scenario, you can use the `framework_fail_' function
  instead.

  For tests that use the `parallel-tests' Automake option, set the shell
  variable `am_parallel_tests' to "yes" before including ./defs.  For tests
  that are *not* meant to work with the `parallel-tests' Automake option
  (these should be very very few), set the shell variable `am_parallel_tests'
  to "no" before including ./defs.

  Some tests in the Automake testsuite are auto-generated; those tests
  might have custom extensions, but their basename (that is, with such
  extension stripped) is expected to end with "-w" string, optionally
  followed by decimal digits.  For example, the name of a valid
  auto-generated test can be `color-w.test' or `tap-signal-w09.tap'.
  Please don't name hand-written tests in a way that could cause them
  to be confused with auto-generated tests; for example, `u-v-w.test'
  or `option-w0.tap' are *not* valid name for hand-written tests.

  ./defs brings in some commonly required files, and sets a skeleton
  configure.in.  If possible, append to this file.  In some cases
  you'll have to overwrite it, but this should be the exception.  Note
  that configure.in registers Makefile.in but do not output anything by
  default.  If you need ./configure to create Makefile, append AC_OUTPUT
  to configure.in.  In case you don't want ./defs to pre-populate your
  test directory (which is a rare occurrence), set `$am_create_destdir'
  to "empty" before sourcing ./defs.

  By default, the testcases are run with the `errexit' shell flag on,
  to make it easier to catch failures you might not have thought of.
  If  this is undesirable in some testcase, you can use `set +e' to
  disable the `errexit' flag (but please do so only if you have a
  very good reason).

  End the test script with a `:' or `Exit 0'.  Otherwise, when somebody
  changes the test by adding a failing command after the last command,
  the test will spuriously fail because $? is nonzero at the end.  Note
  that this is relevant even if the `errexit' shell flag is on, in case
  the test contains commands like "grep ... Makefile.in && Exit 1" (and
  there are indeed a lot of such tests).

  Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
  $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
  commands.

  Use $sleep when you have to make sure that some file is newer
  than another.

  Use `cat' or `grep' to display (part of) files that may be interesting
  for debugging, so that when a user send a verbose output we don't have
  to ask him for more details.  Display stderr output on the stderr file
  descriptor.  If some redirected command is likely to fail, display its
  output even in the failure case, before exiting.

  Use `Exit' rather than `exit' to abort a test.

  Use `$PATH_SEPARATOR', not hard-coded `:', as the separator of
  PATH's entries.

  It's more important to make sure that a feature works, than
  make sure that Automake's output looks correct.  It might look
  correct and still fail to work.  In other words, prefer
  running `make' over grepping `Makefile.in' (or do both).

  If you run $AUTOMAKE or $AUTOCONF several times in the same test
  and change `configure.in' by the meantime, do
    rm -rf autom4te.cache
  before the following runs.  On fast machines the new `configure.in'
  could otherwise have the same timestamp as the old `autom4te.cache'.
  Alternatively, use `--force' for subsequent runs of the tools.

  Use filenames with two consecutive spaces when testing that some
  code preserves filenames with spaces.  This will catch errors like
  `echo $filename | ...`.

  Make sure your test script can be used to faithfully check an
  installed version of automake (as with "make installcheck").  For
  example, if you need to copy or grep an automake-provided script,
  do not assume that they can be found in the `$top_srcdir/lib'
  directory, but use `$am_scriptdir' instead.  The complete list of
  such `$am_..dir' variables can be found in tests/defs-static.in.

  Before commit: make sure the test is executable, add the tests to
  TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
  write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.


Do not
------

  Do not test an Automake error with `$AUTOMAKE && Exit 1', or in three
  years we'll discover that this test failed for some other bogus reason.
  This happened many times.  Better use something like
     AUTOMAKE_fails
     grep 'expected diagnostic' stderr
  (Note this doesn't prevent the test from failing for another
  reason, but at least it makes sure the original error is still
  here.)

  Do not override Makefile variables using make arguments, as in e.g.:
    $MAKE prefix=/opt install
  This is not portable for recursive targets (targets that call a
  sub-make may not pass `prefix=/opt' along).  Use the following
  instead:
    prefix=/opt $MAKE -e install