automake.info: Tests

Go forward to Options
Go backward to Dist
Go up to Top
Go to the top op automake

Support for test suites

   Automake supports two forms of test suites.

Simple Tests

   If the variable `TESTS' is defined, its value is taken to be a list
of programs to run in order to do the testing.  The programs can either
be derived objects or source objects; the generated rule will look both
in `srcdir' and `.'.  Programs needing data files should look for them
in `srcdir' (which is both an environment variable and a make variable)
so they work when building in a separate directory (*note Build
Directories: (autoconf)Build Directories.), and in particular for the
`distcheck' target (*note Dist::).
   The number of failures will be printed at the end of the run.  If a
given test program exits with a status of 77, then its result is ignored
in the final count.  This feature allows non-portable tests to be
ignored in environments where they don't make sense.
   The variable `TESTS_ENVIRONMENT' can be used to set environment
variables for the test run; the environment variable `srcdir' is set in
the rule.  If all your test programs are scripts, you can also set
`TESTS_ENVIRONMENT' to an invocation of the shell (e.g.  `$(SHELL)
-x'); this can be useful for debugging the tests.
   You may define the variable `XFAIL_TESTS' to a list of tests
(usually a subset of `TESTS') that are expected to fail.  This will
reverse the result of those tests.
   Automake ensures that each program listed in `TESTS' is built before
any tests are run; you can list both source and derived programs in
`TESTS'.  For instance, you might want to run a C program as a test.
To do this you would list its name in `TESTS' and also in
`check_PROGRAMS', and then specify it as you would any other program.

DejaGNU Tests

   If `dejagnu' (ftp://ftp.gnu.org/gnu/dejagnu/) appears in
`AUTOMAKE_OPTIONS', then a `dejagnu'-based test suite is assumed.  The
variable `DEJATOOL' is a list of names which are passed, one at a time,
as the `--tool' argument to `runtest' invocations; it defaults to the
name of the package.
   The variable `RUNTESTDEFAULTFLAGS' holds the `--tool' and `--srcdir'
flags that are passed to dejagnu by default; this can be overridden if
necessary.
   The variables `EXPECT' and `RUNTEST' can also be overridden to
provide project-specific values.  For instance, you will need to do
this if you are testing a compiler toolchain, because the default
values do not take into account host and target names.
   The contents of the variable `RUNTESTFLAGS' are passed to the
`runtest' invocation.  This is considered a "user variable" (*note User
Variables::).  If you need to set `runtest' flags in `Makefile.am', you
can use `AM_RUNTESTFLAGS' instead.
   Automake will generate rules to create a local `site.exp' file,
defining various variables detected by `./configure'.  This file is
automatically read by DejaGnu.  It is ok for the user of a package to
edit this file in order to tune the test suite.  However this is not
the place where the test suite author should define new variables: this
should be done elsewhere in the real test suite code.  Especially,
`site.exp' should not be distributed.
   In either case, the testing is done via `make check'.

Install Tests

   The `installcheck' target is available to the user as a way to run
any tests after the package has been installed.  You can add tests to
this by writing an `installcheck-local' target.