tta/tests/README

tta/tests/README

  Copyright 2010-2026 Free Software Foundation, Inc.

  Copying and distribution of this README file, with or without
  modification, are permitted in any medium without royalty provided the
  copyright notice and this notice are preserved.

Files anywhere within the texinfo/tta/tests/ subdirectory which have no
other copyright notice are hereby placed in the public domain.

This directory contains tests of the texi2any program.

Prerequisites
=============
To run this testsuite, sed, awk, diff (with -u) are used. 
For LaTeX2HTML tests (not run by default), mktemp 
understanding the -d option is also required.

Some tests may also require a case insensitive filesystem.


Running the testsuite
=====================
This testsuite can be run one of the following ways:

* Using the makefile rules: "make -k check" (in this directory)
These tests use the scripts under the test_scripts directory.

* Using ./run_parser_all.sh with the directories one want to run the 
tests in given as arguments, for example "./run_parser_all.sh formatting".
To run just one test, give the test name, e.g.
"./run_parser_all.sh formatting cond".

In the default case, ./run_parser_all.sh tests the Perl command, to test
the native command, call with -native option:

"./run_parser_all.sh -native formatting cond".

The native command tests are enabled in the Makefile only if the native
command is built.

Checking test results
---------------------
The test results are in the out_perl and out_native directories; the
reference results are in res_parser directories.  There can be these pairs.
- out_perl/ and res_parser/ for the perl command test;
- out_native/ and res_parser/ for the native command test;
and diffs are put in the diffs/ subdir.

For example, to investigate a failure in the 
test_scripts/layout_formatting_docbook.sh test, compare the contents of the
files in the layout/res_parser/formatting_docbook and
layout/out_perl/formatting_docbook directories, e.g.

diff {res_parser,out_perl}/formatting_docbook/formatting.2

Also look in the layout/diffs directory.

If the test results are as expected, copy the output files onto the 
reference files.  Otherwise, if they need more investigation, see "Tests 
specification" below, to find out what input file, etc. was used for the 
test.  Sometimes you need to use "texi2any -F" to force an output file 
to be created.

When running tests via parser_tests.sh or run_parser_all.sh, then
a test failure causes
 F: a_test_directory
to be printed.  If there is a diff with the references, then
 D: a_test_directory
is printed. If there are no reference to compare with, then
 no res: a_test_directory
is printed.

If a whole directory check run using './parser_tests.sh' had no failure, 
the directory name is printed followed by a ok; it is followed
by a fail if there was a failure.

The test results are in a directory below the out directories. This 
directory name can be considered as the test name. Let's call it a_test
for the remainder of the explanation. The stderr output is in 
out_perl/a_test/a_test.2, stdout output is in
out_perl/a_test/a_test.1.  All the commands called are output in
tests.log.

Note that "make all-checks" also runs the testsuite, but also runs
TeX4ht and LaTeX2HTML tests and other tests that are not as portable
as the main testsuite and may require additional softwares.  See
below for more.


Regenerating test results
=========================
To update the test results, first the tests must be run as above.
A quick way to update all the tests is to run "make -k check copy-tests".

Giving -copy as the first argument to ../run_parser_all.sh or
parser_tests.sh will cause the reference results to be copied
from the results the last time the tests were run.  For example,
to update the reference results for one particular test:

$ ./parser_tests.sh -copy formatting
The copy-* make targets update all the references for various large sets
of tests via this method.

Exception: in the many_input_files subdirectory, the *parser*.sh scripts
can't be used for updating.  Instead, the Makefile's copy-tests and
copy-tex-html targets do it directly.

The Makefile copy targets use the Perl command results since they should
always be generated.


Testsuite and document strings translations
===========================================
At least one test, layout/formatting_fr_icons, depends heavily on
the translation of strings that can be translated and incorporated in
output documents.  The translations are in the po_document directory.
These translations are managed by the translation project and only
updated for the release.  They may lag behind the changes in code,
which may be problematic for test results.  The translations in
po_document should never be changed, as they will be overwritten by
the translation project.  If the translations need to be modified
for the test results, a filter script should be used instead.  An
example of such a script can be seen in po_document/fix-translations.sh.


Test requiring recoded file names
=================================

Some test require the possibility to recode file names, in general from
UTF-8 to a non UTF-8 encoding whan the test require a file name in a
non UTF-8 encoding.  The recoding of file names required by these
tests are done by make check, and it is possible to do it manually
with make input_file_names_recoded_stamp.txt.  If the recoding fails,
for instance if the recoding leads to invalid UTF-8 and the file system does
not accept invalid UTF-8, input_file_names_recoded_stamp.txt will contain
FAILED, and the test requiring the files with recoded file names will
be skipped.  The recoded files are generated in the built_input/ directory.

Tests requiring other non-ASCII file names
==========================================

There are rules in Makefile.am to generate input/non_ascii.tar which
contains test input files with non-ASCII characters in their names.
This allows us to create a distribution tarball without such files.
If these input files change, you may need to force regeneration of
non_ascii.tar by deleting the file.

TeX4ht and LaTeX2HTML tests
===========================
The tex4ht and latex2HTML related tests are not run automatically, as
some test results involving latex2HTML or tex4ht depend on the setup and 
version of these tools.

These tests may be run by
  make tex-html-checks
or, together with all the other tests by
  make all-checks

For the tests that use latex2html, to avoid the test failing if there is 
a dot in the cwd, mktemp is used to create a temporary directory, and 
the directory is passed through on the command line.


Other tests requiring specific softwares
========================================
Other tests requiring specific software are not run automatically, as
they may depend on the version of these specific software and require
these software to be present.  The required software are source-highlight
and latex with the preview package.  dvipng is also required for some
tests, if not found the tests will be skipped.

These tests may be run by
  make other-checks
or, together with all the other tests by
  make all-checks


Tests specification
===================
Test runs are driven by the contents of the list-of-tests file in
each subdirectory. This is a line-oriented file. A # starts a
comment.

Each non-empty non-comment line describes a test.  The first
word on the line is the test name which corresponds also with the
resulting directory (what we called a_test above).  It is followed by
the source manual name. The source manual name has to have the .texi
extension.  Optionally, additional arguments can be given on the rest of
the line.

So, for example, the line

a_test manual.texi

specifies that for the test a_test, the file manual.texi is processed
and results are put are put in the a_test directory.  The line:

a_test_split_chapter manual.texi --split chapter

specifies that the results of the processing of manual.texi with additional
command line arguments --split chapter will be in the a_test_split_chapter
directory.

"Need recoded file names" should appear on a line for a test requiring
recoded file names. for example

manual_include_accented_file_name_latin1 manual_include_accented_file_name_latin1.texi -D 'needrecodedfilenames Need recoded file names'

Common .texi
------------
A common .texi file should be in the top-level directory (there is such an
example with coverage_macro.texi).

Init files
----------
Init files are searched for in ../perl/t/init/, ../perl/init and ../perl/ext.


Creating a new test
===================
- choose the subdir here for the new test.
- add the line to subdir/list-of-tests as described above.
- most probably, create the needed subdir/*.texi file
- ./run_parser_all.sh subdir newtest

This will create subdir/out_perl/newtest/ with the test results
(as explained above), as well as stdout in newtest.1 and stderr in newtest.2.
Look carefully to be sure they are as they should be.

When things look good, use the -copy option to create the
subdir/res_parser/newtest/ subdirs and copy the output files there.

The test run will be logged in subdir/test_log/perl_newtest.log,
including the exact invocation of the tested command, which can alter
anything.  For even more, use sh -vx to show exactly what run_parser_all
 is doing.

When satisfied, add the .texi in subdir/Makefile.am.

Of course commit all the files and try a make check afterward to verify
it gets run, and passes.

See tta/perl/t/README for info about those tests.


Deleting a test
===============

Run "make" to recreate the test scripts (which runs
../maintain/regenerate_cmd_tests.sh).

git status test_scripts/

Then "git rm" the missing files.

Delete directories containing reference test results, e.g.

ls -d formatting/*/unknown_nodes_renamed
rm -r formatting/out_*/unknown_nodes_renamed
git rm formatting/res_parser/unknown_nodes_renamed