tta/TODO

This is the todo list for texi2any

  Copyright 2012-2026 Free Software Foundation.

  Copying and distribution of this file, with or without modification,
  are permitted in any medium without royalty provided the copyright
  notice and this notice are preserved.


Before next release
===================

Revisit (dir) by language or not.  Check the remaining of the thread?
 https://lists.gnu.org/archive/html/bug-texinfo/2026-08/msg00029.html
Also check "Texinfo manuals translations" below to make sure that
everything is decided.  Add to Texinfo manual?


Revisit tests and CRLF?
https://lists.gnu.org/archive/html/bug-texinfo/2026-10/msg00022.html


Add shadow target/automatic nodes for sections without nodes to NEWS


Document LATEX_TAGGED_PDF + NEWS, change variable name if needed in code
and tests.
To document LaTeX customization variable to get accessible PDF.  This adds
\DocumentMetadata{tagging=on,
 <value of the customization variable>
}
with default for the value being
 lang=<BCP47 document-language-from-texinfo>,
 pdfstandard=ua-2,

and replace
\usepackage{amsmath}
\usepackage{amsfonts}
by
\usepackage{unicode-math}


Change "Node Line Requirements" on "node name may not start with a left
parenthesis" to state what is quoted?  Wait a bit to know what Emacs Info
reader support.


Bugs
====

To be investigated/fixed as C without libperl/Perl is more important now, as
proposed by Gavin.

Without XS, on Cygwin32 linking leads to an empty library:
 libtool: link: gcc -shared  main/.libs/libtexinfo_la-tree.o main/.libs/libtexinfo_la-text.o main/.libs/libtexinfo_la-base_utils.o main/.libs/libtexinfo_la-builtin_commands.o main/.libs/libtexinfo_la-types_data.o main/.libs/libtexinfo_la-debug.o main/.libs/libtexinfo_la-errors.o .... -Wl,--whole-archive ./.libs/libcallperl_libtexinfo.a -Wl,--no-whole-archive  -liconv -lintl  -g -O2   -o .libs/cygtexinfo-1.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libtexinfo.dll.a

The following link flags are in Perl flags and are probably needed
 -Wl,--enable-auto-import -Wl,--export-all-symbols

Reported here:
https://savannah.gnu.org/support/index.php?111465


Documentation
=============

Tree documentation
------------------

Document the tree in texi2any_api instead of ParserNonXS.pm.

Document extra and info together as 'arguments' as in the SWIG interface
instead of separately?

Document the tree by showing the relations of inclusion?

elided_rawpreformatted, elided_brace_command_arg types.
linemacro_arg

alias_of in info hash

source marks.

special_unit_element type (only in HTML code)

extra node_number, heading_number and section_number are not documented.

Other
-----

Document the "language" used for the tta/perl/t/*.t tests results for the tree,
node and sectioning relations, output units, floats, indices and error
messages?

No documentation of Texinfo::Options hashes used in converters:
multiple_at_command_options, converter_cmdline_options,
converter_customization_options, unique_at_command_options.


Delayed bugs/features
=====================

GCC use a simple Perl script to convert Texinfo to POD, as an intermediary
for man pages generation.  There are also special macros and "magic"
comments, probably mainly to determine the blocks to use.
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=contrib/texi2pod.pl
https://gcc.gnu.org/onlinedocs/gccint/Man-Page-Generation.html
It would be nice to have a conversion to POD in texi2any.  A difficulty
is that there is nothing that corresponds to @anchor in POD (@*ref and
@anchor are simply ignored in texi2pod.pl).

Use SWIG interface in pod2texi, but wait first for the SWIG interface
to become common.

According to libtool documentation and Bruno experience, on Windows,
variable objects exported/imported from dll need to be declared, at
least when they are accessed from one library, which needs not happen
when they are const:
https://mail.gnu.org/archive/html/bug-texinfo/2026-02/msg00139.html
https://mail.gnu.org/archive/html/bug-texinfo/2026-02/msg00150.html
Eli and the github CI tests do not fail with XS for msys/mingw.  However,
ctexi2any on msys/mingw segfaults in the CI and in Eli tests.
In 2026-02 the variables that could be relevant are all set in libtexinfo.
The list:
C/main/customization_options.h: txi_base_options, txi_base_sorted_options,
txi_options_command_map => libtexinfo-convert (converter.c, format_html.c...)
C/main/document.h: txi_paths_info => libtexinfo-convert (convert_html.c...)
C/main/translations.h: translation_cache
  => libtexinfo-convert (converter.c, convert_to_text.c)
C/main/unicode.h: unicode_character_brace_no_arg_commands
  => libtexinfo-convert (html_prepare_converter.c)
C/main/utils.h: output_conversions, input_conversions
  => libtexinfo-convert (html_prepare_converter.c...),
     libtexinfo-main (swig_interface.c)
C/main/utils.h: null_device_names
  => texi2any.c

On Windows, the DLL are not found by texi2any.exe.
A solution hinted at by Bruno use libtool -bindir option, implemented in
https://cgit.git.savannah.gnu.org/cgit/texinfo.git/commit/?id=facdf1455f7ff333502ff8348d44ab434455a9c3
waiting for testing.
If it does not work, another possibility from Eli could be to copy the DLLs
into the directory where texi2any.exe is installed (in addition to the
directory where they are already copied by "make install".  That is, on
Windows there will be two copies of each DLL installed, not one.
https://lists.gnu.org/archive/html/bug-texinfo/2026-02/msg00090.html


hyphenation: should only appear in toplevel.


Some dubious nesting could be warned against.  The parsers context
command stacks could be used for that.

Some erroneous constructs not already warned against:

 @table in @menu

 @example
 @heading A heading
 @end example
 Example in heading/heading_in_example.

 @group outside of @example  (maybe there is no need for command stack for
                              this one if @group can only appear directly
                              in @example).

There is no warning with a block command between @def* and @def*x,
only for paragraph.  Not sure that this can be detected with
the context command stack.

 @defun a b c d e f

 @itemize @minus
 truc
 @item t
 @end itemize

 @defunx g h j k l m

 @end defun


Modules included in tta/maintain/lib/ are stable, but still need
to be updated from time to time.

Unicode::EastAsianWidth \p{InFullwidth} could be replaced
by native \p{East_Asian_Width=Fullwidth} + \p{East_Asian_Width=Wide}
when the oldest supported Perl version is 5.12.0 (released in 2010).


Future features
===============

Language dependent index collation has been removed from the documentation
and is only available in Perl.  In C texi2any, the undocumented
XS_STRXFRM_COLLATION_LOCALE customization variable can be used to specify a
locale and use it with strxfrm_l to sort, but it is only for testing and
should not be kept in the long term because of the lack of Non-ignorable
Variable Weighting.  Indeed, sorting indices in C with strxfrm_l using an
utf-8 locale with LC_COLLATE_MASK on Debian GNU/Linux with glibc is quite
consistent with the collation used in texi2any for number and letters,
but leads to a different output for non alphanumeric characters.  It is
because in texi2any we collate using Variable Weighting set to Non-ignorable
(see http://www.unicode.org/reports/tr10/#Variable_Weighting).  For spaces,
the output with Non-Ignorable Variable Weighting looks better for index
sorting, as it allows to have spaces and punctuation marks sort before
letters.
The plan is to wait for native C collating language dependent code
with customizable Variable Weighting before reenabling this feature.

Related glibc enhancement request (obsoleted by main/unicode-collation/
code?):
 request for Non-Ignorable Variable Weighting Unicode collation
 https://sourceware.org/bugzilla/show_bug.cgi?id=31658


Add the possibility to add text to a parsed document by restarting
parsing, when called as parse_piece or parse_string, by
storing the parser document state not already in document in document.
There would be a customization variable to set the parser to be
restartable, and then parse_piece and parse_string could pass
a document to retrieve the parsing state.  This should probably
wait for a clear use case.  Currently, the parser is never reused
for different documents in the main codes, only in specific tests.
If it is useful, it would only need to be available through the
SWIG interface, not in texi2any.


From Gavin on the preamble_before_beginning implementation:
Another way might be to add special input code to trim off and return
a file prelude.  This would moves the handling of this from the "parser" code
to the "input" code.  This would avoid the problematic "pushing back" of input
and would be a clean way of doing this.  It would isolate the handling of
the "\input" line from the other parsing code.

I understand that the main purpose of the preamble_before_beginning element
is not to lose information so that the original Texinfo file could be
regenerated.  If that's the case, maybe the input code could return
all the text in this preamble as one long string - it wouldn't have to be
line by line.


See message/thread from Reißner Ernst: Feature request: api docs
https://lists.gnu.org/archive/html/bug-texinfo/2022-02/msg00000.html

Right now VERBOSE is almost not used.

Should we warn if output is on STDOUT and OUTPUT_ENCODING_NAME != MESSAGE_OUTPUT_ENCODING_NAME?

Handle better @exdent in html? (there is a FIXME in the code)

For plaintext, implement an effect of NO_TOP_NODE_OUTPUT
* if true, output some title, possibly based on titlepage
  and do not output the Top node.
* if false, current output is ok
Default is false.

In Plaintext, @quotation text could have the right margin narrowed to be more
in line with other output formats.

Node, headings and section relations are explicitly and correctly documented in
the HTML customization manual.  However, they are also mentioned in
Texinfo::Structuring POD without being clearly introduced in this POD.  See
also FIXMEs in ParserNonXS.pm in =begin comment blocks.  It is not clear
whether there need to be something more explicit in the POD documentation,
what is in the HTML customization manual may be sufficient.


DocBook
-------

  deftypevr, deftypecv: use type and not returnvalue for the type

  also informalfigure in @float

  also use informaltable or table, for multitable?

  Add an @abstract command or similar to Texinfo?
  And put in DocBook <abstract>?  Beware that DocBook abstract is quite
  limited in term of content, only a title and paragraphs.  Although block
  commands can be in paragraphs in DocBook, it is not the case for Texinfo,
  so it would be very limited.

  what about @titlefont in docbook?

  maybe use simpara instead of para. Indeed, simpara is for paragraph without
  block element within, and it should be that way in generated output.

* in docbook, when there is only one section <article> should be better
  than book. Maybe the best way to do that would be passing the
  information that there is only one section to the functions formatting
  the page header and page footer.

  there is a mark= attribute for itemizedlist element for the initial mark
  of each item but the standard "does not specify a set of appropriate keywords"
  so it cannot be used.


Manual tests
============

Some tests are interesting but are not in the test suite for various
reasons.  It is not really expected to have much regressions with these
tests.  They are shown here for information.  It was up to date in
March 2024, it may drift away as tests files names or content change.
From tta/perl directory.


Tests in non utf8 locale
------------------------

In practice these tests were tested in latin1.  They are not
in the main test suite because a latin1 locale cannot be expected
to be present reliably.

Tests with correct or acceptable results
****************************************

File not found error message with accented characters in its name:
./texi2any.pl not_éxisting.texi

t/formats_encodings.t manual_simple_utf8_with_error
utf8 manual with errors involving non ascii strings
./texi2any.pl ./t/input_files/manual_simple_utf8_with_error.texi

t/formats_encodings.t manual_simple_latin1_with_error
latin1 manual with errors involving non ascii strings
./texi2any.pl ./t/input_files/manual_simple_latin1_with_error.texi

tests/formatting cpp_lines
CPP directive with non ascii characters, utf8 manual
./texi2any.pl -I ./t/include/ ./t/input_files/cpp_lines.texi
accentêd:7: warning: làng is not a valid language code
The file is UTF-8 encoded, the @documentencoding is obeyed which leads
in the Parser, to an UTF-8 encoding of include file name, and not to the latin1
encoding which should be used for the output messages encoding.
This output is by (Gavin) design.

many_input_files/output_dir_file_non_ascii.sh
non ascii output directory, utf8 manual
./texi2any.pl -o encodé/ ./t/input_files/simplest.texi

test of non ascii included file name in utf8 locale is already in formatting:
formatting/osé_utf8.texi:@include included_akçentêd.texi
./texi2any.pl --force -I ../tests/ ../tests/input/non_ascii/os*_utf8.texi
The file name is utf-8 encoded in messages, which is expected as we do not
decode/encode file names from the command line for messages
osé_utf8.texi:15: warning: undefined flag: vùr

t/80include.t cpp_line_latin1
CPP directive with non ascii characters, latin1 manual
./texi2any.pl --force ./t/input_files/cpp_line_latin1.texi

Need to have recoded file name to latin1 OK, see ../tests/README
tests/encoded manual_include_accented_file_name_latin1
./texi2any.pl --force -I ../tests/built_input/ ../tests/encoded/manual_include_accented_file_name_latin1.texi

latin1 encoded and latex2html in latin1 locale
./texi2any.pl --html --init ext/latex2html.pm ../tests/tex_html/tex_encode_latin1.texi

latin1 encoded and tex4ht in latin1 locale
./texi2any.pl --html --init ext/tex4ht.pm ../tests/tex_html/tex_encode_latin1.texi

cp -p ../tests/tex_html/tex_encode_latin1.texi tex_encodé_latin1.texi
./texi2any.pl --html --init ext/tex4ht.pm tex_encodé_latin1.texi
Firefox can't find tex_encod%uFFFD_latin1_html/Chapter.html (?)
Opened from within the directory, works well.

epub for utf8 encoded manual in latin1 locale
./texi2any.pl --force -I ../tests/ --init ext/epub3.pm ../tests/input/non_ascii/os*_utf8.texi

epub for latin1 encoded manual in latin1 locale
cp ../tests/tex_html/tex_encode_latin1.texi tex_encodé_latin1.texi
./texi2any.pl --init ext/epub3.pm tex_encodé_latin1.texi

./texi2any.pl --force -I ../tests/ -c TEXINFO_OUTPUT_FORMAT=rawtext ../tests/input/non_ascii/os*_utf8.texi
output file name is in latin1, but the encoding inside is utf8 consistent
with the document encoding.

./texi2any.pl --force -I ../tests/ -c TEXINFO_OUTPUT_FORMAT=rawtext ../tests/input/non_ascii/os*_utf8_no_setfilename.texi
output file name is utf8 because the utf8 encoded input file name
is decoded using the locale latin1 encoding keeping the 8bit characters
from the utf8 encoding, and the encoding inside is utf8
consistent with the document encoding.

./texi2any.pl --force -I ../tests/ -o encodé/raw.txt -c TEXINFO_OUTPUT_FORMAT=rawtext ../tests/input/non_ascii/os*_utf8.texi
encodé/raw.txt file name encoded in latin1, and the encoding inside is utf8
consistent with the document encoding.

./texi2any.pl --force -I ../tests/ -c TEXINFO_OUTPUT_FORMAT=rawtext -c 'SUBDIR=subdîr' ../tests/input/non_ascii/os*_utf8.texi
subdîr/osé_utf8.txt file name encoded in latin1, and the encoding inside is utf8
consistent with the document encoding.

./texi2any.pl --set TEXINFO_OUTPUT_FORMAT=debugtree -o résultat/encodé.txt ./t/input_files/simplest_no_node_section.texi
résultat/encodé.txt file name encoded in latin1.

./texi2any.pl --set TEXINFO_OUTPUT_FORMAT=debugtree -o char_latin1_latin1_in_refs_tree.txt ./t/input_files/char_latin1_latin1_in_refs.texi
char_latin1_latin1_in_refs_tree.txt content encoded in latin1

utf8 encoded manual name and latex2html in latin1 locale
./texi2any.pl --verbose -c 'COMMAND_LINE_ENCODING=utf-8' --html --init ext/latex2html.pm -c 'L2H_CLEAN 0' ../tests/input/non_ascii/tex_encod*_utf8.texi
COMMAND_LINE_ENCODING=utf-8 is required in order to have the
input file name correctly decoded as document_name which is used
in init file to set the file names.

latin1 encoded manual name and latex2html in latin1 locale
cp ../tests/tex_html/tex_encode_latin1.texi tex_encodé_latin1.texi
./texi2any.pl -c 'L2H_CLEAN 0' --html --init ext/latex2html.pm tex_encodé_latin1.texi

Tests with incorrect results, though not bugs
*********************************************

utf8 encoded manual name and latex2html in latin1 locale
./texi2any.pl --html --init ext/latex2html.pm -c 'L2H_CLEAN 0' ../tests/input/non_ascii/tex_encod*_utf8.texi
No error, but the file names are like
 tex_encodé_utf8_html/tex_encodÃ'$'\203''©_utf8_l2h.html
That's in particular because the document_name is incorrect because it is
decoded as if it was latin1.

utf8 encoded manual name and tex4ht in latin1 locale
./texi2any.pl --html --init ext/tex4ht.pm ../tests/input/non_ascii/tex_encod*_utf8.texi
html file generated by tex4ht with content="text/html; charset=iso-8859-1">,
with character encoded in utf8 <img src="tex_encodé_utf8_tex4ht_tex0x.png" ...>


Tests in utf8 locales
---------------------

The archive epub file is not tested in the automated tests.

epub for utf8 encoded manual in utf8 locale
./texi2any.pl --force -I ../tests/ --init ext/epub3.pm ../tests/input/non_ascii/osé_utf8.texi

The following tests require latin1 encoded file names.  Note that it
could be done automatically now with
tta/maintain/copy_change_file_name_encoding.pl.
However, there is already a test with an include file in latin1, it
is enough.
Create the latin1 encoded from a latin1 console:
cp ../tests/tex_html/tex_encode_latin1.texi tex_encodé_latin1.texi
Run from an UTF-8 locale console.  The resulting file has a ? in the name
but result is otherwise ok.
./texi2any.pl tex_encod*_latin1.texi

The following tests not important enough to have regression test
./texi2any.pl --force -I ../tests/ -o encodé/raw.txt -c TEXINFO_OUTPUT_FORMAT=rawtext ../tests/input/non_ascii/os*_utf8.texi
./texi2any.pl --force -I ../tests/ -c TEXINFO_OUTPUT_FORMAT=rawtext -c 'SUBDIR=subdîr' ../tests/input/non_ascii/os*_utf8.texi

Test more interesting in non utf8 locale
./texi2any.pl --set TEXINFO_OUTPUT_FORMAT=debugtree -o résultat/encodé.txt ./t/input_files/simplest_no_node_section.texi
résultat/encodé.txt file name encoded in utf8

./texi2any.pl --set TEXINFO_OUTPUT_FORMAT=debugtree -o char_latin1_latin1_in_refs_tree.txt ./t/input_files/char_latin1_latin1_in_refs.texi
char_latin1_latin1_in_refs_tree.txt content encoded in latin1


Texinfo manuals translations
============================

A po4a texinfo parser based on the Texinfo SWIG interface is to be
integrated in po4a, maybe for the next Texinfo release, which
should also have the SWIG interface built by default if SWIG is detected.

Using po4a with one of the two po4a texinfo parsers could be the
recommended way to translate manuals, solving some of the issues discussed
in the past.

Discussions started post 7.3 release:
https://lists.gnu.org/archive/html/bug-texinfo/2026-07/msg00050.html
https://lists.gnu.org/archive/html/bug-texinfo/2026-07/msg00051.html
https://lists.gnu.org/archive/html/bug-texinfo/2026-07/msg00052.html

Manual name:

The recommendation for translated manual Texinfo name is to use the
untranslated manual name and postpend a language code LL taking into account
@documentlanguage, @documentscript and @documentlanguagevariants, with
a -, for example between the untranslated manual name and the LL postfix, for
example manualname-LL.texi.  That way it is easy to make cross-references to
the translated manual name and register independent information in htmlxref for
cross-references to external translated HTML manuals.

Anchors and node names in translated manuals:

When starting from an untranslated manual, sectioning commands, nodes, anchor
and float names (hereafter called targets) would be in the untranslated
language (English).  Then the targets and the associated commands should be
translated as the manual is translated.  In the short term, it is recommended
to keep anchors with the untranslated target names such that cross-references
from external manuals can be either to translated or untranslated target names.
In the longer term, it is recommended to keep a mention of the untranslated
targets, such that it is possible for persons not understanding the translation
language to move sections around, but explicit anchors could be removed, and it
is even recommended to remove them in the long term.

Example of macros that could help the transition and remain in the final
translated manual.  "name" would be the translated name, and "en" the
untranslated name.

@c to be used to have an anchor added, for the short term
@macro nodetranslatedanchor {name, en}
@node \name\
@anchor{\en\}
@end macro

@c to be used to have a mention to the untranslated node, but no anchor
@c for the long term
@macro nodetranslatedmention {name, en}
@node \name\
@end macro

@c use in document
@nodetranslatedanchor{Mon noeud, My node}

Past discussions on that subject
--------------------------------

Main TODO
Support installation of manuals in different languages, along these lines:
  . support a LINGUAS file or variable saying which subdirs LL in the
    source to descend into (under doc/).
  . within each subdir LL, install the info files into $infodir/LL,
    and run install-info on $infodir/LL/dir.
  . info (both emacs and standalone) should read $infodir/$LANG/dir
    as the first dir file, and likewise read info files first from
    $infodir/$LANG, before falling back to $infodir.
  . consider ways to avoid installing images in both places.
    In fact, images probably need to be installed in a subdir
    $infodir/MANUAL/ in the first place, to avoid conflicts of having
    the same image name in different manuals.
  For a test case, see texinfo cvs, with its one translated manual
  (info-fr.texi).
  From Wojciech Polak.
  This is only one possibility, and should not necessarily need a
  change to the info readers, changing installation and INFOPATH
  could be enough.  There is another often used possibility, naming
  manuals with a suffix for the language.

https://lists.gnu.org/archive/html/bug-texinfo/2008-07/msg00005.html
https://lists.gnu.org/archive/html/bug-texinfo/2008-07/msg00022.html
using GUILE to store parsed Texinfo input, so one can write Scheme
code to play
- generating skeletons of Texinfo files to start a translation (it's
very useful to mention original node names in English in the
translation),
"linking" the same nodes in different languages,
so that e.g. a cross-reference to a node untranslated in some language
can be replaced by a cross-reference to the same node in a different
language.
- making statistics about translations (e.g. which parts of a document
are translated), which is useful for huge documents that cannot be
easily translated in one go, or looking for nodes that exist in one
language but not other (if you allow translators to write new
documentation in their own language).

https://lists.gnu.org/archive/html/bug-texinfo/2014-09/msg00002.html
https://lists.gnu.org/archive/html/bug-texinfo/2014-09/msg00010.html
Currently the usual practice is that a translated document is just another
document with some -xx ending in the base name (where xx refers to the
language, e.g. fr for French).
In my opinion an alternative method would be that translated document should
rather bare exactly the same name but just be in a language specific directory
named xx, with some fall backs to another language (meaning ../oo, where oo
refers to the other language) when manual does not exists.
I think that the alternative method is closer to what people usually do for Web
sites, in which each language is its own tree replica (file/directory names are
same, but only file contents differ).
About node name translation for presentation to
user (there should remain a node true name that should remain the same whatever
the translation.

https://lists.gnu.org/archive/html/bug-texinfo/2025-02/msg00079.html
About a second argument for anchor for the text displayed
* For translators, having the same anchor name as in the original
  document helps a lot in translation.  And vice versa, it helps
  maintainers who don't speak the particular language to still do
  various maintenance tasks easier.

* It helps avoid issues with transliteration.  All redirection file
  names are in a single language, namely English.

https://lists.gnu.org/archive/html/bug-texinfo/2025-02/msg00080.html
My feeling is that another solution would be better, for example a macro
that separates the association to the english label from the output
itself, based on the document language.  Something like

@macro translateanchor {en, name}
@anchor{\name\}
@end macro

@macro translateref {en, name}
\name\
@end macro

@translateanchor{in english, en français}

@ref{@translateref{in english, en français}, ...}.

it could probably
be possible to do something using customization functions to get the
macro call arguments by using the source marks

https://lists.gnu.org/archive/html/bug-texinfo/2025-05/msg00008.html
entry in 'dir'

* Guix: (guix.es).              Gestión del software instalado y la
                                  configuración del sistema.
...
* Guix: (guix).                 Manage installed software and system
                                  configuration.

I don't know if any changes are needed to Texinfo to support installation
of Info files in multiple languages.  It is something that could
perhaps be implemented with existing Info features, such as search paths
and installing Info files in different directories depending on the language.

https://lists.gnu.org/archive/html/help-texinfo/2024-01/msg00009.html
@dircategory
what about using “Translated manuals” for now,
since the number of translated manuals is very low?

The day when we have a reasonable amount of translated manuals, it will
make sense to have them under the original English manuals. I guess it
won’t be too difficult to make the change.

Would @author be also used for the translator?

For ex. emacs-gnutls.txi has
@author by Ted Zlatanov

In French that would be
@author par Ted Zlatanov
@author traduit en français par Achille Talon

https://lists.gnu.org/archive/html/help-texinfo/2024-01/msg00057.html
# Translating the Emacs manuals

## Location

Translated manuals sources are located in the ’doc/lang’ directory, under the
directory whose name corresponds to the translated language.

For ex. French manual sources are found under ’doc/lang/fr’.

The structure of the language folders should match the structure of the English
manuals (i.e. ’misc’, ’man’, ’lispref’, ’lispintro’, ’emacs’).

## Format

The translated manuals should be in the same format as the English sources:
TexInfo.

### TexInfo specific issues

Until Emacs/TexInfo provide better solutions, here are a few rules to follow:

- ’@node’ items should be translated but should be accompanied by an ’@anchor’
that contains the original English ’@node’ contents.

- You should add a ’@documentlanguage’ directive that includes your language.

For ex. ’@documentlanguage zh’

- ’@author’ can be used for the translator’s name.

Fr ex. `@author traduit en français par Achile Talon`

## Committing the files

Like other source files, translations should be committed to a separate branch
for revision. Ideally, the branch name should be suggestive of what they
contain.

For ex: ’origin/translations/emacs-lisp-intro-ar.texi’

Before committing the files for revision, ensure that they have been properly
checked for spelling/grammar/typography by at least using the tools that Emacs
provides.

You should also make sure that the TexInfo files build properly on your system.

Once the files are committed, announce the commit to the emacs-devel list so
that fellow translators can check the file and review it.

## Discussions about translation issues

Translation-related discussions are welcome on the emacs-devel list.
Discussions specific to your language do not have to take place in English.

## Notes about the original document

During the course of the translation, you will find parts of the original
document that needs to be updated or otherwise fixed. If you do not intend to
modify the original documents right away, do not add notes to the original
documents but rather keep such notes inside your translation as TODO items
until you action them.

## Translation teams

The number of words in the Emacs manuals is above 2,000,000 words. While one
individual could theoretically translate all the files, it is more practical to
work in language teams.

If you have a small group of translators willing to help, make sure that the
files are properly reviewed before committing them (see above.)

## Translation processes

Emacs does not provide yet tools that significantly help the translation
process. A few ideal functions would be:

- automatic lookup of a list of glossary items when starting to work on a
translation “unit” (paragraph or otherwise), such glossary terms should be
easily insertable at point

- automatic lookup of past translations to check for similarity and improve
homogeneity over the whole document set, such past translation matches should
be easily insertable at point

Although the PO format has not been developed with documentation in mind, it is
well known among free software translation teams and you can easily use the
’po4a’ utility to convert TexInfo to PO for work in translation tools that
support the PO format.

https://lists.gnu.org/archive/html/bug-texinfo/2014-09/msg00010.html
---> related to docstrings in Emacs
One second thing was internationalisation for *DocStrings*, in that context I
was suggesting that docstring could contain some link to some manual variable
or function description, so switching the Help language based on configured
locale would just mean switching to the correct manual translation.


LilyPond translations system
----------------------------

LilyPond has a very elaborate system:
https://lilypond.org/doc/v2.25/Documentation/contributor-big-page#cross_002dreferences
This system handles both translations and inclusion of LilyPond
snippets in output manuals.

Cross-references link target @rchanges: Changes file
All these commands also have a @...named version, which allows to specify the
displayed text for the reference as a second argument. This is mainly used in
translations, for example @rlearningnamed{I'm hearing voices, J'entends des
voix}.

Documentation/en/macros.itexi
@macro rchanges{TEXT}
@ref{\TEXT\,,,changes-big-page,Changes}
@end macro

@macro rchangesnamed{TEXT,DISPLAY}
@ref{\TEXT\,,\DISPLAY\,changes-big-page,Changes}
@end macro

https://lilypond.org/doc/v2.25/Documentation/contributor-big-page#translating-the-website-and-other-texinfo-documentation

Node names are not translated, only the section titles are. That is, every
piece in the original file like

@node Foo bar
@section_command Bar baz

should be translated as

@node Foo bar
@section_command translation of Bar baz

you should at least write the node definition in the expected source file and
define all its parent nodes; for each node you have defined this way but have
not translated, insert a line that contains @untranslated. That is, you should
end up for each untranslated node with something like

@node Foo bar
@section_command translation of Bar baz

@untranslated

...
However, some music snippets containing text that shows in the rendered music,
and sometimes translating this text really helps the user to understand the
documentation; in this case, and only in this case, you may as an exception
translate text in the music snippet, and then you must add a line immediately
before the @lilypond block, starting with

@c KEEP LY

When you encounter

@lilypondfile[...,texidoc,...]{filename.ly}

in the source, open Documentation/snippets/filename.ly, translate the texidoc header field it contains, wrap it with texidocMY-LANGUAGE = "...", and write it into Documentation/MY-LANGUAGE/texidocs/filename.texidoc. Additionally, you may translate the snippet’s title in the doctitle header field in case doctitle is a fragment option used in @lilypondfile; you can do this exactly the same way as texidoc. For instance, Documentation/es/texidocs/filename.texidoc may contain

doctitlees = "Spanish title baz"
texidoces = "
Spanish translation blah
"

https://lilypond.org/doc/v2.25/Documentation/contributor-big-page#documentation-translation-maintenance

make ISOLANG=MY-LANGUAGE check-translation

At the beginning of each translated file (except for .po files), there is a
committish (i.e., an SHA-1 tag consisting of 40 hexadecimal digits that
uniquely identify a specific commit in a git repository) to represent the
revision of the sources that you have used to translate this file from the file
in English.

When you have pulled and updated a translation, it is very important to update
this committish in the files you have completely updated (and only these);


Notes on classes names in HTML
==============================

In january 2022 the classes in HTML elements were normalized.  There are no
rules, but here is descriptions of the choices made at that time in case one
want to use the same conventions.  The objective was to have the link between
@-commands and classes easy to understand, avoid ambiguities, and have ways to
style most of the output.

The class names without hyphen were only used for @-commands, with one
class attribute on an element maximum for each @-command appearing in the
Texinfo source.  It was also attempted to have such a class for all
the @-commands with an effect on output, though the coverage was not perfect,
sometime it is not easy to select an element that would correspond to the
most logical association with the @-command (case of @*ref @-commands with
both a <cite> and a <a href> for example).

Class names <command>-* with <command> a Texinfo @-command name were
only used for classes marking elements within an @-command but in other
elements that the main element for that @-command, in general sub elements.
For example, a @flushright lead to a <div class="flushright"> where the
@flushright command is and to <p class="flushright-paragraph"> for the
paragraphs within the @flushright.

Class names *-<command> with <command> a Texinfo @-command name were
reserved for uses related to @-command <command>.  For example
classes like summary-letter-printindex, cp-entries-printindex or
cp-letters-header-printindex for the different parts of the @printindex
formatting.

def- and -def are used for classes related to @def*, in general without
the specific command name used.

For the classes not associated with @-commands, the names were selected to
correspond to the role in the document rather than to the formatting style.


In HTML, some @-commands do not have an element with a class associated, or the
association is not perfect.  There is @author in @quotation, @-command affected
by @definfoenclose.  @pxref and similar @-commands have no class for references
to external nodes, and don't have the 'See ' in the element for references to
internal nodes.  In general, it is because gdt() is used instead of direct
HTML.


Interrogations and remarks
==========================

Could libraries be compiled with Perl C compiler or other C compiler
and be mixed at the library level?  Probably not, we had to use the
Perl compilation and linking flags everywhere such that they match.

In general, it is better to avoid extra information in the tree that
can be retrieved, or that is better in separate data, such as node,
sectioning commands and heading relations.  In particular to avoid
referring to elements in other parts of the tree.  Right now, there
isn't any extra information referring to elements in other parts of
the tree (no extra element, contents nor directions).  It would
be better to keep it like this, but we keep the corresponding code in
case we need those extra categories back again.

For converters in C, agreed with Gavin that, in general, it is better not to
translate a perl tree in input, but access directly the C tree that was setup
by the XS parser.  When using the TreeElements interface, being able to
create a C element based on a Perl element is needed, though.

There is disagreement on the strategy to use for libraries versioning.
For Gavin, the code that can link against libraries should be the code
with the exact same version than the libraries.
For Patrice it is possible to explicitly say that mixing versions is
not supported, but still follow the libtool versioning scheme such that
if better and compatible libraries are installed they will be prefered
to same version library.

Gavin on using CSV files and a Perl script to generate code for Perl and C.
For something like html_style_commands_element.csv, it could potentially
make the code more impenetrable.  Suppose for example someone wants to
find which part of the code handles the @sansserif command.  If they
search HTML.pm for 'sansserif' that string isn't there.  As this file
is shorter, there is less benefit to avoiding duplicating its contents.
However, the purpose, structure and meaning of this file is quite clear.
(Files such as HTML.pm are also not self-contained, accessing information
in files such as Commands.pm, so having another file to access does not
really change the situation.)

For shorter files like default_special_unit_info.csv and
default_direction_strings.csv it is less clear that it offers a net
benefit.  To be honest, the function of these files is not particularly
clear to me other than one of them is something to do with node direction
pointers.  I don't think someone looking at these files for the first time
would have an easy time figuring out what they are for.

There is no forward looking code in the Parsers anymore, so maybe a
lex/yacc parser could be used for the main loop.  More simply, a binary
tokenizer, at least, could make for a notable speedup.

From vincent Belaïche. About svg image files in HTML:

I don't think that supporting svg would be easy: its seems that to embed an
svg picture you need to declare the width x height of the frame in
which you embed it, and this information cannot be derived quite
straightforwardly from the picture.
With @image you can declare width and height but this is intended for
scaling. I am not sure whether or not that these arguments can be used
for the purpose of defining that frame...
What I did in 5x5 is that coded the height of the frame directly in
the macro @FIGURE with which I embed the figure, without going through
an argument.
The @FIGURE @macro is, for html:
@macro FIGURE {F,W}
@html
<div align="center">
<embed src="5x5_\F\.svg" height="276"
 type="image/svg+xml"
 pluginspage="http://www.adobe.com/svg/viewer/install/" /></div>
@end html
@end macro


In general, external information for cross-references to other manuals from
htmlxref.cnf or htmlxref.d/*.cnf files should be used to determine
the split of a reference manual, and if no information is set, it
is a good idea to generate both a mono and a split manual.  Therefore the
following situation is not something that needs to be supported/implemented,
however we keep the information on the javascript code here.

If a manual is split and the person generating the manual wants that
references to a mono manual to be redirected to the split files, it should
be possible to create a manual.html file that redirects to the
manual_html/node.html files using the following javascript function:

 function redirect() {
   switch (location.hash) {
     case "#Node1":
       location.replace("manual_html/Node1.html#Node1"); break;
     case "#Node2" :
       location.replace("manual_html/Node2.html#Node2"); break;
     ...
     default:;
   }
 }

And, in the <body> tag of manual.html:
  <body onLoad="redirect();">


Need to make sure that a fix needed
-----------------------------------

Not sure that it can be solved easily.  When braced substrings in translated
strings are not in the translation result, they are not part of the returned
tree and are therefore never destroyed.  This happened in tests in the
translation_in_parser_in_translation test when '{name} on {class}'
translation did not contain {name} nor {class}.  It the translated strings
are not user-defined (for HTML), it is unlikely that the translations do
not contain the braced substrings.  For user-defined translated strings, it
could be possible.
It could be theoretically possible to determine if the braced substrings
are not all in the translated strings and destroy the non-substituted
element in the code perfoming the substrings substitution.  However, the
situation is not likely enough to warrant doing special code.

Similarly, if a braced substrings appears twice, it will probably make
the program crash as a tree element is destroyed twice leading to double
free and/or access to unallocated memory.


Do an XS interface for converter current_lang_translations?  It is
somewhat similar to customization setting based on @-commands in the
document, so it would be more consistent, but it requires adding quite a
few accessors, as cdt/gdt cannot be overriden.

We mark Perl tree element parent as weaken for tree copies to avoid having
to remove parent in trees returned after translation, such that they
do not have cycles already and can be destroyed by Perl.

It could be possible to mark Perl tree element parent as weak reference
with Scalar::Util weaken in Texinfo::ParserNonXS more generally, such that
there is no need to remove the parent to break cycles and remove tree element
references.  In perlapi for XS there is sv_rvweaken for the elements built from
C.  However, this would slow down the Parsers although for the majority of
cases, we do not call the code that deletes the tree element parent, we do it
only if there is more than one manual converted.  So it is not clear that it
is a good idea.

Make building "source marks" optional?  It does not use that much memory,
most probably even less time, and would complexify the code, it is not
clearly a good idea.

In HTML, HEADERS is used.  But not in other modules, especially not in
Plaintext.pm or Info.pm, this is determined by the module used (Plaintext.pm
or Info.pm).  No idea whether it is right or wrong.

def/end_of_lines_protected_in_footnote.pl the footnote is
   (1)  -- category: deffn_name arguments arg2 more args with end of line
and not
   (1)
 -- category: deffn_name arguments arg2 more args with end of line
It happens this way because the paragraph starts right after the footnote
number.

in HTML, the argument of a quotation is ignored if the quotation is empty,
as in
 @quotation thing
 @end quotation
Is it really a bug?

In @copying things like some raw formats may be expanded.  However it is
not clear that it should be the same than in the main converter.  Maybe a
specific list of formats could be passed to Texinfo::Convert::Text::convert,
which would be different (for example Info and Plaintext even if converting
HTML).  Not clear that it is a good idea.  Also this requires a test, to begin
with.

Punctuation and spaces before @image do not lead to a doubling of space.
In fact @image is completly formatted outside of usual formatting containers.
Not sure what should be the right way?
test in info_test/image_and_punctuation

in info_tests/error_in_footnote there is an error message for each
listoffloats;  Line numbers are right, though, so maybe this is not
an issue.

converters_tests/things_before_setfilename there is no error
for anchor and footnote before setfilename.  It is not clear that
there should be, though.

In Info, image special directive on sectioning command line length
is taken into account for the underlying characters line count inserted
below the section title.  There is no reason to underline the image
special directive.  Since the image rendering and length of replacement
text depends on the Info viewer, however, there is no way to know in
advance the lenght of text to underline (if any).  It is therefore unclear
what would be the correct underlying characters count.
An example in formats_encodings/at_commands_in_refs.

When using Perl modules, many strings in debugging output are internal
Perl strings not encoded before being output, leading to
'Wide character in print' messages (in C those strings are always encoded
in UTF-8).  Not clear that it is an issue.  For example with
export TEXINFO_XS=omit
/usr/bin/perl -w ./../perl/texi2any.pl  --force --conf-dir ./../perl/t/init/ --conf-dir ./../perl/init --conf-dir ./../perl/ext -I ./coverage/ -I coverage// -I ./ -I . -I built_input --error-limit=1000 -c TEST=1  --output coverage//out_parser/formatting_macro_expand/ --macro-expand=coverage//out_parser/formatting_macro_expand/formatting.texi -c TEXINFO_OUTPUT_FORMAT=structure ./coverage//formatting.texi --debug=1 2>t.err

This does not happen for t/*.t tests, as STDERR is encoded in UTF-8
in test_utils.pl.

Following was relevant when there were environment variables that allowed
to do part only part of parsing/structuring/converting in C.  Now that
it is not possible anymore, the next two paragraphs are probably obsolete.

The Perl/C interface when no XS code is used for structuring works because
all the C data is built as Perl structure at one point after which C code
is not called anymore to modify the tree or other Document data.  At that
point the F_DOCM_* flags should remain unset as there is no modification in
C data needing to be built to Perl.  Therefore all the calls to XS interfaces
afterwards, mainly on a Document, returns the Perl data.  This is not
fundamentaly problematic, as it works.  However, the design of the Document XS
interface was that the Perl structure could be built on demand, and in that
case, it is not possible.

When the structure code is done in C and conversion is done in Perl, there
is no need to rebuild all the document data to Perl.  This is probably
because there is no Perl code modifying the document data without also
modifying the underlying C code, ie no case of missing XS interface in
code run in tests at that point, before conversion.  But it is probably
more by chance than by design, if code changing the structure of the tree
that lack XS to change the C data was called at that point, the C and Perl
data would diverge.


HTML5 validation tidy errors that do not need fixing
----------------------------------------------------

# to get only errors:
tidy -qe *.html

Some can also be validation errors in other HTML versions.

missing </a> before <a>
discarding unexpected </a>
nested <a> which happens for @url in @xref, which is valid Texinfo.

Warning: <a> anchor "..." already defined
Should only happen with multiple insertcopying.

Warning: trimming empty <code>
Normally happens only for invalid Texinfo, missing @def* name, empty
@def* line...

<td> attribute "width" not allowed for HTML5
<th> attribute "width" not allowed for HTML5
These attributes are obsolete (though the elements are
still part of the language), and must not be used by authors.
The CSS replacement would be style="width: 40%".
However, width is kept as an attribute in texi2any @multitable output and not
as CSS because it is not style, but table or even line specific formatting.
If the _INLINE_STYLE_WIDTH undocumented option is set, CSS is used.
It is set for EPUB.
See
https://lists.gnu.org/archive/html/bug-texinfo/2024-09/msg00065.html


Specialized synopsis in DocBook
-------------------------------

Use of specialized synopsis in DocBook is not a priority and it is not even
obvious that it is interesting to do so.  The following notes explain the
possibilities and issues extensively.

Instead of synopsis it might seem to be relevant to use specialized synopsis,
funcsynopsis/funcprototype for deftype* and some def*, and other for object
oriented.  There are many issues such that this possibility do not appear
appealing at all.

1) there is no possibility to have a category.  So the category must be
   added somewhere as a role= or in the *synopsisinfo, or this should only
   be used for specialized @def, like @defun.

2) @defmethod and @deftypemethod cannot really be mapped to methodsynopsis
   as the class name is not associated with the method as in Texinfo, but
   instead  the method should be in a class in docbook.

3) From the docbook reference for funcsynopsis
  "For the most part, the processing application is expected to
  generate all of the parentheses, semicolons, commas, and so on
  required in the rendered synopsis. The exception to this rule is
  that the spacing and other punctuation inside a parameter that is a
  pointer to a function must be provided in the source markup."

  So this mean it is language specific (C, as said in the docbook doc)
  and one have to remove the parentheses, semicolons, commas.

  See also the mails from Per Bothner bug-texinfo, Sun, 22 Jul 2012 01:45:54.

specialized @def, without a need for category:
@defun and @deftypefun
<funcsynopsis><funcprototype><funcdef>TYPE <function>NAME</function><paramdef><parameter>args</parameter></paramdef></funcprototype></funcsynopsis>

specialized @def, without a need for category, but without DocBook synopsis
because of missing class:
@defmethod, @deftypemethod: methodsynopsis cannot be used since the class
is not available
@defivar and @deftypeivar: fieldsynopsis cannot be used since the class
is not available

Generic @def with a need for a category
For deffn deftypefn (and defmac?, defspec?), the possibilities of
funcsynopsis, with a category added could be used:
  <funcsynopsis><funcprototype><funcdef role=...>TYPE <function>NAME</function></funcdef><paramdef>PARAMTYPE <parameter>PARAM</parameter></paramdef></funcprototype></funcsynopsis>

Alternatively, use funcsynopsisinfo for the category.

Generic @def with a need for a category, but without DocBook synopsis because
of missing class:
@defop and @deftypeop: methodsynopsis cannot be used since the class
is not available
defcv, deftypecv: fieldsynopsis cannot be used since the class
is not available

Remaining @def without DocBook synopsis because there is no equivalent,
and requires a category
defvr (defvar, defopt), deftypevr (deftypevar)
deftp


Solaris
=======

# recent Test::Deep requires perl 5.12
cpan> o conf urllist push http://backpan.perl.org/
cpan RJBS/Test-Deep-1.127.tar.gz

Also possible to install Texinfo dependencies with openCSW, like
pkgutil -y -i CSWhelp2man CSWpm-data-compare CSWpm-test-deep

The system perl may not be suitable to build XS modules, and the
system gawk may be too old, openCSW may be needed.  For example,
on Solaris 11:
./configure PERL=/opt/csw/bin/perl GAWK=/opt/csw/bin/gawk CFLAGS='-g' LDFLAGS=-L/opt/csw/lib/ CPPFLAGS='-I/opt/csw/include/' PERL_EXT_CFLAGS='-g' PERL_EXT_CPPFLAGS='-I/opt/csw/include/' PERL_EXT_LDFLAGS=-L/opt/csw/lib/

On Solaris 10, gnulib requires C99, may need to add -D_STDC_C99 to CFLAGS.

On Solaris 10, there may be an error with Perl compilation flags like:
ld: fatal: file /usr/local/lib/libiconv.so: wrong ELF class: ELFCLASS64
because Perl cflags select 32 bits, but the library being linked in
/usr/local/lib/ is 64 bits.  In that case, the library should be found
in /opt/csw/lib/.
To mitigate this issue, --without-libiconv-prefix can be used.

With --without-libiconv-prefix, if gcc is used as compiler (without
the Perl compilation flags), there can be a runtime linking error,
because /opt/csw/lib is not included in the program runpath as there
is no -R/opt/csw/lib issued.  To use gcc, one possibility could be to do
 export LD_OPTIONS=/opt/csw/lib
Beware that it insert the runpath before any other path set during build.

Alternatively, to use cc, which is more interesting for Texinfo to test Solaris
10 specificities and reproduce the OpenCSW environment:
./configure --without-libiconv-prefix PERL=/opt/csw/bin/perl GAWK=/opt/csw/bin/gawk CC=cc CFLAGS='-g -D_STDC_C99=' LDFLAGS=-L/opt/csw/lib/ CPPFLAGS='-I/opt/csw/include/' PERL_EXT_CFLAGS='-g -D_STDC_C99=' PERL_EXT_CPPFLAGS='-I/opt/csw/include/' PERL_EXT_LDFLAGS=-L/opt/csw/lib/


OpenBSD
=======

compiling libintl-perl
gettext_xs/Makefile.PL:         LIBS => ['-L/usr/local/lib' . ' -lintl -liconv'],

Misc notes
==========

# differences between dist in and out source after maintainer-clean
rm -f texinfo-7.*.tar.gz
./autogen.sh
./configure
make maintainer-clean
./autogen.sh
./configure
make dist
rm -rf in_source_dist_contents
mkdir in_source_dist_contents
(cd in_source_dist_contents && tar xvf ../texinfo-7.*.tar.gz)
make maintainer-clean
rm -rf bb
mkdir bb
cd bb
../configure
make dist
tar xvf texinfo-7.*.tar.gz
diff -u -r ../in_source_dist_contents/texinfo-7.*/ texinfo-7.*/ > ../build_in_out_source_differences.diff

Test validity of Texinfo XML or docbook
export XML_CATALOG_FILES=~/src/texinfo/tta/maintain/catalog.xml
xmllint --nonet --noout --valid commands.xml

tidy does not seems to be released and/or maintained anymore.  It incorrectly
emits an error for ol type attribute.
tidy -qe *.html

profiling: package on debian:
libdevel-nytprof-perl
In doc:
perl -d:NYTProf ../tta/perl/texi2any.pl texinfo.texi --html
perl -d:NYTProf ../tta/perl/texi2any.pl texinfo.texi
nytprofhtml
# firefox nytprof/index.html

Test with 8bit locale:
export LANG=fr_FR; export LANGUAGE=fr_FR; export LC_ALL=fr_FR
xterm &

Turkish locale, interesting as ASCII upper-case letter I can become
a (non-ASCII) dotless i when lower casing.  (Eli recommendation).
export LANG=tr_TR.UTF-8; export LANGUAGE=tr_TR.UTF-8; export LC_ALL=tr_TR.UTF-8

On ExtUtils::Embed flags (the documentation tends to be inaccurate):
  ldopts: ccdlflags ldflags "libperl and perllibs through MakeMaker + static_ext"
  ccopts: ccflags perl_inc

convert to pdf from docbook
xsltproc -o intermediate-fo-file.fo /usr/share/xml/docbook/stylesheet/docbook-xsl/fo/docbook.xsl texinfo.xml
fop -r -pdf texinfo-dbk.pdf -fo intermediate-fo-file.fo

dblatex -o texinfo-dblatex.pdf texinfo.xml

Open a specific info file in Emacs Info reader: C-u C-h i

Count reference of the SV
Devel::Peek::SvREFCNT.  Not actually used because it cannot load in an eval.
Instead implemented in
Texinfo::ManipulateTree::SvREFCNT
Count reference of the object pointed to
Devel::Refcount::refcount

In tta/tests/, generate Texinfo file for Texinfo TeX coverage
../perl/texi2any.pl --force --error=100000 -c TEXINFO_OUTPUT_FORMAT=plaintexinfo -D valid layout/formatting.texi > formatting_valid.texi

From doc/
texi2pdf -I ../tta/tests/layout/ ../tta/tests/formatting_valid.texi

Check includes with iwyu.  Note that iwyu does not know when config.h
has to be included before a header that is replaced by a Gnulib header.
rm -rf ../res_iwyu_hdr
mkdir ../res_iwyu_hdr

for file in */*.h ; do bfile=`basename $file .h`; include-what-you-use -DDATADIR='"/usr"' -I ../ -I main $file 2>../res_iwyu_hdr/$bfile.2 ; done

for file in main/build_perl_info.h main/get_perl_info.h */build_html_perl_info.h *//build_html_perl_state.h */build_perl_tree_elements.h convert/get_converter_perl_info.h */get_perl_tree_elements.h ../perl/XSTexinfo/*/*.h ; do bfile=`basename $file .h`; include-what-you-use -I . -I main -I convert -D_LARGEFILE64_SOURCE -I /usr/lib/x86_64-linux-gnu/perl/5.42/CORE/ $file 2>../res_iwyu_hdr/$bfile.2 ; done

rm -rf ../res_iwyu_code
mkdir ../res_iwyu_code
for file in */*.c ; do bfile=`basename $file .c`; include-what-you-use -DDATADIR='"/usr"' -DHAVE_CONFIG_H -I ../ -I main -I structuring_transfo -I parsetexi -I ../gnulib/lib $file 2>../res_iwyu_code/$bfile.2 ; done

Remove filles that are directly included and files that are not compiled
for file in accent_tables_8bit_codepoints cmd_unicode command_data global_multi_commands_case global_unique_commands_case build_perl_tree_elements get_perl_tree_elements ; do rm -f ../res_iwyu_code/${file}.2 ; done

for file in main/api_to_perl.c convert/build_html_perl_info.c convert/build_html_perl_state.c main/build_perl_info.c convert/call_conversion_perl.c main/call_document_perl_functions.c convert/call_embed_perl.c convert/call_html_perl_function.c main/call_perl_function.c convert/get_converter_perl_info.c main/get_perl_info.c main/options_get_perl.c ; do bfile=`basename $file .c`; include-what-you-use -I . -I main -I convert -D_LARGEFILE64_SOURCE -I /usr/lib/x86_64-linux-gnu/perl/5.42/CORE/ $file 2>../res_iwyu_code/$bfile.2 ; done

NOTE: run make after changing an XS file.
rm -rf ../res_iwyu_xs
mkdir ../res_iwyu_xs
for file in ../perl/XSTexinfo/*.c ../perl/XSTexinfo/*/*.c ; do bfile=`basename $file .c`; include-what-you-use -I . -I main -I convert -I structuring_transfo -I parsetexi -D_LARGEFILE64_SOURCE -I /usr/lib/x86_64-linux-gnu/perl/5.42/CORE/ $file 2>../res_iwyu_xs/$bfile.2 ; done

include-what-you-use -I main ../perl/XSTexinfo/xsmisc.c


To generate valgrind .supp rules: --gen-suppressions=all --log-file=gen_supp_rules.log

for file in t/*.t ; do bfile=`basename $file .t`; echo $bfile; valgrind --suppressions=../texi2any.supp -q --leak-check=full perl -w $file > val_res/$bfile.out 2>&1 ; done

Check memory leaks
mkdir -p val_res
PERL_DESTRUCT_LEVEL=2
export PERL_DESTRUCT_LEVEL
for file in t/*.t ; do bfile=`basename $file .t`; echo $bfile; valgrind -q --leak-check=full perl -w $file > val_res/$bfile.out 2>&1 ; done
(for file in t/z_misc/*.t ; do bfile=`basename $file .t`; echo z_misc/$bfile; valgrind -q --leak-check=full perl -w $file ; done) > val_res/z_misc.out 2>&1

For tests in tta/tests, a way to have valgrind call prependend is to add,
in tta/defs (libtool exec is needed for ctexi2any):
prepended_command='libtool exec valgrind --leak-check=full -q'
prepended_command='libtool exec valgrind --leak-check=full -q --suppressions=../texi2any.supp'

Before the code reorganization and the separation of code compiled against
Gnulib+autoconf headers and code compiled against Perl, in some cases, memory
not released/freed still accessible at the end of conversion was shown by
valgrind as being leaked.  Mainly static/global variables freed upon reuse,
or left unreleased on purpose (parser conf, for example).  Also, some symbols
were shown as ???.  It is unclear what the problem was, the documentation
of valgrind hints that the memory appearing as leaked could have come a
dlclosed object.  In that case adding --keep-debuginfo=yes showed the
missing symbols as seen in the valgrind documentation.

To complement the "check that the debug output of pure Perl modules and XS/C
are the same" described in README-hacking, an additional check with memory leaks
in debug:
PERL_DESTRUCT_LEVEL=2
export PERL_DESTRUCT_LEVEL
unset TEXINFO_XS; for file in t/*.t ; do bfile=`basename $file .t`; valgrind -q --leak-check=full perl -w $file -d 1 2>t/check_debug_differences/XS_$bfile.err ; done

With debugging level 2
  rm -rf t/check_debug_differences/
  mkdir t/check_debug_differences/
  unset TEXINFO_XS; for file in t/*.t ; do bfile=`basename $file .t`; perl -w $file -d 2 2>t/check_debug_differences/XS_$bfile.err ; done
  export TEXINFO_XS=omit; for file in t/*.t ; do bfile=`basename $file .t`; perl -w $file -d 2 2>t/check_debug_differences/PL_$bfile.err ; done
  for file in t/*.t ; do bfile=`basename $file .t`; sed 's/^C|//' t/check_debug_differences/XS_$bfile.err | diff -u t/check_debug_differences/PL_$bfile.err - > t/check_debug_differences/debug_$bfile.diff; done


Check of XS interface to Perl
export TEXINFO_XS_EXTERNAL_FORMATTING=1
export TEXINFO_XS_EXTERNAL_CONVERSION=1


Analysing memory use (2026)
mkdir -p massif

Info
----
valgrind --tool=massif --massif-out-file=massif/massif_info.out perl -w texi2any.pl ../../doc/texinfo.texi
ms_print massif/massif_info.out > massif/ms_print_info.out
total 42.6M, extra-heap 6M (alignement, administrative bytes), useful-heap 36.2M
useful-heap approximate distribution:
16.3M Perl 9 + 4 + 1.7 + 1 + 0.6
17M C tree 5.3 + 5 + 3.4 + (0.8 + 0.7) text_alloc + 1.3 + 0.5 get_associated_info_key
1M conversion text
1.8M unknown
= 36.1M (good match with useful-heap)

libtool exec valgrind --tool=massif --massif-out-file=massif/massif_c_info.out ../C/ctexi2any ../../doc/texinfo.texi
ms_print massif/massif_c_info.out > massif/ms_print_c_info.out
total 25M, extra-heap 5.3M, useful-heap 19.7M
useful-heap total and distribution same as C above.

HTML
----
With full XS (7.2 64M, with text separate 58.5M, without info_info 56M
              with integer extra keys 54M, with source marks as pointers 52.3M)
valgrind --tool=massif --massif-out-file=massif/massif_html.out perl -w texi2any.pl --html ../../doc/texinfo.texi
ms_print massif/massif_html.out > massif/ms_print_html.out
total 58M, extra-head 6.9M, useful-heap 51.1M
useful-heap distribution:
22.6M Perl
 = 13.3 + 5.1 + 2.1 + 1.3 + 0.8
17.3M Tree
 5.5 + 5.3 = 10.8M new_element
 3.6M reallocate_list
 2.9M = (0.8 + 0.7) text + 1.4 new_text_element
6.2M Conversion
 4.4 (text) + 0.7 (text printindex) + 1.1 add_element_target_to_list
   mainly text in convert_output_output_unit*
1.7 M Text not imputed
 = 8.3 - (4.4 + 0.7 + 0.7 + 0.8)
3.4M remaining not imputed
51.2M TOTAL (for 51.1M reported)

Using callgrind to find the time used by functions

valgrind --tool=callgrind perl -w texi2any.pl ../../doc/texinfo.texi --html
# to avoid cycles (in Perl only code, in recursive C calls) that mess up the graph:
valgrind --tool=callgrind --separate-callers=3 --separate-recs=10 perl -w texi2any.pl ../../doc/texinfo.texi --html
valgrind --tool=callgrind --separate-callers=4 --separate-recs=11 perl -w texi2any.pl ../../doc/texinfo.texi
kcachegrind callgrind.out.XXXXXX

For the Texinfo manual with full XS, in 2026, Perl uses 39% of the time
(for html) for the main program and some HTML parser initialization.
Decomposition of the time used for the Texinfo manual with full XS
(in percent):
 parser: 17
 main conversion to HTML: 42
SUM: 98
Similar for ctexi2any for HTML, 30% parser, 70% conversion.
ctexi2any for Info 45% parser, 55% conversion.


Setting flags
# some features are only enabled at -O2, but sometime -O0 is better
# for debugging with valgrind
our_CFLAGS='-g -O0 -Wformat-security -Wstrict-prototypes -Wall -Wno-parentheses -Wno-unused-parameter -Wextra'
# keep cpp expanded files
# -save-temps
# Without -Wstack-protector there is no message on functions not protected.
# All these are in gnulib or gettext for now.
# -fno-omit-frame-pointer is better for debugging with valgrind, but has
# some speed penalty
our_CFLAGS='-g -O2 -D_FORTIFY_SOURCE=2 -Wformat-security -Wstrict-prototypes -Wall -Wno-parentheses -Wno-missing-braces -Wno-unused-parameter -fstack-protector-all -Wextra -fno-omit-frame-pointer'
our_CFLAGS='-g -O2 -Wformat-security -Wstrict-prototypes -Wall -Wno-parentheses -Wno-missing-braces -Wno-unused-parameter -Wextra'
./configure --enable-additional-checks "CFLAGS=$our_CFLAGS" "PERL_EXT_CFLAGS=$our_CFLAGS"
./configure --enable-using-c-texi2any --enable-additional-checks "CFLAGS=$our_CFLAGS" "PERL_EXT_CFLAGS=$our_CFLAGS"
unset our_CFLAGS

# test non-XS build
./configure --disable-perl-xs --enable-additional-checks "CFLAGS=$our_CFLAGS" PERL_EXT_CFLAGS=fail
or
./configure --enable-additional-checks "CFLAGS=$our_CFLAGS" PERL_EXT_CC=fail PERL_EXT_CFLAGS=fail