This is an overview of the current SCons documentation toolchain. As a prospective doc editor, you should be able to quickly understand the basic concepts (if not, please let the project know how it falls short). It is also a reference for core developers and the release team.
The diagram above roughly shows the steps that we currently use. Note there are two separate sets of documentation maintained with the project code itself: the reference manual and user guide, which have xml source files; and the docstrings in the Python code, from which the API documentation is generated. Most of the time when we talk about "SCons Documentation" we mean the former, as that's what is intended for end-user consumption.
If you look at the diagram, it looks a bit complicated. after reading this overview, it will be clear what is actually happening, and why we need all these steps.
Our toolchain doesn't only produce HTML and PDF files that are nice to look at, it also performs a lot of processing under the covers. We try to have our documentation as consistent as possible to the current behaviour of the source code, but this requires some extra steps.
So let's start right at the top...
SCons documentation is written in Docbook XML. The toolchain is set up so a writer has a restricted view of the whole "document processing thingy". All you should need to be concerned with is to edit existing text or write new sections and paragraphs. Sometimes even a completely new chapter has to be added. The hope is that you can fire up your XML editor of choice and type away.
XML is easy to get wrong, so you need to care about validating the XML files against our special "SCons Docbook DTD/XSD". If you're not using an XML editor, validate by
python bin/docs-validate.py
from the top source folder. If you are able to use
an XML editor, many of the potential problems are avoided,
as it will complain real-time.
The most common error is not matching tag opening and closing
(for example <tag>foo<tag> is an easy typing error to make,
as is starting a <para>, typing text, and not adding the
closing </para>, as is mistyping one of the two
tag markers). XML editors make it much harder to make
those errors, which, minor though they seem, will completely
break the document build. Unfortunately, the validation tools
(which rely on the capabilities of the XML parser in use)
are not that hard to fool, so you may get errors which are
not that easy to deciper. There isn't much we can do about that.
Everything's looking okay, all validation passed? Good, then simply commits your new work, and create a pull request on Github. That's it!
Additionally, you can create the single documents locally if you want
to get a feel for how the final result looks (and who doesn't?). Each of
the document folders (design, developer, man, python10,
reference, and user) contains an SConstruct file along with
the actual XML files. You can call
python ../../scripts/scons.py
from within the directory, and have the MAN pages or HTML created...even
PDF, if you have a renderer installed (fop, xep or jw).
At least fop doesn't like everything the docbook/xml chain
produces and will spew a lot of errors, which we think are harmless.
Just a few more words about the validation step. We are using our
own DTD/XSD as a kind of hook, which only exists to link our own SCons
documentation tags into the normal Docbook XSD. For the output, we always
have an intermediary step (see diagram above), where we rewrite tags like
cvar into a block of Docbook formatting elements representing it.
The toolchain, and all the Python scripts supporting it, are based on the prerequisite that all documents are valid against the SCons Docbook XSD. This step guarantees that we can accept the pull request of a user/writer with all his changes, and can create the documentation for a new release of SCons without any problems at a later time.
We are using entities for special keywords like SCons that should
appear with the same formatting throughout the text. This allows a
single place to make styling changes if needed. These are kept in
a single file doc/scons.mod which gets included by the documents,
and can be used anywhere in the documentation files.
Additionally, for the definitions of the four special types available
in the SCons doctype - Tool, Builder, Construction Variable and Function -
a bunch of reference links in the form of entities are generated.
These entities can be used in the MAN page and the User manual.
Note that the four type tags themselves (<tool>, <builder>,
<cvar> and <function>) can only be used in documentation
sources in the SCons directory; the build will not scan for these
tags in files in the doc directory.
When you add tool documentation using the <tool> tag,
let's say for a tool named foobar, you can use the two
automatically generated entities
- t-foobar
- which prints the name of the Tool, and
- t-link-foobar
- which is a link to the description of the Tool
The link will be to the appropriate Appendix in the User Guide, or to the proper section in the manpage.
The <builder> tag similarly generates entities with the b- prefix,
the <function> tag generates entities with the f- prefix,
and the <cvar> tag generates entities with the cv- prefix.
In the case of Functions, there may be pairs of these, depending on the value of the signature attribute: this attribute tells whether only the global function form, or only the environment method form, or both, exist. If all four exist you will get:
- f-foobar
- which prints the name of the global Function
- f-env-foobar
- which prints the name of the environment method
- f-link-foobar
- which is a link to the description of the global Function
- f-link-env-foobar
- which is a link to the description of the environment method
By calling the script
python bin/docs-update-generated.py
you can recreate the lists of entities (*.mod) in the generated
folder. At the same time, this will generate the matching *.gen
files, which hold the full descriptions of all the Builders, Tools,
Functions and CVars for the MAN page and the User Guide's appendix.
Thus, you want to regenerate when there's a change to
any of those four special elements, or an added or deleted element.
These generated files are left checked in so in the normal case you
can just rebuild the docs without having to first generate the entity
files.
For more information about how to properly describe these elements,
refer to the start of the Python script bin/SConsDoc.py. It explains
the available tags and the exact syntax in detail.
Normal Docbook (v4.5 style, as of this writing) in-document linking
is supported, as is linking to documents with a web address.
For any element in a document, you can include an id=name
attribute to set an identifier, and write a link to that identifier.
Many of the section headings already have such identifiers,
and it is fine to add more, as long as they remain unique.
As noted in the previous section, for the special types,
entities are generated which contain links,
so you can just use those entities instead
of writing the link reference manually.
There is something to keep in mind about linking, however.
Cross-document links between the MAN page and the User Guide
do not work. But some text is shared between the two, which
allows the appearance of such linking, and this is where it
gets a little confusing. The text defined by the four special
types is generated into the *.gen files,
which get included both in the appropriate places in the MAN page,
and in the Appendix in the User Guide. Using entities within
this shared content is fine. Writing links in this shared
content to element identifiers defined elsewhere is not.
That sounds a little confusing so here is a real example:
an xml source file in SCons defines the SCANNERS
construction variable by using <cvar name="SCANNERS"> ... </cvar>.
This will generate the linking entity &cv-link-SCANNERS;,
which can be used anywhere the doc/generated/variables.gen
file is included (i.e. MAN page and User Guide for now)
to leave a link to this definition.
But the text written inside the SCANNERS definition
also wants to refer to the "Builder Objects" and "Scanner
Objects" sections in the MAN page, as this contains relevant
further description. This reference should not include an
XML link, even though the MAN page defines the two identifiers
scanner_objects and builder_objects, because this
definition will also be included in the User Guide, which
has no such section names or identifiers. It is better here
to write it all in text, as in See the manpage section
"Builder Objects" than to leave a dangling reference in one
of the docs.
While it is very convenient to document related
things together in one xml file, and this is encouraged
as it helps writers keep things in sync,
be aware the information recorded inside the four special tags
will not be presented together in the output documents.
For example, when documenting a Tool in
SCons/Tool/newtool.xml using the <tool> tag,
and noting that the tool <uses> or <sets>
certain construction variables,
those construction variables can be documented
right there as well using <cvar> tags.
When processed with SConsDoc module,
this will generate xml from the
<tool> tag into the tools.{gen,mod} files,
and xml from the <cvar> tag into
the variables.{gen,mod} files;
those files are then included each into their own
section, so the entries may end up separated by
hundreds of lines in the final output.
The special entries will also be sorted in their
own sections, which might cause two entries using the
same tag in the same source file to be separated.
All this to say: do not write your doc text
with the idea that the locality you see in the xml source file
will be preserved when consumed in a web browser,
manpage viewer, PDF file, etc. Provide sufficient context
so entries can stand on their own.
Another quirk is that SConsDoc
will take all occurrences of a special tag and
combine those contents into a single entry in the generated file.
As such, normally there should be only one definition of
each element project-wide. This particularly comes up in tool definitions,
as several tools may refer to the same construction variable.
It is suggested to pick one file to write the documentation in,
and then in the other tool documents referencing it,
place a comment indicating which file the variable is documented in -
this will keep future editors from having to hunt too far for it.
In the User Guide, we support automatically created examples. This means that the output of the specified source files and SConstructs is generated by running them with the current SCons version. We do this to ensure that the output displayed in the manual is identical to what you get when you run the example on the command-line, without having to remember to manually update the example outputs all the time.
A short description about how these examples have to be defined can be
found at the start of the file bin/SConsExamples.py. Call
python bin/docs-create-example-outputs.py
from the top level source folder, to run all examples through SCons.
Before this script starts to generate any output, it checks whether the
names of all defined examples are unique. Another important prerequisite
is that for every example all the single scons_output blocks need
to have a suffix attribute defined. These suffixes also have to be
unique, within each example, as this controls the ordering.
All example output files (*.xml) get written to the folder
doc/generated/examples, together with all files defined via the
scons_example_file tag. They are put under version control, in
part so that the version control system can show any unexpected
changes in the outputs after editing the docs:
git diff doc/generated/examples
Some of the changes in example text are phony: despite best efforts to eliminate system-specifics, sometimes they leak through. There is at least one example that gets the pathname to the build directory of the machine the example is generated on.
Note that these output files are not actually needed for editing the
documents. When loading the User manual into an XML editor, you will
always see the example's definition. Only when you create some output,
the files from doc/generated/examples get XIncluded and all special
scons* tags are transformed into Docbook elements.
Documents are in the folders design, developer, man,
python10, reference, and user. Note that of these,
only man and user are actively maintained and some of
the others are vastly out of date. If submitting a github
Pull Request for a new SCons feature, you will only be required
to update the documentation that goes into the manpage and the
User Guide.
- editor_configs
- Prepared configuration sets for the validating WYSIWYG XML editors
XmlMind and Serna. You'll probably want to try the latter, because
the XXE config requires you to have a full version (costing a few
hundred bucks) and is therefore untested. For installing the Serna
config, simply copy the
sconsfolder into thepluginsdirectory of your installation. Likewise, the XXE files from thexmlmindfolder have to be copied into~/.xxe4/under Linux. - generated
- Entity lists and outputs of the UserGuide examples. They get generated
by the update scripts
bin/docs-update-generated.pyandbin/docs-create-example-outputs.py. - images
- Images for the
overview.rstdocument. - xsd
- The SCons Docbook schema (XSD), based on the Docbook v4.5 DTD/XSD.
- xslt
- XSLT transformation scripts for converting the special SCons
tags like
scons_outputto valid Docbook during document processing.