[Scons-users] Feedback request: an AsciiDoc tool

[Marc Joliet]

Hi all!

I've been working on an asciidoc tool for slightly over a week now and think
that it has reached the point where it makes sense for me to request feedback
on it. It also helps that I've grown tired of working almost exclusively on it
in my free time, so am looking for an excuse to take a break :) .

Background: in one hobby project [0] I opted to write the documentation in
AsciiDoc. Originally, this was just because I liked the syntax and thought it
would make a good format for the README file. Now I have recently started
actually using it as the basis for the project website and a PDF manual. As a
result, I have become frustrated by the lack of a proper builder for it. Since
I could not find an already existing one on the web, in traditional open source
fashion I scratched my own itch and started one:

  https://github.com/marcecj/scons_asciidoc

The README explains it in some detail from a usage standpoint, so I'll just
list some of its features:

- adds two new builders, AsciiDoc and A2X
- finds implicit dependencies from include:[] and image:[] inline macros and the
  include::[], image::[] and include1::[] block macros; this does not handle
  conditional inclusion through ifdef:[] macros
- adds various *.conf files in the source directory that asciidoc looks for
  to the target dependencies (note that it does not handle global paths
  like /usr/local/etc/), along with manually specified *.conf files
- extracts dependencies from certain construction variables
- adds files found in A2X_FLAGS to the target dependencies, mostly to handle
  options like '--dblatex-opts="--texstyle=path/to/style/file.sty"' generically
- cleans up intermediary output files produced by a2x
- does simple sanity checking on doctype/backend/format combinations (e.g.,
  the "book" doctype only works with the asciidoc "docbook" backend)

Even so, it does not do as much as it could. The reasons for that are that I
don't know AsciiDoc inside out, and have no experience with its more advanced
features. It also doesn't help that I have no experience at all with DocBook.

It also lacks a test suite, but I have tested it informally in [0] to make sure
it works in general (the project is only 9 days old, after all). However, I do
consider it to be the single biggest TODO. And speaking of which: where is the
best place to start reading up on testing SCons tools? The API documentation?

Now to get to the point: despite my reservations above, I would like to receive
some feedback on it! Feel free to open bugs, too. Even if you don't use
AsciiDoc yourself, feedback on the implementation would be nice, too :) .

I do have some questions, though:

1.) Exactly what calls exists(env) and why? I've wondered about this before,
and I tried unconditionally returning False, with no change in build system
behavior. So what is the effect of this function supposed to be? The best
google gave me was an old blog post where somebody was trying out SCons
and wondered the same thing (the person liked SCons, BTW).

2.) Is it bad/wrong to add a pseudo-builder to env['BUILDERS']? I do this
because the call signature is made compatible to other builders, e.g., you can
leave out the target argument and have it automatically be generated from
source[0]. Calling AddMethod() cannot ensure this (since you can add arbitrary
functions with it). I assume it's OK, if issue 780 still represents the
consensus, but wanted to ask anyway. It it does, than I would say that the user
guide ought to be updated :) .

3.) When I pass skeys to the constructor of a recursive Scanner, why does it
scan files with non-matching suffixes? I wanted to have skeys replace a
scan_check function that only returns True if the nodes suffix is ".txt", since
the scanner is recursive and I only want it to scan AsciiDoc files (as apposed
to, e.g., images).

Other than that (and my longer "question" on emitters below), my experience
writing this tool has been good so far. I especially came to like the power of
variable substitution. That made it much easier to handle option-dependent
dependencies (e.g., the $AD_CONFFILES and $AD_GET_CONF pair).

4.) Now a longer bit about emitters. Sorry in advance for the long text, I just
want the motivation for the later questions to be clear (or at least complete).

I originally thought that it would be best to let SCons know about the build
intermediaries by adding them to the targets via an emitter (see the source
code at [1]). However, this did weird things to the dependency tree.

To elaborate: if I have A2X() build "chunked" HTML output (one HTML file per
section/chapter in a predictably named directory), it produces an XML
intermediary: the asciidoc file converted to docbook.

Now the XML file suddenly depended on the "chunked" output files. Every time I
tried to build the target, it would get rebuilt because (according to
--debug=explain), the *xml file* had to be rebuilt because *some* of the HTML
files changed (never all).  Which they couldn't have, because the asciidoc
source file hadn't changed. For example:

  scons: rebuilding `doc/index.xml' because:
             `doc/index.chunked/ar01s02.html' changed
             `doc/index.chunked/ar01s03.html' changed
             `doc/index.chunked/ar01s06.html' changed
             `doc/index.chunked/ar01s07.html' changed
             `doc/index.chunked/index.html' changed

To work around this I called Ignore() on the intermediary files, which however
didn't change the fact that the intermediary depended on its own outputs.

To clarify, the build process looks something like this:

  source.txt -> source.xml -> source.chunked/*

And the dependencies looked like this:

  +-pdf
    +-doc/index.chunked
    | +-doc/index.txt
    | [...]
    | +-/usr/bin/a2x
    | +-doc/index.chunked/apa.html
    | +-doc/index.chunked/apb.html
    | [...]
    +-doc/index.xml
    | +-doc/index.txt
    | [...]
    | +-/usr/bin/a2x
    | +-doc/index.chunked/apa.html
    | +-doc/index.chunked/apb.html
    | [...]
    +-doc/index.chunked/apa.html
    +-doc/index.chunked/apb.html
  [...]

That "index.xml" depends on "index.chunked/*" is, well, wrong, because that's
the actual output of the builder. Everything else is fine. It's almost as if
implicit dependencies of the actual target (the contents of the directory) get
grafted onto the emitted targets.

I'm not sure what the problem was.  I see in the SCons wiki that some builders
use emitters for exactly this (e.g., http://www.scons.org/wiki/CudaTool).

So I guess what I want to know is, given the above: what, specifically, are
emitters designed for? Was I using them for something they are simply not
intended for? And can anybody tell me what, exactly, is going on with the
dependency tree above?

I think I just wanted to be able to tell SCons about an opaque builder: a2x can
be seen as something of a mini build system that calls asciidoc and various
programs for you, which is hidden from SCons. Ideally, you could say: these are
the files being built, and this is their build/dependency order and scons would
"realize" that the intermediaries are built before the actual target by the
builder. Sort of like declaring a multi-step builder, only what actually
happens is opaque to SCons.

But I suspect that falls outside the scope of a build system and the correct way
to deal with it is to just tell SCons which files a2x produced along the way and
should be cleaned up (like A2X() does now).

The real solution then would be to write a set of builders that does what a2x
does. And that basically already exists: Dirk Bächle's docbook tool (only it
depends on fop and I do not want Java on my system if I can avoid it).

If any of this has been asked/explained before, feel free to point me to
existing threads. I simply could not phrase a concise search query for google.

[0] http://msndfile.sf.net
[1]
https://github.com/marcecj/scons_asciidoc/blob/331f20e4601fb8ad1c8d2d043e9e698f60e38397/__init__.py

Thanks in advance
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : <http://four.pairlist.net/pipermail/scons-users/attachments/20121126/81542019/attachment.pgp>