[Scons-users] Problems with a pseudo-builder

[Marc Joliet]

Hi all

[Note: my MUA ruined the line wrapping of the scons output and source code.  I
adjusted everything back manually, so apologies if I missed anything.]

[Another Note: while unsystematically trying things out I found something that
mostly fixes things for "-j1", but will post this email anyway to better
document how things progress.  I will write a follow-up email after I've had
some dinner.]

Yesterday I tried writing a somewhat complicated pseudo-builder that wraps three
different builders.

First some background info, so you can understand what I'm trying to do: This
pseudo-builder is part of a tool I'm working on for the FAUST language (see
https://github.com/marcecj/scons_faust). In short, FAUST is a functional DSL
for time-domain digital signal processing that compiles to C++ (or in the
FAUST2 branch to C, LLVM IR, etc.).  It also creates SVGs of the signal flow
graphs and — and this is what the pseudo-builder is for — mathematical
documentation.  This mathematical documentation is output as a LaTeX document
that references source code and images within a directory layout that looks
like this (for a source file named "rmfbs_sum.dsp"):

  rmfbs_sum-mdoc/
  ├── cpp  # can contain the generated C++ source code
  ├── pdf  # empty, the generated PDF should go here
  ├── src  # contains the FAUST sources
  ├── svg  # contains generated SVGs in a subdirectory "svg-01"
  └── tex  # contains the LaTeX document

The LaTeX document generated by default only sources one image:
svg/svg-01/process.pdf. This is the only case I want to handle for now.

So what I want the pseudo-builder to do is:

1.) generate the directory structure using "faust -mdoc -o /dev/null
<source_file>"

2.) convert svg/svg-01/process.svg to generate svg/svg-01/process.pdf (using
cairosvg in this case, but any tool would do)

3.) compile the LaTeX document in tex/ to a PDF file located in pdf/

I've got this working to a certain degree, but not quite.  Here are the sources
(since I haven't pushed any of this to my repo).

In builders.py I have:

    cairosvg = SCons.Builder.Builder(
        action = 'cairosvg --format=pdf -o $TARGET $SOURCE',
        suffix = '.pdf',
        src_suffix = '.svg',
        single_source = True
    )

    doc = SCons.Builder.Builder(
        action = ['$FAUST_FAUST ${FAUST_FLAGS} -o /dev/null -mdoc $SOURCE',
                  SCons.Script.Move('$TARGET', '${SOURCE.base}-mdoc')],
        suffix = lambda env,srcs: "-mdoc",
        src_suffix = '.dsp',
        single_source = True,
        source_scanner = dsp_src_scanner,
        target_factory = SCons.Script.Dir
    )

And in pseudo_builders.py I have:

    def doc_builder(env, target, source, *args, **kwargs):
        """
        A pseudo-builder for generating documentation via faust -mdoc.

        In addition to what the doc builder does, this pseudo-builder marks the
        mdoc directory for cleaning.
        """

        source = [env.File(s) for s in source]
        r = builders.doc(env, target, source, *args, **kwargs)

        for t, s in zip(r, source):

            # get a bunch of paths
            svg_dir     = env.Dir(os.sep.join([t.abspath, "svg", "svg-01"]))
            process_svg = svg_dir.File("process.svg")
            process_pdf = svg_dir.File("process.pdf")
            s_basename  = os.path.splitext(os.path.basename(s.path))[0]
            tex_path    = env.File(os.sep.join([t.abspath, "tex", s_basename+".tex"]))
	    pdf_path    = env.File(os.sep.join([t.abspath, "pdf", s_basename+".pdf"]))

            print(pdf_path)

            # compile the LaTeX sources to PDF
            process_pdf = builders.cairosvg(env, process_pdf.abspath, process_svg.abspath)
            cur_pdf  = env.PDF(pdf_path, tex_path)

            # set the dependencies straight
            env.Depends(process_pdf, t)
            env.Depends(cur_pdf, process_pdf)

            # the svg and pdf subdirectories change because of process_pdf and
            # cur_pdf, so make sure the *-mdoc directory ignores them
            env.Ignore(t, t.Dir("svg"))
            env.Ignore(t, t.Dir("pdf"))

            # make sure to clean up the documentation directory
            env.Clean(t, t.abspath)

            r.extend([cur_pdf, process_pdf])

        return r

doc_builder() is put into the BUILDERS variable as "FaustMDoc".

The general situation is that I am calling this from a sub-SConscript.
Everything goes into a build directory except for generated SVGs (which go
into "#svgs/") and the documentation.

So now the first situation. I call the pseudo-builder as

    cur_doc = env.FaustMDoc(os.sep.join(["#mdoc", dsp_name]), dsp)

The first time I call scons I get the following output (don't forget the
print() call in the pseudo-builder, which generates the extra output):

    $ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace
    scons: Reading SConscript files ...

    scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
    File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
    scons: done reading SConscript files.
    scons: Building targets ...
    /usr/bin/faust -t 4800 -vec -o /dev/null -mdoc src/rmfbs_sum.dsp
    Move("mdoc/rmfbs_sum-mdoc", "src/rmfbs_sum-mdoc")
    scons: *** [mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf] Source `mdoc/rmfbs_sum-mdoc/svg/svg-01/process.svg' not found, needed by target `mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf'.
    scons: internal stack trace:
    File "/usr/lib64/python2.7/site-packages/SCons/Job.py", line 387, in start
        task.prepare()
    File "/usr/lib64/python2.7/site-packages/SCons/Script/Main.py", line 168, in prepare
        return SCons.Taskmaster.OutOfDateTask.prepare(self)
    File "/usr/lib64/python2.7/site-packages/SCons/Taskmaster.py", line 189, in prepare
        executor.prepare()
    File "/usr/lib64/python2.7/site-packages/SCons/Executor.py", line 392, in prepare
        raise SCons.Errors.StopError(msg % (s, self.batches[0].targets[0]))
    scons: building terminated because of errors.

But when I call it a second time, it finishes exactly how I want it to:

    $ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace
    scons: Reading SConscript files ...

    scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
    File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
    scons: done reading SConscript files.
    scons: Building targets ...
    cairosvg --format=pdf -o mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf mdoc/rmfbs_sum-mdoc/svg/svg-01/process.svg
    cd mdoc/rmfbs_sum-mdoc/pdf && pdflatex -interaction=nonstopmode -recorder rmfbs_sum.tex
    This is pdfTeX, Version 3.1415926-2.4-1.40.13 (TeX Live 2012)
     restricted \write18 enabled.
    entering extended mode

    (/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/tex/rmfbs_sum
    .tex
    [...]
    scons: done building targets.

When I call it without "-j2" the error comes one step later (which I don't
understand at all since I explicitly specified the dependencies):

    $ scons rmfbs_sum-mdoc
    scons: Reading SConscript files ...

    scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
    File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
    scons: done reading SConscript files.
    scons: Building targets ...
    /usr/bin/faust -t 4800 -vec -o /dev/null -mdoc src/rmfbs_sum.dsp
    Move("mdoc/rmfbs_sum-mdoc", "src/rmfbs_sum-mdoc")
    cairosvg --format=pdf -o mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf mdoc/rmfbs_sum-mdoc/svg/svg-01/process.svg
    scons: *** [mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf] Source `mdoc/rmfbs_sum-mdoc/tex/rmfbs_sum.tex' not found, needed by target `mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf'.
    scons: building terminated because of errors.

The second situation is when I call the pseudo-builder with a target argument
whose path is relative (i.e., not prefixed with a "#"):

    cur_doc = env.FaustMDoc(dsp_name, dsp)

It behaves identically to the above.  Also, using

    cur_doc = env.FaustMDoc(os.sep.join(["#src", dsp_name]), dsp)

i.e., putting it in the same directory as the sources, behaves the same, too.

Now the third situation is especially odd and counter-intuitive.  I call the
pseudo-builder *without* the target argument:

    cur_doc = env.FaustMDoc(dsp)

The first time I call scons it behaves the same as before, but notice the paths:

    $ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace   
    scons: Reading SConscript files ...

    scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
    File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
    scons: done reading SConscript files.
    scons: Building targets ...
    /usr/bin/faust -t 4800 -vec -o /dev/null -mdoc src/rmfbs_sum.dsp
    Move("build-cpp-jack-qt/rmfbs_sum-mdoc", "src/rmfbs_sum-mdoc")
    scons: *** [build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf] Source `src/rmfbs_sum-mdoc/svg/svg-01/process.svg' not found, needed by target `build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf'.
    scons: internal stack trace:
    File "/usr/lib64/python2.7/site-packages/SCons/Job.py", line 387, in start
        task.prepare()
    File "/usr/lib64/python2.7/site-packages/SCons/Script/Main.py", line 168, in prepare
        return SCons.Taskmaster.OutOfDateTask.prepare(self)
    File "/usr/lib64/python2.7/site-packages/SCons/Taskmaster.py", line 189, in prepare
        executor.prepare()
    File "/usr/lib64/python2.7/site-packages/SCons/Executor.py", line 392, in prepare
        raise SCons.Errors.StopError(msg % (s, self.batches[0].targets[0]))
    scons: building terminated because of errors.

Unlike the first two cases, the paths are wrong:  the target path is in the
source directory, not the variant directory.  Furthermore, the second call to
scons fails:

    $ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace
    scons: Reading SConscript files ...

    scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
    File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
    /home/marcec/programming/faust_rm_filter_bank/src/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
    scons: done reading SConscript files.
    scons: Building targets ...
    scons: *** [build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf] Source `src/rmfbs_sum-mdoc/svg/svg-01/process.svg' not found, needed by target `build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf'.
    scons: internal stack trace:
    File "/usr/lib64/python2.7/site-packages/SCons/Job.py", line 387, in start
        task.prepare()
    File "/usr/lib64/python2.7/site-packages/SCons/Script/Main.py", line 168, in prepare
        return SCons.Taskmaster.OutOfDateTask.prepare(self)
    File "/usr/lib64/python2.7/site-packages/SCons/Taskmaster.py", line 189, in prepare
        executor.prepare()
    File "/usr/lib64/python2.7/site-packages/SCons/Executor.py", line 392, in prepare
        raise SCons.Errors.StopError(msg % (s, self.batches[0].targets[0]))
    scons: building terminated because of errors.

The reason for that is apparently the incorrect target paths, but there's also
another thing: the directory structure in the variant directory is wrong:

    $ tree -d build-cpp-jack-qt
    build-cpp-jack-qt
    └── rmfbs_sum-mdoc      # <- this directory shouldn't be here
        └── rmfbs_sum-mdoc  # <- should be a subdirectory of build-cpp-jack-qt
            ├── cpp
            ├── pdf
            ├── src
            ├── svg
            │   └── svg-01
            └── tex

This only ever happens in this particular scenario.

Finally, the same as the first two cases also happens when I call the
pseudo-builder directly in the top-level SConstruct: it fails on the first run
but completes correctly on the second.

Also as a note: just out of curiosity I added a "sleep 0.5s && ..." to the
cairosvg action string, but it didn't change anything (not that I expected it
to, but hey).

So I'm flummoxed :-/ .  I don't *think* I'm doing anything wrong, but maybe I
am?  Any help understanding what's going on here and maybe getting this working
is greatly appreciated :-) .

Greetings,
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://four.pairlist.net/pipermail/scons-users/attachments/20130607/417d2b28/attachment-0001.pgp