[Scons-users] Problems with a pseudo-builder

Marc Joliet marcec at gmx.de
Fri Jun 7 15:24:09 EDT 2013


Hi all

[Note: my MUA ruined the line wrapping of the scons output and source code. I
adjusted everything back manually, so apologies if I missed anything.]

[Another Note: while unsystematically trying things out I found something that
mostly fixes things for "-j1", but will post this email anyway to better
document how things progress. I will write a follow-up email after I've had
some dinner.]

Yesterday I tried writing a somewhat complicated pseudo-builder that wraps three
different builders.

First some background info, so you can understand what I'm trying to do: This
pseudo-builder is part of a tool I'm working on for the FAUST language (see
https://github.com/marcecj/scons_faust). In short, FAUST is a functional DSL
for time-domain digital signal processing that compiles to C++ (or in the
FAUST2 branch to C, LLVM IR, etc.). It also creates SVGs of the signal flow
graphs and — and this is what the pseudo-builder is for — mathematical
documentation. This mathematical documentation is output as a LaTeX document
that references source code and images within a directory layout that looks
like this (for a source file named "rmfbs_sum.dsp"):

rmfbs_sum-mdoc/
├── cpp # can contain the generated C++ source code
├── pdf # empty, the generated PDF should go here
├── src # contains the FAUST sources
├── svg # contains generated SVGs in a subdirectory "svg-01"
└── tex # contains the LaTeX document

The LaTeX document generated by default only sources one image:
svg/svg-01/process.pdf. This is the only case I want to handle for now.

So what I want the pseudo-builder to do is:

1.) generate the directory structure using "faust -mdoc -o /dev/null
<source_file>"

2.) convert svg/svg-01/process.svg to generate svg/svg-01/process.pdf (using
cairosvg in this case, but any tool would do)

3.) compile the LaTeX document in tex/ to a PDF file located in pdf/

I've got this working to a certain degree, but not quite. Here are the sources
(since I haven't pushed any of this to my repo).

In builders.py I have:

cairosvg = SCons.Builder.Builder(
action = 'cairosvg --format=pdf -o $TARGET $SOURCE',
suffix = '.pdf',
src_suffix = '.svg',
single_source = True
)

doc = SCons.Builder.Builder(
action = ['$FAUST_FAUST ${FAUST_FLAGS} -o /dev/null -mdoc $SOURCE',
SCons.Script.Move('$TARGET', '${SOURCE.base}-mdoc')],
suffix = lambda env,srcs: "-mdoc",
src_suffix = '.dsp',
single_source = True,
source_scanner = dsp_src_scanner,
target_factory = SCons.Script.Dir
)

And in pseudo_builders.py I have:

def doc_builder(env, target, source, *args, **kwargs):
"""
A pseudo-builder for generating documentation via faust -mdoc.

In addition to what the doc builder does, this pseudo-builder marks the
mdoc directory for cleaning.
"""

source = [env.File(s) for s in source]
r = builders.doc(env, target, source, *args, **kwargs)

for t, s in zip(r, source):

# get a bunch of paths
svg_dir = env.Dir(os.sep.join([t.abspath, "svg", "svg-01"]))
process_svg = svg_dir.File("process.svg")
process_pdf = svg_dir.File("process.pdf")
s_basename = os.path.splitext(os.path.basename(s.path))[0]
tex_path = env.File(os.sep.join([t.abspath, "tex", s_basename+".tex"]))
pdf_path = env.File(os.sep.join([t.abspath, "pdf", s_basename+".pdf"]))

print(pdf_path)

# compile the LaTeX sources to PDF
process_pdf = builders.cairosvg(env, process_pdf.abspath, process_svg.abspath)
cur_pdf = env.PDF(pdf_path, tex_path)

# set the dependencies straight
env.Depends(process_pdf, t)
env.Depends(cur_pdf, process_pdf)

# the svg and pdf subdirectories change because of process_pdf and
# cur_pdf, so make sure the *-mdoc directory ignores them
env.Ignore(t, t.Dir("svg"))
env.Ignore(t, t.Dir("pdf"))

# make sure to clean up the documentation directory
env.Clean(t, t.abspath)

r.extend([cur_pdf, process_pdf])

return r

doc_builder() is put into the BUILDERS variable as "FaustMDoc".

The general situation is that I am calling this from a sub-SConscript.
Everything goes into a build directory except for generated SVGs (which go
into "#svgs/") and the documentation.

So now the first situation. I call the pseudo-builder as

cur_doc = env.FaustMDoc(os.sep.join(["#mdoc", dsp_name]), dsp)

The first time I call scons I get the following output (don't forget the
print() call in the pseudo-builder, which generates the extra output):

$ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace
scons: Reading SConscript files ...

scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
scons: done reading SConscript files.
scons: Building targets ...
/usr/bin/faust -t 4800 -vec -o /dev/null -mdoc src/rmfbs_sum.dsp
Move("mdoc/rmfbs_sum-mdoc", "src/rmfbs_sum-mdoc")
scons: *** [mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf] Source `mdoc/rmfbs_sum-mdoc/svg/svg-01/process.svg' not found, needed by target `mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf'.
scons: internal stack trace:
File "/usr/lib64/python2.7/site-packages/SCons/Job.py", line 387, in start
task.prepare()
File "/usr/lib64/python2.7/site-packages/SCons/Script/Main.py", line 168, in prepare
return SCons.Taskmaster.OutOfDateTask.prepare(self)
File "/usr/lib64/python2.7/site-packages/SCons/Taskmaster.py", line 189, in prepare
executor.prepare()
File "/usr/lib64/python2.7/site-packages/SCons/Executor.py", line 392, in prepare
raise SCons.Errors.StopError(msg % (s, self.batches[0].targets[0]))
scons: building terminated because of errors.

But when I call it a second time, it finishes exactly how I want it to:

$ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace
scons: Reading SConscript files ...

scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
scons: done reading SConscript files.
scons: Building targets ...
cairosvg --format=pdf -o mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf mdoc/rmfbs_sum-mdoc/svg/svg-01/process.svg
cd mdoc/rmfbs_sum-mdoc/pdf && pdflatex -interaction=nonstopmode -recorder rmfbs_sum.tex
This is pdfTeX, Version 3.1415926-2.4-1.40.13 (TeX Live 2012)
restricted \write18 enabled.
entering extended mode

(/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/tex/rmfbs_sum
.tex
[...]
scons: done building targets.

When I call it without "-j2" the error comes one step later (which I don't
understand at all since I explicitly specified the dependencies):

$ scons rmfbs_sum-mdoc
scons: Reading SConscript files ...

scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/mdoc/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
scons: done reading SConscript files.
scons: Building targets ...
/usr/bin/faust -t 4800 -vec -o /dev/null -mdoc src/rmfbs_sum.dsp
Move("mdoc/rmfbs_sum-mdoc", "src/rmfbs_sum-mdoc")
cairosvg --format=pdf -o mdoc/rmfbs_sum-mdoc/svg/svg-01/process.pdf mdoc/rmfbs_sum-mdoc/svg/svg-01/process.svg
scons: *** [mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf] Source `mdoc/rmfbs_sum-mdoc/tex/rmfbs_sum.tex' not found, needed by target `mdoc/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf'.
scons: building terminated because of errors.

The second situation is when I call the pseudo-builder with a target argument
whose path is relative (i.e., not prefixed with a "#"):

cur_doc = env.FaustMDoc(dsp_name, dsp)

It behaves identically to the above. Also, using

cur_doc = env.FaustMDoc(os.sep.join(["#src", dsp_name]), dsp)

i.e., putting it in the same directory as the sources, behaves the same, too.

Now the third situation is especially odd and counter-intuitive. I call the
pseudo-builder *without* the target argument:

cur_doc = env.FaustMDoc(dsp)

The first time I call scons it behaves the same as before, but notice the paths:

$ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace
scons: Reading SConscript files ...

scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
scons: done reading SConscript files.
scons: Building targets ...
/usr/bin/faust -t 4800 -vec -o /dev/null -mdoc src/rmfbs_sum.dsp
Move("build-cpp-jack-qt/rmfbs_sum-mdoc", "src/rmfbs_sum-mdoc")
scons: *** [build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf] Source `src/rmfbs_sum-mdoc/svg/svg-01/process.svg' not found, needed by target `build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf'.
scons: internal stack trace:
File "/usr/lib64/python2.7/site-packages/SCons/Job.py", line 387, in start
task.prepare()
File "/usr/lib64/python2.7/site-packages/SCons/Script/Main.py", line 168, in prepare
return SCons.Taskmaster.OutOfDateTask.prepare(self)
File "/usr/lib64/python2.7/site-packages/SCons/Taskmaster.py", line 189, in prepare
executor.prepare()
File "/usr/lib64/python2.7/site-packages/SCons/Executor.py", line 392, in prepare
raise SCons.Errors.StopError(msg % (s, self.batches[0].targets[0]))
scons: building terminated because of errors.

Unlike the first two cases, the paths are wrong: the target path is in the
source directory, not the variant directory. Furthermore, the second call to
scons fails:

$ scons CXX=clang++ -j2 rmfbs_sum-mdoc --debug=stacktrace
scons: Reading SConscript files ...

scons: warning: QT4DIR variable is not defined, using moc executable as a hint (QT4DIR=/usr)
File "/home/marcec/.scons/site_scons/site_tools/qt4/__init__.py", line 384, in _detect
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_sum-mdoc/pdf/mbstereophonyd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonyd_syn-mdoc/pdf/mbstereophonyd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_sum-mdoc/pdf/mbstereophonys_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/mbstereophonys_syn-mdoc/pdf/mbstereophonys_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbd_sum-mdoc/pdf/rmfbd_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbd_syn-mdoc/pdf/rmfbd_syn.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbs_sum-mdoc/pdf/rmfbs_sum.pdf
/home/marcec/programming/faust_rm_filter_bank/src/rmfbs_syn-mdoc/pdf/rmfbs_syn.pdf
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf] Source `src/rmfbs_sum-mdoc/svg/svg-01/process.svg' not found, needed by target `build-cpp-jack-qt/rmfbs_sum-mdoc/svg/svg-01/process.pdf'.
scons: internal stack trace:
File "/usr/lib64/python2.7/site-packages/SCons/Job.py", line 387, in start
task.prepare()
File "/usr/lib64/python2.7/site-packages/SCons/Script/Main.py", line 168, in prepare
return SCons.Taskmaster.OutOfDateTask.prepare(self)
File "/usr/lib64/python2.7/site-packages/SCons/Taskmaster.py", line 189, in prepare
executor.prepare()
File "/usr/lib64/python2.7/site-packages/SCons/Executor.py", line 392, in prepare
raise SCons.Errors.StopError(msg % (s, self.batches[0].targets[0]))
scons: building terminated because of errors.

The reason for that is apparently the incorrect target paths, but there's also
another thing: the directory structure in the variant directory is wrong:

$ tree -d build-cpp-jack-qt
build-cpp-jack-qt
└── rmfbs_sum-mdoc # <- this directory shouldn't be here
└── rmfbs_sum-mdoc # <- should be a subdirectory of build-cpp-jack-qt
├── cpp
├── pdf
├── src
├── svg
│   └── svg-01
└── tex

This only ever happens in this particular scenario.

Finally, the same as the first two cases also happens when I call the
pseudo-builder directly in the top-level SConstruct: it fails on the first run
but completes correctly on the second.

Also as a note: just out of curiosity I added a "sleep 0.5s && ..." to the
cairosvg action string, but it didn't change anything (not that I expected it
to, but hey).

So I'm flummoxed :-/ . I don't *think* I'm doing anything wrong, but maybe I
am? Any help understanding what's going on here and maybe getting this working
is greatly appreciated :-) .

Greetings,
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://four.pairlist.net/pipermail/scons-users/attachments/20130607/417d2b28/attachment-0001.pgp


More information about the Scons-users mailing list