[Scons-users] integrating scons with other (non-scons) projects

[Mats Wichmann]

In common with, I presume, many others, our project builds using scons,
but also builds on the work of other projects rather than reinventing
the wheel where not necessary - so we use external bits for, to name a
few, json, cbor, coap, tls etc. as well as using googletest for unit
testing support and so on.

Because of the multi-platform nature (including cross-compilation
capability in some combinations), we can't depend on native packaging as
we could for an all-Linux solution (for example).  In other words,
installing libcoap and libcoap-dev on the build host does not solve
anything.

Of course, these external projects aren't built using scons, which means
we have dependency issues - you have to fetch, unpack, and sometimes
subprocess call out to build (in other cases, our build points to the
unpacked files).  So when scons does the initial scan, some of the
things it's scanning for may not exist yet.  Libraries are not found by
library search methods, headers may not exist, etc.  In effect, these
parts are not part of scons's view of the world.

A lot of work has gone into bolting bits and pieces of Python code
together to work with this, and while it generally works, it's the
source of most of the ongoing hiccups in our build system.   As
discussed on irc yesterday, for example, one of the downloaders is found
to be broken when the host is Windows and the Python is 3.x.  Which we
didn't find before because nobody's testbed happened to be a Clean
Environment - those testing Py3 usage already had their downloaded
external projects in place, and the CI system uses another method to
pre-populate things which have to be downloaded from a cache so the
download cost isn't borne x15 for each and every change submitted to the
CI system... and then the actual fail was covered up by a try block in
our code which was there to catch a different kind of fail.

So, before launching into an even longer essay, what are the best
practices for dealing with this situation of integrating other projects
which are not under scons control?

Someone in the past obviously read this page:

https://github.com/SCons/scons/wiki/DownloadUnpack

because both of the scripts listed on that page in fact appear in our
repository; but only one external lib uses that downloader; several use
the unpacker, but it has been modified to create instead a construction
environment method (AddMethod), for reasons which the commit history
does nothing to make clear, and the devs in question are long gone.  Is
that approach overkill, or is it "the best way" - trying very hard to
make scons aware of everything?

Sorry, this still got longer than I planned :)

-- mats