[Scons-users] parallel invocation of SCons

[Damien]

We do this by having separate codebases for each build target, even 
though the codebase supports all the targets.  Each SCons is run in a 
separate process for each codebase.  We then have a final SCons pass 
that just does a set of env.Install('VariousVersionsOfBlahTarget') to 
collect everything together.  It sounds cumbersome, but it's very 
reliable, once you have the scripts set up it's straightforward.  Also, 
when one build breaks, only one is broken.

Damien

On 2/14/2019 9:57 AM, Bill Deegan wrote:
> Jason,
>
> Assuming the CI builds are building the same code, just changing the 
> sconsign file won't be sufficient.
> Especially if the builders are windows, with the joy of windows file 
> locking...
>
> Best way to solve this is:
> 1) fix the ci
> 2) cause SCons to wait until the other build is done before starting.
>
> On Thu, Feb 14, 2019 at 7:06 AM Jason Kenny <dragon512 at live.com 
> <mailto:dragon512 at live.com>> wrote:
>
>
>     Hi,
>
>     As I understand it, Scons will write most of the data at the end
>     of the run. There is a race as it will write out the whole file
>     with updated state from the current run. This leads to the race
>     issue you are seeing as the state of one run will replace the data
>     from a previous run, leading to bad states. I normally work around
>     this via one of two ways:
>
>     1) run scons with two targets and a -j[n] option. This case
>     normally solves the issue for me. it can however be hard to read
>     output, or from your CI point of view not break up the tasks as
>     you might like.
>     2) use the SConsignFile() in your different runs to use different
>     DB files so there is no conflicts. This case may allow you to work
>     around your issue as you can prevent the race condition on a
>     common file.
>
>     Information about the SConsignFile below.
>
>     Jason
>
>
>     |SConsignFile([file, dbm_module])| , |env.SConsignFile([file,
>     dbm_module])|
>
>         This tells |scons| to store all file signatures in the
>         specified database |file|. If the |file| name is omitted,
>         |.sconsign| is used by default. (The actual file name(s)
>         stored on disk may have an appropriated suffix appended by the
>         |dbm_module|.) If |file| is not an absolute path name, the
>         file is placed in the same directory as the top-level
>         |SConstruct| file.
>
>         If |file| is |None|, then |scons| will store file signatures
>         in a separate |.sconsign| file in each directory, not in one
>         global database file. (This was the default behavior prior to
>         SCons 0.96.91 and 0.97.)
>
>         The optional |dbm_module| argument can be used to specify
>         which Python database module The default is to use a custom
>         |SCons.dblite| module that uses pickled Python data
>         structures, and which works on all Python versions.
>
>         Examples:
>
>         # Explicitly stores signatures in ".sconsign.dblite"
>         # in the top-level SConstruct directory (the
>         # default behavior).
>         SConsignFile()
>
>         # Stores signatures in the file "etc/scons-signatures"
>         # relative to the top-level SConstruct directory.
>         SConsignFile("etc/scons-signatures")
>
>         # Stores signatures in the specified absolute file name.
>         SConsignFile("/home/me/SCons/signatures")
>
>         # Stores signatures in a separate .sconsign file
>         # in each directory.
>         SConsignFile(None)
>
>
>     ------------------------------------------------------------------------
>     *From:* Scons-users <scons-users-bounces at scons.org
>     <mailto:scons-users-bounces at scons.org>> on behalf of Herzog,
>     Tobias (CQSP) <tobias.herzog at carmeq.com
>     <mailto:tobias.herzog at carmeq.com>>
>     *Sent:* Thursday, February 14, 2019 3:14 AM
>     *To:* Scons-users at scons.org <mailto:Scons-users at scons.org>
>     *Subject:* [Scons-users] parallel invocation of SCons
>     Hi SCons users,
>
>     due to a parallelization in a CI build pipeline, I have the use
>     case, that SCons (possibly) is invoked concurrently in the same
>     tree (i.e. using database) but with different build targets. I
>     already noticed, that this can lead to a crashing SCons and/or
>     corrupted database.
>     For me it would be sufficient, if SCons just waits until the other
>     SCons process has finished. My Idea was, to acquire any kind of
>     inter process lock right in the SConstruct, that is active until
>     the process terminates. So my assumption here is, that database
>     access takes place after the reading of SConstruct/SConscript is done.
>     Is this correct? Will this solution be safe? Has anyone solved
>     this problem in another/better way?
>
>     Thanks an best regards,
>     Tobias
>     _______________________________________________
>     Scons-users mailing list
>     Scons-users at scons.org <mailto:Scons-users at scons.org>
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>     _______________________________________________
>     Scons-users mailing list
>     Scons-users at scons.org <mailto:Scons-users at scons.org>
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20190214/b63a904e/attachment.html>