[Scons-users] Parallel builds sometimes fails

Fri May 25 18:05:33 EDT 2018

I wrote out a whole email in response before finding an old minimal case I had lying around. The bug in my main project seems to be a different one; changing the scanners worked in this example, but not in my project for some reason, but it’s still an example of how the scanner can act a little oddly at times.

Apparently scanners will still run even if you’re not using the corresponding builder.

My setup:
Root
+ SConstruct
+ src
| + HelloWorld.c
| | + GenerationRoot.h
| | +SomeHeader.h

HelloWorld.c includes HelloWorld.h
HelloWorld.h includes SomeOtherHeader.h

HelloWorld.h comes from GenerationRoot.h (I run a copy to simulate generation)
SomeOtherHeader.h comes from SomeHeader.h

SConstruct:

# environments
env = Environment(tools = ['mingw'])

# directories
src = Dir('src')
include = src.Dir('include')
build = Dir('build')
export = Dir('export')

env.Append(CPPPATH = build)
genv.Append(CPPPATH = build)

# targets
generateA = env.Command(target = build.File('SomeOtherHeader.h'), source = include.File('SomeHeader.h'),
                          action = Copy('$TARGET', '$SOURCE'))
genv.Alias('generateA', generateA)

generateB = env.Command(target = build.File('HelloWorld.h'), source = include.File('GenerationRoot.h'),
                         action = Copy('$TARGET', '$SOURCE'))
env.Alias('generateB', generateB)

Running generateB still tries to run generateA, even though it is not necessary. generateB is purely a copy command, and doesn’t depend on anything but the source, but running generateB with –tree=all is still revealing:

C:\ExampleRoot>scons generateB --tree=all
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
Copy("build\SomeOtherHeader.h", "src\include\SomeHeader.h")
Copy("build\HelloWorld.h", "src\include\GenerationRoot.h")
+-generateB
  +-build\HelloWorld.h
  | +-src\include\GenerationRoot.h
  | +-build\SomeOtherHeader.h
  |   +-src\include\SomeHeader.h
  +-build\SomeOtherHeader.h
    +-src\include\SomeHeader.h
scons: done building targets.

It scanned HelloWorld.h, even though it’s not building anything with it.

The solution I found was to replace the SCANNERS environment variable with an empty scanner:
emptyScanner = SCons.Scanner.Base(function = lambda *args, **kwargs: [],
                                   skeys = '.h')

And use this in a special generation environment, but I’m not sure if this risks scanning .h files and returning an empty list when they should have been scanned properly.

Moreover this solution didn’t even fix the problem in my actual project. All the bug does in my project is cause an unnecessary copy in specific conditions, so I haven’t bothered trying to do much more investigating, though.

Essentially what this shows is that the scanners in the global env[‘SCANNERS’] variable are a little too enthusiastic about scanning, and will scan things even when you don’t want them to. Moreover, code generation can cause scanners to act oddly if .h files are used both for generation and for building, because implicit dependencies are stored by the node rather than by the executor (I think that’s what it’s called?). This means that either the compile-type scanner adds dependencies that the generator doesn’t need, or the generator-type scanner can omit dependencies that the compiler needs.

Since I couldn’t figure out exactly what was going on in my main project, I thought maybe the problem Hua was having was related, but now I don’t think so.

-Matthew

From: Scons-users <scons-users-bounces at scons.org> On Behalf Of Bill Deegan
Sent: May 25, 2018 12:35
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] Parallel builds sometimes fails

Matthew,

It should know that foo.c -> foo.h, and that foo.h isn't ready yet (hasn't been rebuilt) and thus know foo.c's not ready.

-Bill

On Fri, May 25, 2018 at 11:42 AM, Matthew Marinets <Matthew.Marinets at kardium.com<mailto:Matthew.Marinets at kardium.com>> wrote:
I’m working on a project with a lot of code generation, and I have run into a simple and unavoidable problem that can only be solved by setting an explicit dependency on a header file.

If a generated header file includes another header file, SCons has no way of knowing. It doesn’t scan built files, because there is no guarantee that the file exists or is up to date at scan-time. Hence, imagine the simple example:

Foo.c:
#include “Foo.h”

Foo.h is generated, and when it is up to date, it includes:
                #include “Bar.h”

SCons has no way of knowing that Foo.c depends on Bar.h. The simplest solution is to set “Bar.h” as an explicit dependency.

-Matthew

From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> On Behalf Of Bill Deegan
Sent: May 25, 2018 11:30
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] Parallel builds sometimes fails

You should not have to explicitly tell SCons that a source file depends on a header...

On Fri, May 25, 2018 at 11:47 AM, Hua Yanghao <huayanghao at gmail.com<mailto:huayanghao at gmail.com>> wrote:
I am also using generated header files, which in turn is
unconditionally included by all C files.
The way I use is to first collect the entire C objects and mark an
explicit dependency: env.Depends(obj_list, depend_list).

However you need to make the scons build aware of the generated header
files, which in my case is generated by a customer scons builder.

And my builds worked fine with parallel builds.

On Fri, May 25, 2018 at 4:48 PM, mingqian Han <mq_han at hotmail.com<mailto:mq_han at hotmail.com>> wrote:
> Hi all,
>
>      I am trying to implement a building infrastructure based on Scons.
> Currently it works nicely when doing a non-parallel build (-j1)
>      However when I run a parallel build, sometimes it fails.
>      The build is mostly C source files, and some of them include a specific
> header file, for example, #include "myheader.h"
>       But myheader.h is a generated file from some configuration files. I
> have created a custom SCons builder, put under scons_tools, and it generates
> this file when Scons runs.
>       However in parallel build, sometimes some source files get to compile
> before myheader.h is generated, which results in error: myheader.h not
> found.
>       And sometimes, when my custom SCons builder gets running, it reports
> error, saying that it is unable to access this file because other process is
> using it. In such case, myheader.h is only partially generated.
>       I know SCons reads all the SConscripts first then decides the order of
> building itself. So is there any way to make the parallel build success?
>       Thanks all.
>
> Regards,
>
> Rafael
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org<mailto:Scons-users at scons.org>
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180525/e311b18a/attachment-0001.html>