[Scons-users] Parallel builds sometimes fails

[Bill Deegan]

Rafael,

How is the header created? Do you have a env.Command()?

As I've stated previously, if you need to use env.Depends() most likely
you're missing something in your build logic.
(SCons is unaware that some builder/command is actually generating that
file and thus starting build steps which require it before it's available)

On Mon, May 28, 2018 at 12:02 AM, mingqian Han <mq_han at hotmail.com> wrote:

> Hi everyone,
>
>      Thanks for your help, I finally figure out that I need a correct
> Depends.
>      Previously I was setting Depends on a Library builder, for example:
>      a = env.Library(...)
>      env.Depends(a, ...)
>
>      However that still does not ensure the custom header generator to run
> first.
>      Now I set the Depends on object file list, and it works.
>      Thanks!
>
> Best,
> Rafael
> ------------------------------
> *From:* Scons-users <scons-users-bounces at scons.org> on behalf of Matthew
> Marinets <Matthew.Marinets at Kardium.com>
> *Sent:* Saturday, May 26, 2018 6:05 AM
> *To:* SCons users mailing list
> *Subject:* Re: [Scons-users] Parallel builds sometimes fails
>
>
> I wrote out a whole email in response before finding an old minimal case I
> had lying around. The bug in my main project seems to be a different one;
> changing the scanners worked in this example, but not in my project for
> some reason, but it’s still an example of how the scanner can act a little
> oddly at times.
>
>
>
> Apparently scanners will still run even if you’re not using the
> corresponding builder.
>
>
>
> My setup:
>
> Root
>
> + SConstruct
>
> + src
>
> | + HelloWorld.c
>
> | | + GenerationRoot.h
>
> | | +SomeHeader.h
>
>
>
> HelloWorld.c includes HelloWorld.h
>
> HelloWorld.h includes SomeOtherHeader.h
>
>
>
> HelloWorld.h comes from GenerationRoot.h (I run a copy to simulate
> generation)
>
> SomeOtherHeader.h comes from SomeHeader.h
>
>
>
> SConstruct:
>
>
>
> # environments
>
> env = Environment(tools = ['mingw'])
>
>
>
> # directories
>
> src = Dir('src')
>
> include = src.Dir('include')
>
> build = Dir('build')
>
> export = Dir('export')
>
>
>
> env.Append(CPPPATH = build)
>
> genv.Append(CPPPATH = build)
>
>
>
> # targets
>
> generateA = env.Command(target = build.File('SomeOtherHeader.h'), source
> = include.File('SomeHeader.h'),
>
>                           action = Copy('$TARGET', '$SOURCE'))
>
> genv.Alias('generateA', generateA)
>
>
>
> generateB = env.Command(target = build.File('HelloWorld.h'), source =
> include.File('GenerationRoot.h'),
>
>                          action = Copy('$TARGET', '$SOURCE'))
>
> env.Alias('generateB', generateB)
>
>
>
> Running generateB still tries to run generateA, even though it is not
> necessary. generateB is purely a copy command, and doesn’t depend on
> anything but the source, but running generateB with –tree=all is still
> revealing:
>
>
>
> C:\ExampleRoot>scons generateB --tree=all
>
> scons: Reading SConscript files ...
>
> scons: done reading SConscript files.
>
> scons: Building targets ...
>
> Copy("build\SomeOtherHeader.h", "src\include\SomeHeader.h")
>
> Copy("build\HelloWorld.h", "src\include\GenerationRoot.h")
>
> +-generateB
>
>   +-build\HelloWorld.h
>
>   | +-src\include\GenerationRoot.h
>
>   | +-build\SomeOtherHeader.h
>
>   |   +-src\include\SomeHeader.h
>
>   +-build\SomeOtherHeader.h
>
>     +-src\include\SomeHeader.h
>
> scons: done building targets.
>
>
>
> It scanned HelloWorld.h, even though it’s not building anything with it.
>
>
>
> The solution I found was to replace the SCANNERS environment variable with
> an empty scanner:
>
> emptyScanner = SCons.Scanner.Base(function = lambda *args, **kwargs: [],
>
>                                    skeys = '.h')
>
>
>
> And use this in a special generation environment, but I’m not sure if this
> risks scanning .h files and returning an empty list when they should have
> been scanned properly.
>
>
>
> Moreover this solution didn’t even fix the problem in my actual project.
> All the bug does in my project is cause an unnecessary copy in specific
> conditions, so I haven’t bothered trying to do much more investigating,
> though.
>
>
>
> Essentially what this shows is that the scanners in the global
> env[‘SCANNERS’] variable are a little too enthusiastic about scanning, and
> will scan things even when you don’t want them to. Moreover, code
> generation can cause scanners to act oddly if .h files are used both for
> generation and for building, because implicit dependencies are stored by
> the node rather than by the executor (I think that’s what it’s called?).
> This means that either the compile-type scanner adds dependencies that the
> generator doesn’t need, or the generator-type scanner can omit dependencies
> that the compiler needs.
>
>
>
> Since I couldn’t figure out exactly what was going on in my main project,
> I thought maybe the problem Hua was having was related, but now I don’t
> think so.
>
>
>
> -Matthew
>
>
>
>
>
> *From:* Scons-users <scons-users-bounces at scons.org> *On Behalf Of *Bill
> Deegan
> *Sent:* May 25, 2018 12:35
> *To:* SCons users mailing list <scons-users at scons.org>
> *Subject:* Re: [Scons-users] Parallel builds sometimes fails
>
>
>
> Matthew,
>
>
>
> It should know that foo.c -> foo.h, and that foo.h isn't ready yet (hasn't
> been rebuilt) and thus know foo.c's not ready.
>
>
>
> -Bill
>
>
>
> On Fri, May 25, 2018 at 11:42 AM, Matthew Marinets <
> Matthew.Marinets at kardium.com> wrote:
>
> I’m working on a project with a lot of code generation, and I have run
> into a simple and unavoidable problem that can only be solved by setting an
> explicit dependency on a header file.
>
>
>
> If a generated header file includes another header file, SCons has no way
> of knowing. It doesn’t scan built files, because there is no guarantee that
> the file exists or is up to date at scan-time. Hence, imagine the simple
> example:
>
>
>
> Foo.c:
>
> #include “Foo.h”
>
>
>
> Foo.h is generated, and when it is up to date, it includes:
>
>                 #include “Bar.h”
>
>
>
> SCons has no way of knowing that Foo.c depends on Bar.h. The simplest
> solution is to set “Bar.h” as an explicit dependency.
>
>
>
> -Matthew
>
>
>
> *From:* Scons-users <scons-users-bounces at scons.org> *On Behalf Of *Bill
> Deegan
> *Sent:* May 25, 2018 11:30
> *To:* SCons users mailing list <scons-users at scons.org>
> *Subject:* Re: [Scons-users] Parallel builds sometimes fails
>
>
>
> You should not have to explicitly tell SCons that a source file depends on
> a header...
>
>
>
> On Fri, May 25, 2018 at 11:47 AM, Hua Yanghao <huayanghao at gmail.com>
> wrote:
>
> I am also using generated header files, which in turn is
> unconditionally included by all C files.
> The way I use is to first collect the entire C objects and mark an
> explicit dependency: env.Depends(obj_list, depend_list).
>
> However you need to make the scons build aware of the generated header
> files, which in my case is generated by a customer scons builder.
>
> And my builds worked fine with parallel builds.
>
>
> On Fri, May 25, 2018 at 4:48 PM, mingqian Han <mq_han at hotmail.com> wrote:
> > Hi all,
> >
> >      I am trying to implement a building infrastructure based on Scons.
> > Currently it works nicely when doing a non-parallel build (-j1)
> >      However when I run a parallel build, sometimes it fails.
> >      The build is mostly C source files, and some of them include a
> specific
> > header file, for example, #include "myheader.h"
> >       But myheader.h is a generated file from some configuration files. I
> > have created a custom SCons builder, put under scons_tools, and it
> generates
> > this file when Scons runs.
> >       However in parallel build, sometimes some source files get to
> compile
> > before myheader.h is generated, which results in error: myheader.h not
> > found.
> >       And sometimes, when my custom SCons builder gets running, it
> reports
> > error, saying that it is unable to access this file because other
> process is
> > using it. In such case, myheader.h is only partially generated.
> >       I know SCons reads all the SConscripts first then decides the
> order of
> > building itself. So is there any way to make the parallel build success?
> >       Thanks all.
> >
> > Regards,
> >
> > Rafael
> >
>
> > _______________________________________________
> > Scons-users mailing list
> > Scons-users at scons.org
> > https://pairlist4.pair.net/mailman/listinfo/scons-users
> >
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180528/df420c33/attachment-0001.html>