[Scons-users] Problems with SConscript files using variant_dir and duplicate=False

Hans-Christian Wild hchr.wild at gmail.com
Fri May 22 16:29:31 EDT 2020 

Hi, to show off what I am actually after I have created a second example
showing a different and maybe easier to understand usecase. But the exposed
'problems' and the style to structure the build system using SConscript
files is exactly the same in both repos.

https://github.com/hchrwild/project01/tree/v0.1

However I the second usecase, I hope that it is much easier to understand
why this top-level SConscript file exists in the first place. The two
examples are closely related to each other: The scons01 project shows a
possible structure how 'mylib' is developed in a dedicated repo, together
with some examples for testing and unit tests  (not included in the demo)
etc.. . The second example shows mylib being used in a microcontroller
project.

The second example has a SConscruct at top-level, "composing" the build.
There are two secondary SConscript files, one knowing how a "bootloader" is
sonstructed, one building a "firmware". These two SConscript relate
conceptually 1:1 to the "example" SConstruct from the first example
project. There is a ternary SConscript file library/mylib/SConscript which
1:1 corresponds to the top-level SConscript of the first example.

I hope that this makes clearer why there is a top-level SConscript in the
first example in the first place: When the library is used (think of
library/mylib being a subtree/submodule copy of the first example, with
examples etc. being stripped/there but not used in any fashion). When
somebody uses the library he can immediately use it in the projects for it
comes preconfigured in a SConscript file, allowing it to be included
seamlessly in the project.

Back to the example: All three software partitions (bootloader0/1 and
firmware) use the same mylib, but compile it each in a different variant.
As example they just inject a global string via a cppdefine APP_NAME, which
is really the stupidest example I could come up with...

*Summary:*
/SConscructs calls /{bootloader,firmware}/SConscripts to generate some
executables.
/{bootloader,firmware}/SConscripts call /library/mylib/SConscript because
they all need their dedicated version of mylib (eg. different defines,
target architecture, cflags, ....).
All outputs artifacts are to be placed in /output-*
/library/mylib/SConscript  is there as convinience, hiding most of the
details how mylib is sconstructed.

*The example again should work out of the box, provided there is a gcc in
path:*

$ scons run-all
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o output-bl0\src\main.obj -c -Ilibrary\mylib\include
output-bl0\src\main.c
gcc -o output-bl0\mylib\src\AppName.obj -c -DAPP_NAME="Bootloader0"
-Ioutput-bl0\mylib\include -Ioutput-bl0\mylib\src\include
output-bl0\mylib\src\AppName.c
ar rc output-bl0\mylib\libmylib.a output-bl0\mylib\src\AppName.obj
gcc -o output-bl0\bootloader.exe output-bl0\src\main.obj
output-bl0\mylib\libmylib.a
gcc -o output-bl1\src\main.obj -c -Ilibrary\mylib\include
output-bl1\src\main.c
gcc -o output-bl1\mylib\src\AppName.obj -c -DAPP_NAME="Bootloader1"
-Ioutput-bl1\mylib\include -Ioutput-bl1\mylib\src\include
output-bl1\mylib\src\AppName.c
ar rc output-bl1\mylib\libmylib.a output-bl1\mylib\src\AppName.obj
gcc -o output-bl1\bootloader.exe output-bl1\src\main.obj
output-bl1\mylib\libmylib.a
gcc -o output-fw\src\main.obj -c -DAPP_NAME=Firmware
-Ilibrary\mylib\include output-fw\src\main.c
gcc -o output-fw\mylib\src\AppName.obj -c -DAPP_NAME=Firmware
-Ioutput-fw\mylib\include -Ioutput-fw\mylib\src\include
output-fw\mylib\src\AppName.c
ar rc output-fw\mylib\libmylib.a output-fw\mylib\src\AppName.obj
gcc -o output-fw\firmware.exe output-fw\src\main.obj
output-fw\mylib\libmylib.a
output-bl0\bootloader.exe
Hello, this is Bootloader0!
output-bl1\bootloader.exe
Hello, this is Bootloader1!
output-fw\firmware.exe
Hello, this is Firmware!
scons: done building targets.

Now how to selectively produce the problems:

*1) If you set duplicate=0 in /SConstruct in line 7 but remove srcnode() in
line 6 of /library/mylib/SConscript the build breaks.*

$ scons run-all
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o output-bl0\src\main.obj -c -Ioutput-bl0\mylib\include
bootloader\src\main.c
bootloader\src\main.c:1:10: fatal error: AppName.h: No such file or
directory
 #include <AppName.h>
          ^~~~~~~~~~~
compilation terminated.
scons: *** [output-bl0\src\main.obj] Error 1
scons: building terminated because of errors.

*2) If you set duplicate=0 in /SConstruct in line 7 but keep srcnode() in
line 6 of /library/mylib/SConscript the build succeeds, but the include
paths are duplicated: *

$ scons run-all
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o output-bl0\src\main.obj -c -Ilibrary\mylib\include
bootloader\src\main.c
gcc -o output-bl0\mylib\src\AppName.obj -c -DAPP_NAME="Bootloader0"
-Ioutput-bl0\mylib\include
-Ilibrary\mylib\include -Ibootloader\mylib\include
-Ioutput-bl0\mylib\src\include -Ilibrary\mylib\src\include
-Ibootloader\mylib\src\include library\mylib\src\AppName.c
ar rc output-bl0\mylib\libmylib.a output-bl0\mylib\src\AppName.obj
gcc -o output-bl0\bootloader.exe output-bl0\src\main.obj
output-bl0\mylib\libmylib.a
gcc -o output-bl1\src\main.obj -c -Ilibrary\mylib\include
bootloader\src\main.c
gcc -o output-bl1\mylib\src\AppName.obj -c -DAPP_NAME="Bootloader1"
-Ioutput-bl1\mylib\include
-Ilibrary\mylib\include -Ibootloader\mylib\include
-Ioutput-bl1\mylib\src\include -Ilibrary\mylib\src\include
-Ibootloader\mylib\src\include library\mylib\src\AppName.c
ar rc output-bl1\mylib\libmylib.a output-bl1\mylib\src\AppName.obj
gcc -o output-bl1\bootloader.exe output-bl1\src\main.obj
output-bl1\mylib\libmylib.a
gcc -o output-fw\src\main.obj -c -DAPP_NAME=Firmware
-Ilibrary\mylib\include firmware\src\main.c
gcc -o output-fw\mylib\src\AppName.obj -c -DAPP_NAME=Firmware
-Ioutput-fw\mylib\include
-Ilibrary\mylib\include -Ifirmware\mylib\include
-Ioutput-fw\mylib\src\include -Ilibrary\mylib\src\include
-Ifirmware\mylib\src\include library\mylib\src\AppName.c
ar rc output-fw\mylib\libmylib.a output-fw\mylib\src\AppName.obj
gcc -o output-fw\firmware.exe output-fw\src\main.obj
output-fw\mylib\libmylib.a
output-bl0\bootloader.exe
Hello, this is Bootloader0!
output-bl1\bootloader.exe
Hello, this is Bootloader1!
output-fw\firmware.exe
Hello, this is Firmware!
scons: done building targets

*3) (This one is new) If you exchange all occurrences (6 total) of
`output-` in /SConstruct with `output/`, example `output-bl0` becomes
`output/bl0`, scons fails with a stacktrace:*

/w/Projects/myproject2
$ scons run-all
scons: Reading SConscript files ...

scons: warning: Calling missing SConscript without error is deprecated.
Transition by adding must_exist=0 to SConscript calls.
Missing SConscript 'output\fw\mylib\SConscript'
File "W:\Projects\myproject2\output\fw\SConscript", line 5, in <module>
TypeError: cannot unpack non-iterable NoneType object:
  File "W:\Projects\myproject2\SConstruct", line 10:
    firmware = SConscript('firmware/SConscript', exports=exports,
variant_dir='output/fw/', duplicate=duplicate)
  File
"w:\projects\myproject2\.venv\lib\site-packages\scons\SCons\Script\SConscript.py",
line 660:
    return method(*args, **kw)
  File
"w:\projects\myproject2\.venv\lib\site-packages\scons\SCons\Script\SConscript.py",
line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File
"w:\projects\myproject2\.venv\lib\site-packages\scons\SCons\Script\SConscript.py",
line 286:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "W:\Projects\myproject2\output\fw\SConscript", line 5:
    mylib_a, mylib_h = env.SConscript('../library/mylib/SConscript',
exports={'env': mylib_env}, variant_dir='mylib', duplicate=duplicate)

Regards

On Fri, May 22, 2020 at 6:27 PM Bill Deegan <bill at baddogconsulting.com>
wrote:

> If you have to go that route.. I can pretty much guarantee you've done
> something wrong.
> I've cloned your repo and I'll take a look at it this weekend.
> Pretty sure it's the combo of SConscript variant dir and partial separate
> VariantDir() for other dirs which is causing an issue.
>
> On Fri, May 22, 2020 at 4:20 AM Hans-Christian Wild <hchr.wild at gmail.com>
> wrote:
>
>> OK, thank you for the clarification!
>>
>> On Fri, May 22, 2020 at 12:32 AM Daniel Moody <dmoody256 at gmail.com>
>> wrote:
>>
>>> Ah my mistake, I ran scons first with duplication on, then modified the
>>> SConstruct to disable it and ran scons -c, which scons then did not cleaned
>>> the duplicated files because of my change, so they were still there when I
>>> built again. So that is not working for me either.
>>>
>>> On Thu, May 21, 2020 at 4:51 PM Hans-Christian Wild <hchr.wild at gmail.com>
>>> wrote:
>>>
>>>> Hi Daniel,
>>>>
>>>> thank you for taking the time to answer! Unfortunately, there are still
>>>> problems.
>>>>
>>>> Firstly, your suggestion does not work for me?! I am puzzled,,
>>>>
>>>> When I just change the following line, the output still reads:
>>>>
>>>> mylib_a, mylib_h = SConscript('../../SConscript', exports='env',
>>>> variant_dir='../../target/mylib', duplicate=False)
>>>>
>>>> /w/Projects/mylib/example/mingw
>>>> $ scons
>>>> scons: Reading SConscript files ...
>>>> scons: done reading SConscript files.
>>>> scons: Building targets ...
>>>> scons: building associated VariantDir targets: .
>>>> gcc -o target\src\main.obj -c -IW:\Projects\mylib\target\mylib\include
>>>> target\src\main.c
>>>> target\src\main.c:4:10: fatal error: Types.h: No such file or directory
>>>>  #include <Types.h>
>>>>           ^~~~~~~~~
>>>> compilation terminated.
>>>> scons: *** [target\src\main.obj] Error 1
>>>> scons: building terminated because of errors.
>>>>
>>>> So only the effect was that SCons has decided to put a absolute path to
>>>> the wrong location, instead of the relative one.
>>>>
>>>> Secondly, I am aware of that sentence from the docs and this is what I
>>>> would like! I need X examples building X different version of mylib.
>>>>
>>>> Also, I assumed that when I do a (in /SConscript)
>>>>
>>>> headers = env.Dir('include/')
>>>>
>>>> by the docs:
>>>>
>>>> Dir(name, [directory]) , env.Dir(name, [directory])
>>>> [...] If no directory is specified, the current script's directory is
>>>> used as the parent.
>>>>
>>>> .. that this would mean that scons remembers a relative path and the
>>>> current SConscript file as parent and is able to compute and abs or
>>>> relative path to if from where ever scons is called our what variant dirs
>>>> you specify.
>>>>
>>>> SCons does so correctly when you have duplication on, if not, it
>>>> doesn't.
>>>>
>>>> -hans
>>>>
>>>> On Thu, May 21, 2020 at 10:22 PM Daniel Moody <dmoody256 at gmail.com>
>>>> wrote:
>>>>
>>>>> Also should note, I am not saying its "not a bug", just what worked
>>>>> when I was messing with your example. It seems like its "not a bug" and if
>>>>> you really want your variant dir in the location of the SConstruct you
>>>>> should use duplicated tree. So probably more of a feature request if you
>>>>> were to submit an issue.
>>>>>
>>>>> On Thu, May 21, 2020 at 2:12 PM Daniel Moody <dmoody256 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> the docs say:
>>>>>> https://scons.org/doc/3.1.2/HTML/scons-man.html#f-SConscript
>>>>>> "The variant_dir argument is interpreted relative to the directory
>>>>>> of the calling SConscript file."
>>>>>>
>>>>>> So in the case the SConstruct is the "calling SConscript" and you are
>>>>>> saying the variant dir for the SConscript should be target/mylib, but its
>>>>>> not, its ../../target/mylib relative to the calling SConstruct.
>>>>>>
>>>>>> So you need to call it like this:
>>>>>> mylib_a, mylib_h = SConscript('../../SConscript', exports='env',
>>>>>> variant_dir='../../target/mylib', duplicate=False)
>>>>>>
>>>>>> For a non duplicated variant dir, the variant dir will be
>>>>>> substituted with the source dir when resolving paths for the build, which
>>>>>> means substituting "target/mylib/include" to "./include" where
>>>>>> 'target/mylib' is the variant dir and '.' is the source dir, will not work
>>>>>> from the calling SConstructs location. It should be substituted with
>>>>>> "../../target/mylib" so the variant dir resolved to the correct location.
>>>>>>
>>>>>> On Thu, May 21, 2020 at 12:54 PM Hans-Christian Wild <
>>>>>> hchr.wild at gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have two questions about SCons, I have uploaded a *working*
>>>>>>> example as base where SCons does everything nearly as I want it. The
>>>>>>> problems start to occur only when we do slight modifications to this
>>>>>>> example.
>>>>>>>
>>>>>>> https://github.com/hchrwild/scons01/tree/master
>>>>>>>
>>>>>>> In the project we have a main SConstruct file in `/example/mingw/`
>>>>>>> which imports a library SConscript file from `/`. The SConscript file
>>>>>>> creates its little library for the given environment and returns it
>>>>>>> together with its public header include path. The SConstruct file in turn
>>>>>>> builds an executable linking the static library and using the public header
>>>>>>> include path.
>>>>>>>
>>>>>>> Output:
>>>>>>>
>>>>>>> scons: Reading SConscript files ...
>>>>>>> scons: done reading SConscript files.
>>>>>>> scons: Building targets ...
>>>>>>> scons: building associated VariantDir targets:
>>>>>>> target\mylib\example\mingw
>>>>>>> gcc -o target\src\main.obj -c -Itarget\mylib\include
>>>>>>> target\src\main.c
>>>>>>> gcc -o target\mylib\src\Assert.obj -c -Itarget\mylib\include
>>>>>>> -Itarget\mylib\src\include target\mylib\src\Assert.c
>>>>>>> ar rc target\mylib\libmylib.a target\mylib\src\Assert.obj
>>>>>>> gcc -o target\mingw-example.exe target\src\main.obj
>>>>>>> target\mylib\libmylib.a
>>>>>>> scons: done building targets.
>>>>>>>
>>>>>>> *Perfect!*
>>>>>>> (Although I don't understand why it prints `building associated
>>>>>>> VariantDir targets: *target\mylib\example\mingw*` which to me does
>>>>>>> not make sense, IMO it should be *`example/mingw/target/mylib`* if
>>>>>>> anything, but let's put that aside..)
>>>>>>>
>>>>>>> *Now the actual problems:*
>>>>>>>
>>>>>>> 1) I don't like that SCons duplicates everything. What I really want
>>>>>>> is an out-of-source variant build. When I set `duplicate=False`, the build
>>>>>>> fails, however:
>>>>>>>
>>>>>>> mylib_a, mylib_h = SConscript('../../SConscript', exports='env',
>>>>>>> variant_dir='target/mylib', duplicate=False)
>>>>>>>
>>>>>>> scons: Reading SConscript files ...
>>>>>>> scons: done reading SConscript files.
>>>>>>> scons: Building targets ...
>>>>>>> scons: building associated VariantDir targets: .
>>>>>>> gcc -o target\src\main.obj -c *-Itarget\mylib\include*
>>>>>>> target\src\main.c
>>>>>>> target\src\main.c:4:10: fatal error: Types.h: No such file or
>>>>>>> directory
>>>>>>>  #include <Types.h>
>>>>>>>           ^~~~~~~~~
>>>>>>> compilation terminated.
>>>>>>> scons: *** [target\src\main.obj] Error 1
>>>>>>> scons: building terminated because of errors.
>>>>>>>
>>>>>>> As marked in above output, the include path is not updated. The
>>>>>>> SConscript file is at `../../` relative to SConstruct, and the include
>>>>>>> folder is ./include relative to SConscript. Therefore, without duplication,
>>>>>>> the correct include path relative to SConstruct would be `../../include` ,
>>>>>>> which SCons does not seem to compute however. Is this expected or a missing
>>>>>>> feature/bug? I really like the idea, that the SConstruct file really only
>>>>>>> needs to know where the SConscript file is located and having everything
>>>>>>> else being figured out by the build system...
>>>>>>>
>>>>>>> 2) Let's put that aside and keep above change, but hardcode the
>>>>>>> expected include path in SConstruct file:
>>>>>>>
>>>>>>> program = env.Program('target/mingw-example', sources, CPPPATH=[
>>>>>>> '../../include'], LIBS=[mylib_a])
>>>>>>>
>>>>>>> scons: Reading SConscript files ...
>>>>>>> scons: done reading SConscript files.
>>>>>>> scons: Building targets ...
>>>>>>> scons: building associated VariantDir targets: .
>>>>>>> gcc -o target\src\main.obj -c -IW:\Projects\mylib\include
>>>>>>> target\src\main.c
>>>>>>> gcc -o target\mylib\src\Assert.obj -c *-Itarget\mylib\include
>>>>>>> -IW:\Projects\mylib\include -Itarget\mylib\src\include
>>>>>>> -IW:\Projects\mylib\src\include* W:\Projects\mylib\src\Assert.c
>>>>>>> ar rc target\mylib\libmylib.a target\mylib\src\Assert.obj
>>>>>>> gcc -o target\mingw-example.exe target\src\main.obj
>>>>>>> target\mylib\libmylib.a
>>>>>>> scons: done building targets.
>>>>>>>
>>>>>>> Now the compilation succeeds. However, scons has now duplicated the
>>>>>>> include paths for the static library! One time with the invalid
>>>>>>> variant-with-duplication path, and one time with the correct (absolute
>>>>>>> path). Why does it do that? Is this a bug?
>>>>>>>
>>>>>>> Sorry for the long post. Any feedback appreciated!
>>>>>>>
>>>>>>> -hans
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20200522/dc1ce61e/attachment-0001.html>

More information about the Scons-users mailing list