[Scons-users] Provide default Import to SConscript
Bill Deegan
bill at baddogconsulting.com
Mon Feb 19 10:42:03 EST 2018
Spencer,
I worked on a similar build system a while back for a client.
My solution was SConstruct did the bulk of the work and no SConscripts were
ever created.
The source stayed the clients home made file because they weren't willing
to change.
The logic walked the files figured out which builders needed to be called
and which environment() variables needed to be set for each .ini file and
then called them. It was actually fairly simple to implement and maintain.
-Bill
On Sun, Feb 18, 2018 at 11:08 PM, Spencer Yost <syost at triad.rr.com> wrote:
> Right, our SConstruct walks the target, looking for the ini files. If it
> finds one, it passes them to the SConscript builder. The builder obviously
> creates an SConscript if the the ini file has changed. We then pass this
> SConscript to the SConscript() call.
>
> Again, our original build configuration files are sort of approximation an
> ini file - but not exact. The builder is actually several hundred lines of
> Python code. Because they are not exactly what the configparser module
> needs we have to create a temporary file that is the result of massaging
> and correcting the original and parse that temp file.
>
> PS: Don't get me wrong, we don't have this nailed down exactly yet. There
> are some deliverables what we are still working on. For example our config
> files deliver individual files that are contained within tar files. This
> seems to be opposite of what the Scons tarfile builder does(builds tar
> files from a collection of files). We have not cracked nut not yet.
>
> Spencer Yost
>
> On Feb 18, 2018, at 9:54 AM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
> Spencer,
>
> So your build info source are these .ini files?
> And your SConscript files just process them?
>
> -Bill
>
> On Sat, Feb 17, 2018 at 10:54 PM, Spencer Yost <syost at triad.rr.com> wrote:
>
>> Just as an offhand comment, that is sort of related to this. First, I
>> don't have to import a lot.
>>
>> We just converted our build system to SCons(~1600+ packages, ~15000
>> source files and tens of millions of lines of code). Because python
>> expertise ranges from completely lacking to extremely poor :-) in our c/c++
>> staff; the powers that be indicated that developers will not maintain
>> SConscript files. They will maintain the original build configuration files
>> that we had(something that approximates a win.ini file). That means I have
>> to rebuild SConscript files for every original configuration file we had.
>> Trying to figure out a way to programmatically consolidate them to minimize
>> the number of SConscript files was nearly impossible. So we have about 1600
>> SConscript files. Stuff like library dependencies vary between
>> "packages" very frequently too. In addition the staff has to be able to
>> build an individual "package" or the whole tree. I have a builder that
>> creates the SConscript file whenever this original build configuration file
>> changes. So it's not a big deal. But consolidating them was just way too
>> big of a fish to fry in the time and budget allowed.
>>
>> Sometimes you just have to have a bunch of them.
>>
>> PS: SCons is working great for us. Dependencies are much more robust,
>> speed is marginally better and the entire arc from source code change to
>> Debian package creation is flawless. Being able to create command line
>> options has been a godsend for us. Previously we would have to create
>> build macros to emulate what command line options do.
>>
>> Spencer Yost
>>
>> On Feb 17, 2018, at 6:17 PM, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>> Hua,
>>
>> From the tree you've shared, I'm going to guess you have way more
>> SConscripts than you need.
>>
>> For example, you're set of test directories could have a single
>> SConscript instead of one per.
>> Which no doubt are just a few lines each.
>>
>> In some cases it looks like you have a SConscript in a directory which
>> only has a header file (or none) and then a directory (or more) of source
>> code. Skip the SConscript in the basically no-op directorie(s).
>>
>> Another suggestion, use variant dirs so you're not creating build files
>> in your source directory.
>>
>> If you find yourself with a ton of very small SConscripts, it's a sign
>> that you may be doing something non-scons'ian.
>>
>> Of course it also depends on how working on each group of code is split
>> up between different developers.
>>
>> There is some art in structuring your build system (and your code) such
>> that engineers rarely end up causing merge conflicts with each other.
>>
>> I can see some value in default_import, but it would be something
>> requiring some debate on getting accepted.
>> As far as I can recall, your's is the first request for something like
>> this in the history of SCons (or at least as long as I've been involved).
>>
>> Philosophically SCons is about being explicit in the build requirements
>> and also about being correct by design. Yielding reproducible builds
>> regardless of the individual developers shell and machine setup.
>>
>>
>> On Sat, Feb 17, 2018 at 5:31 PM, Hua Yanghao <huayanghao at gmail.com>
>> wrote:
>>
>>> Hi Bill,
>>> I do not create a SConscript for every directory, only for directly
>>> which either provide a set of include file or having source code to
>>> build.
>>> I will try to create a minimal version of the build system to show the
>>> idea.
>>>
>>> Not sure if the below tree will give you some idea what I am trying to
>>> implement, at least you should be able to see where I am having a
>>> SConscript file.
>>> build/configs/riscv_example/riscv32-qemu-virt-main/
>>> ├── cmdlist.txt
>>> ├── config
>>> │ └── config.h
>>> ├── firmware
>>> │ ├── arch
>>> │ │ ├── include
>>> │ │ │ └── arch
>>> │ │ │ ├── cache.h
>>> │ │ │ ├── irq.h
>>> │ │ │ ├── misc.h
>>> │ │ │ ├── mmu.h
>>> │ │ │ ├── smp.h
>>> │ │ │ ├── spinlock.h
>>> │ │ │ └── timer.h
>>> │ │ ├── riscv
>>> │ │ │ ├── helper.c
>>> │ │ │ ├── helper.o
>>> │ │ │ ├── libdefault.o
>>> │ │ │ ├── main.c
>>> │ │ │ ├── main.o
>>> │ │ │ ├── SConscript
>>> │ │ │ ├── start.o
>>> │ │ │ └── start.S
>>> │ │ ├── SConscript
>>> │ │ ├── weak.c
>>> │ │ └── weak.o
>>> │ ├── boards
>>> │ │ ├── include
>>> │ │ │ └── board.h
>>> │ │ ├── qemu-riscv32-virt
>>> │ │ │ ├── board.c
>>> │ │ │ ├── board.o
>>> │ │ │ ├── SConscript
>>> │ │ │ ├── uart16550_device.c
>>> │ │ │ └── uart16550_device.o
>>> │ │ └── SConscript
>>> │ ├── common
>>> │ │ ├── cpu.c
>>> │ │ ├── cpu.o
>>> │ │ ├── include
>>> │ │ │ └── common
>>> │ │ │ ├── common.h
>>> │ │ │ ├── cpu.h
>>> │ │ │ └── log.h
>>> │ │ ├── libdefault.o
>>> │ │ ├── log.c
>>> │ │ ├── log.o
>>> │ │ ├── main.c
>>> │ │ ├── main.o
>>> │ │ ├── SConscript
>>> │ │ ├── weak.c
>>> │ │ └── weak.o
>>> │ ├── drivers
>>> │ │ └── uart16550
>>> │ │ ├── fd_ops.c
>>> │ │ ├── fd_ops.o
>>> │ │ ├── include
>>> │ │ │ ├── uart16550_api.h
>>> │ │ │ └── uart16550_device.h
>>> │ │ ├── libdefault.o
>>> │ │ ├── SConscript
>>> │ │ ├── uart16550.c
>>> │ │ └── uart16550.o
>>> │ ├── include
>>> │ │ ├── bootstage.h
>>> │ │ ├── compiler.h
>>> │ │ ├── debug.h
>>> │ │ └── linux
>>> │ │ └── typecheck.h
>>> │ ├── lib
>>> │ │ ├── cmd
>>> │ │ │ ├── cmd.c
>>> │ │ │ ├── cmd.o
>>> │ │ │ ├── include
>>> │ │ │ │ └── cmd
>>> │ │ │ │ └── cmd.h
>>> │ │ │ ├── SConscript
>>> │ │ │ ├── test.c
>>> │ │ │ └── test.o
>>> │ │ ├── fio
>>> │ │ │ ├── cmd.c
>>> │ │ │ ├── cmd.o
>>> │ │ │ ├── include
>>> │ │ │ │ └── usw_device_public.h
>>> │ │ │ ├── libdefault.o
>>> │ │ │ ├── newlib.c
>>> │ │ │ ├── newlib.o
>>> │ │ │ ├── SConscript
>>> │ │ │ ├── usw_device.c
>>> │ │ │ ├── usw_device.o
>>> │ │ │ └── usw_device_private.h
>>> │ │ ├── pipe
>>> │ │ │ ├── include
>>> │ │ │ │ └── pipe
>>> │ │ │ │ ├── pipe.h
>>> │ │ │ │ └── status.h
>>> │ │ │ ├── libdefault.o
>>> │ │ │ ├── pipe.c
>>> │ │ │ ├── pipe.o
>>> │ │ │ └── SConscript
>>> │ │ └── simple_console
>>> │ │ ├── include
>>> │ │ │ └── simple_console.h
>>> │ │ ├── libdefault.o
>>> │ │ ├── print_color.h
>>> │ │ ├── SConscript
>>> │ │ ├── simple_console.c
>>> │ │ └── simple_console.o
>>> │ ├── SConscript
>>> │ └── test
>>> │ ├── common
>>> │ │ ├── SConscript
>>> │ │ ├── test.c
>>> │ │ └── test.o
>>> │ ├── console
>>> │ │ ├── hello.c
>>> │ │ ├── hello.o
>>> │ │ ├── SConscript
>>> │ │ ├── test.c
>>> │ │ └── test.o
>>> │ ├── example
>>> │ │ ├── SConscript
>>> │ │ ├── test.c
>>> │ │ └── test.o
>>> │ ├── libc
>>> │ │ ├── SConscript
>>> │ │ ├── test.c
>>> │ │ └── test.o
>>> │ ├── linker
>>> │ │ ├── SConscript
>>> │ │ ├── test.c
>>> │ │ └── test.o
>>> │ ├── log
>>> │ │ ├── SConscript
>>> │ │ ├── test.c
>>> │ │ └── test.o
>>> │ └── setjmp
>>> │ ├── SConscript
>>> │ ├── test2.c
>>> │ ├── test2.o
>>> │ ├── test.c
>>> │ └── test.o
>>> ├── libriscv_example.a
>>> ├── link.ld
>>> ├── riscv_example.bin
>>> ├── riscv_example.elf
>>> ├── riscv_example.ibi_bin
>>> ├── riscv_example.map
>>> └── riscv_example.S
>>>
>>> 36 directories, 113 files
>>>
>>>
>>>
>>>
>>> On Sat, Feb 17, 2018 at 10:15 PM, Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>> > Likely you don't need a SConscript for every directory in your case?
>>> > Are you wrapping a complete linux build tree?
>>> >
>>> > Is this a build tree you can share for us to look at?
>>> >
>>> > -Bill
>>> >
>>> > On Sat, Feb 17, 2018 at 3:30 PM, Hua Yanghao <huayanghao at gmail.com>
>>> wrote:
>>> >>
>>> >> I see everything passed into env.SConscript(default_import=some_obj),
>>> >> if some_obj is a string it can pass through but if some_obj is a
>>> >> function object it end up as None in Script/SConscript.py/SConscrip
>>> t()
>>> >> function.
>>> >>
>>> >> I could have used a string and then import the right function however
>>> >> my function to be passed is a dynamically generated partial function
>>> >> ...
>>> >>
>>> >> Could anyone please give me some hint how could this be best
>>> implemented?
>>> >>
>>> >> Best Regards,
>>> >> Yanghao
>>> >>
>>> >> On Sat, Feb 17, 2018 at 7:53 PM, Hua Yanghao <huayanghao at gmail.com>
>>> wrote:
>>> >> > @Bill, I have managed to hack the Frame class and the
>>> >> > BuildDefaultGlobals() function to have some thing imported into
>>> >> > SConscript by default.
>>> >> > However I don't quite understand when the "default_import" keyword
>>> >> > argument passed down, somewhere somehow it end up as a set.
>>> >> >
>>> >> > Any documentation to explain what is going on from Base class into
>>> >> > SConscript() calls regarding parameter handling?
>>> >> >
>>> >> > Thanks,
>>> >> > Yanghao
>>> >> >
>>> >> > On Sat, Feb 17, 2018 at 7:45 PM, Hua Yanghao <huayanghao at gmail.com>
>>> >> > wrote:
>>> >> >> It varies a lot, ranging from 0 (yes, no source files at all) to a
>>> few
>>> >> >> dozen (didn't really count, see statistics below).
>>> >> >>
>>> >> >> Here is an example Linux kernel driver sub-module Makefile:
>>> >> >>
>>> >> >> linux/drivers/sn/Makefile:
>>> >> >> 1 #
>>> >> >> 2 # Makefile for the Altix device drivers.
>>> >> >> 3 #
>>> >> >> 4 #
>>> >> >> 5
>>> >> >> 6 obj-$(CONFIG_SGI_IOC3) += ioc3.o
>>> >> >>
>>> >> >> I want to replicate this kind of simplicity in scons.
>>> >> >>
>>> >> >> I actually have implemented almost the same thing in scons, but all
>>> >> >> those SConscript files have to start with Import("my_func") ...
>>> which
>>> >> >> is pretty annoying. I know I need it in every single SConscript,
>>> and I
>>> >> >> have to repeat it ... and this is so anti-pattern.
>>> >> >> I have get rid of the Return() part already, but getting rid of the
>>> >> >> Import() part seems not that straightforward.
>>> >> >>
>>> >> >> hua at huyangha-mobl:~/git/usw $ find . -iname "SConscript" | wc
>>> >> >> 121 121 5953
>>> >> >>
>>> >> >> At the moment the project supports 4 different CPU architectures, a
>>> >> >> little less than 1000 C files, 400K+ lines of code.
>>> >> >>
>>> >> >> And this is no small problem (at least for me). If I start working
>>> >> >> around it with too much python code, I might end up bypassing a
>>> >> >> significant portion of core scons and eventually might find it
>>> easier
>>> >> >> to just wrap around Makefiles.
>>> >> >>
>>> >> >> Someone thought I was criticizing scons, I actually love scons, but
>>> >> >> that doesn't stop me criticizing scons. ;-) Just don't see me as an
>>> >> >> enemy I am a serious user of scons since 10 years ago when I was
>>> still
>>> >> >> in college.
>>> >> >>
>>> >> >> Best Regards,
>>> >> >> Yanghao
>>> >> >>
>>> >> >> On Sat, Feb 17, 2018 at 4:46 PM, Bill Deegan
>>> >> >> <bill at baddogconsulting.com> wrote:
>>> >> >>> How many source files per SConscript?
>>> >> >>> That's a lot of sconscripts.
>>> >> >>> Is this all one project?
>>> >> >>> Are all the SConscripts identical?
>>> >> >>>
>>> >> >>> On Sat, Feb 17, 2018 at 3:05 AM, Hua Yanghao <
>>> huayanghao at gmail.com>
>>> >> >>> wrote:
>>> >> >>>>
>>> >> >>>> Actually I have one and only one import for hundreds of
>>> SConscript :)
>>> >> >>>>
>>> >> >>>> On Sat, Feb 17, 2018 at 01:19 Jonathon Reinhart
>>> >> >>>> <jonathon.reinhart at gmail.com> wrote:
>>> >> >>>>>
>>> >> >>>>> Hua,
>>> >> >>>>>
>>> >> >>>>> If you are importing lots of variables into SConscripts, then
>>> you're
>>> >> >>>>> using SCons wrong. Most of your variables should probably be
>>> >> >>>>> construction
>>> >> >>>>> variables (in the environment).
>>> >> >>>>>
>>> >> >>>>> I've written many, many SConscripts and I've never used more
>>> than 3
>>> >> >>>>> or 4
>>> >> >>>>> Import() statements in a single SConscript. Before you critique
>>> a
>>> >> >>>>> long-established framework, you might want to consider if you're
>>> >> >>>>> using it
>>> >> >>>>> inappropriately.
>>> >> >>>>>
>>> >> >>>>> Jonathon
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Fri, Feb 16, 2018 at 6:02 PM, Hua Yanghao <
>>> huayanghao at gmail.com>
>>> >> >>>>> wrote:
>>> >> >>>>>>
>>> >> >>>>>> Will do. It is really not good if a particular framework force
>>> user
>>> >> >>>>>> to
>>> >> >>>>>> create repeated code. In comparison kbuild can easily create a
>>> one
>>> >> >>>>>> line make
>>> >> >>>>>> file for a module.
>>> >> >>>>>>
>>> >> >>>>>> I am using localized scons anyway if not accepted will just
>>> use it
>>> >> >>>>>> on my
>>> >> >>>>>> own.
>>> >> >>>>>>
>>> >> >>>>>> On Fri, Feb 16, 2018 at 22:46 Bill Deegan
>>> >> >>>>>> <bill at baddogconsulting.com>
>>> >> >>>>>> wrote:
>>> >> >>>>>>>
>>> >> >>>>>>> Feel free to make a pull request, though I can't promise that
>>> it
>>> >> >>>>>>> will
>>> >> >>>>>>> be accepted.
>>> >> >>>>>>>
>>> >> >>>>>>> Is it really that much of an inconvenience?
>>> >> >>>>>>> -Bill
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> On Fri, Feb 16, 2018 at 3:20 PM, Hua Yanghao
>>> >> >>>>>>> <huayanghao at gmail.com>
>>> >> >>>>>>> wrote:
>>> >> >>>>>>>>
>>> >> >>>>>>>> Hi Bill,
>>> >> >>>>>>>> Do you think if it is useful to add a new keyword for
>>> >> >>>>>>>> env.SConscript(default_imports=[xyz,abc,...])?
>>> >> >>>>>>>> For return I think I can abuse some global variables. This
>>> can
>>> >> >>>>>>>> really
>>> >> >>>>>>>> make the SConscript clean/minimal.
>>> >> >>>>>>>>
>>> >> >>>>>>>> Best Regards,
>>> >> >>>>>>>> Yanghao
>>> >> >>>>>>>>
>>> >> >>>>>>>> On Fri, Feb 16, 2018 at 9:09 PM, Bill Deegan
>>> >> >>>>>>>> <bill at baddogconsulting.com> wrote:
>>> >> >>>>>>>> > No, there is no way without hacking the core code.
>>> >> >>>>>>>> >
>>> >> >>>>>>>> > It is the way it is by design.
>>> >> >>>>>>>> >
>>> >> >>>>>>>> > -Bill
>>> >> >>>>>>>> > SCons Project Co-Manager
>>> >> >>>>>>>> >
>>> >> >>>>>>>> > On Fri, Feb 16, 2018 at 2:28 PM, Hua Yanghao
>>> >> >>>>>>>> > <huayanghao at gmail.com>
>>> >> >>>>>>>> > wrote:
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> Does anyone know in scons there are ways to implicitly
>>> make a
>>> >> >>>>>>>> >> object
>>> >> >>>>>>>> >> (e.g. xyz) available for SConscript without explicity
>>> >> >>>>>>>> >> Import(xyz)?
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> Just want to eliminate the annoying repeated Import(xyz)
>>> in
>>> >> >>>>>>>> >> every
>>> >> >>>>>>>> >> single SConscript.
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> And what about implicit Return()? e.g. xyz() returns a
>>> list of
>>> >> >>>>>>>> >> object,
>>> >> >>>>>>>> >> then in every SConscript if the top level SConstruct want
>>> to
>>> >> >>>>>>>> >> collect
>>> >> >>>>>>>> >> the complete list of object returned from every single
>>> >> >>>>>>>> >> SConscript.
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> For example:
>>> >> >>>>>>>> >> SConscript-1:
>>> >> >>>>>>>> >> Import("xyz")
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> ret = xyz(...)
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> Return("ret")
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> Would ideally become:
>>> >> >>>>>>>> >> SConscript-2:
>>> >> >>>>>>>> >> xyz(...)
>>> >> >>>>>>>> >>
>>> >> >>>>>>>> >> Thanks,
>>> >> >>>>>>>> >> Yanghao
>>> >> >>>>>>>> >> _______________________________________________
>>> >> >>>>>>>> >> Scons-users mailing list
>>> >> >>>>>>>> >> Scons-users at scons.org
>>> >> >>>>>>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>>>>>> >
>>> >> >>>>>>>> >
>>> >> >>>>>>>> >
>>> >> >>>>>>>> > _______________________________________________
>>> >> >>>>>>>> > Scons-users mailing list
>>> >> >>>>>>>> > Scons-users at scons.org
>>> >> >>>>>>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>>>>>> >
>>> >> >>>>>>>> _______________________________________________
>>> >> >>>>>>>> Scons-users mailing list
>>> >> >>>>>>>> Scons-users at scons.org
>>> >> >>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> _______________________________________________
>>> >> >>>>>>> Scons-users mailing list
>>> >> >>>>>>> Scons-users at scons.org
>>> >> >>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> _______________________________________________
>>> >> >>>>>> Scons-users mailing list
>>> >> >>>>>> Scons-users at scons.org
>>> >> >>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>>>>
>>> >> >>>>>
>>> >> >>>>> _______________________________________________
>>> >> >>>>> Scons-users mailing list
>>> >> >>>>> Scons-users at scons.org
>>> >> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> _______________________________________________
>>> >> >>>> Scons-users mailing list
>>> >> >>>> Scons-users at scons.org
>>> >> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>>
>>> >> >>>
>>> >> >>>
>>> >> >>> _______________________________________________
>>> >> >>> Scons-users mailing list
>>> >> >>> Scons-users at scons.org
>>> >> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >> >>>
>>> >> _______________________________________________
>>> >> Scons-users mailing list
>>> >> Scons-users at scons.org
>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Scons-users mailing list
>>> > Scons-users at scons.org
>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>>> >
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180219/154befe9/attachment-0001.html>
More information about the Scons-users
mailing list