[Scons-users] Provide default Import to SConscript
Spencer Yost
syost at triad.rr.com
Sat Feb 17 22:54:38 EST 2018
Just as an offhand comment, that is sort of related to this. First, I don't have to import a lot.
We just converted our build system to SCons(~1600+ packages, ~15000 source files and tens of millions of lines of code). Because python expertise ranges from completely lacking to extremely poor :-) in our c/c++ staff; the powers that be indicated that developers will not maintain SConscript files. They will maintain the original build configuration files that we had(something that approximates a win.ini file). That means I have to rebuild SConscript files for every original configuration file we had. Trying to figure out a way to programmatically consolidate them to minimize the number of SConscript files was nearly impossible. So we have about 1600 SConscript files. Stuff like library dependencies vary between "packages" very frequently too. In addition the staff has to be able to build an individual "package" or the whole tree. I have a builder that creates the SConscript file whenever this original build configuration file changes. So it's not a big deal. But consolidating them was just way too big of a fish to fry in the time and budget allowed.
Sometimes you just have to have a bunch of them.
PS: SCons is working great for us. Dependencies are much more robust, speed is marginally better and the entire arc from source code change to Debian package creation is flawless. Being able to create command line options has been a godsend for us. Previously we would have to create build macros to emulate what command line options do.
Spencer Yost
> On Feb 17, 2018, at 6:17 PM, Bill Deegan <bill at baddogconsulting.com> wrote:
>
> Hua,
>
> From the tree you've shared, I'm going to guess you have way more SConscripts than you need.
>
> For example, you're set of test directories could have a single SConscript instead of one per.
> Which no doubt are just a few lines each.
>
> In some cases it looks like you have a SConscript in a directory which only has a header file (or none) and then a directory (or more) of source code. Skip the SConscript in the basically no-op directorie(s).
>
> Another suggestion, use variant dirs so you're not creating build files in your source directory.
>
> If you find yourself with a ton of very small SConscripts, it's a sign that you may be doing something non-scons'ian.
>
> Of course it also depends on how working on each group of code is split up between different developers.
>
> There is some art in structuring your build system (and your code) such that engineers rarely end up causing merge conflicts with each other.
>
> I can see some value in default_import, but it would be something requiring some debate on getting accepted.
> As far as I can recall, your's is the first request for something like this in the history of SCons (or at least as long as I've been involved).
>
> Philosophically SCons is about being explicit in the build requirements and also about being correct by design. Yielding reproducible builds regardless of the individual developers shell and machine setup.
>
>
>> On Sat, Feb 17, 2018 at 5:31 PM, Hua Yanghao <huayanghao at gmail.com> wrote:
>> Hi Bill,
>> I do not create a SConscript for every directory, only for directly
>> which either provide a set of include file or having source code to
>> build.
>> I will try to create a minimal version of the build system to show the idea.
>>
>> Not sure if the below tree will give you some idea what I am trying to
>> implement, at least you should be able to see where I am having a
>> SConscript file.
>> build/configs/riscv_example/riscv32-qemu-virt-main/
>> ├── cmdlist.txt
>> ├── config
>> │ └── config.h
>> ├── firmware
>> │ ├── arch
>> │ │ ├── include
>> │ │ │ └── arch
>> │ │ │ ├── cache.h
>> │ │ │ ├── irq.h
>> │ │ │ ├── misc.h
>> │ │ │ ├── mmu.h
>> │ │ │ ├── smp.h
>> │ │ │ ├── spinlock.h
>> │ │ │ └── timer.h
>> │ │ ├── riscv
>> │ │ │ ├── helper.c
>> │ │ │ ├── helper.o
>> │ │ │ ├── libdefault.o
>> │ │ │ ├── main.c
>> │ │ │ ├── main.o
>> │ │ │ ├── SConscript
>> │ │ │ ├── start.o
>> │ │ │ └── start.S
>> │ │ ├── SConscript
>> │ │ ├── weak.c
>> │ │ └── weak.o
>> │ ├── boards
>> │ │ ├── include
>> │ │ │ └── board.h
>> │ │ ├── qemu-riscv32-virt
>> │ │ │ ├── board.c
>> │ │ │ ├── board.o
>> │ │ │ ├── SConscript
>> │ │ │ ├── uart16550_device.c
>> │ │ │ └── uart16550_device.o
>> │ │ └── SConscript
>> │ ├── common
>> │ │ ├── cpu.c
>> │ │ ├── cpu.o
>> │ │ ├── include
>> │ │ │ └── common
>> │ │ │ ├── common.h
>> │ │ │ ├── cpu.h
>> │ │ │ └── log.h
>> │ │ ├── libdefault.o
>> │ │ ├── log.c
>> │ │ ├── log.o
>> │ │ ├── main.c
>> │ │ ├── main.o
>> │ │ ├── SConscript
>> │ │ ├── weak.c
>> │ │ └── weak.o
>> │ ├── drivers
>> │ │ └── uart16550
>> │ │ ├── fd_ops.c
>> │ │ ├── fd_ops.o
>> │ │ ├── include
>> │ │ │ ├── uart16550_api.h
>> │ │ │ └── uart16550_device.h
>> │ │ ├── libdefault.o
>> │ │ ├── SConscript
>> │ │ ├── uart16550.c
>> │ │ └── uart16550.o
>> │ ├── include
>> │ │ ├── bootstage.h
>> │ │ ├── compiler.h
>> │ │ ├── debug.h
>> │ │ └── linux
>> │ │ └── typecheck.h
>> │ ├── lib
>> │ │ ├── cmd
>> │ │ │ ├── cmd.c
>> │ │ │ ├── cmd.o
>> │ │ │ ├── include
>> │ │ │ │ └── cmd
>> │ │ │ │ └── cmd.h
>> │ │ │ ├── SConscript
>> │ │ │ ├── test.c
>> │ │ │ └── test.o
>> │ │ ├── fio
>> │ │ │ ├── cmd.c
>> │ │ │ ├── cmd.o
>> │ │ │ ├── include
>> │ │ │ │ └── usw_device_public.h
>> │ │ │ ├── libdefault.o
>> │ │ │ ├── newlib.c
>> │ │ │ ├── newlib.o
>> │ │ │ ├── SConscript
>> │ │ │ ├── usw_device.c
>> │ │ │ ├── usw_device.o
>> │ │ │ └── usw_device_private.h
>> │ │ ├── pipe
>> │ │ │ ├── include
>> │ │ │ │ └── pipe
>> │ │ │ │ ├── pipe.h
>> │ │ │ │ └── status.h
>> │ │ │ ├── libdefault.o
>> │ │ │ ├── pipe.c
>> │ │ │ ├── pipe.o
>> │ │ │ └── SConscript
>> │ │ └── simple_console
>> │ │ ├── include
>> │ │ │ └── simple_console.h
>> │ │ ├── libdefault.o
>> │ │ ├── print_color.h
>> │ │ ├── SConscript
>> │ │ ├── simple_console.c
>> │ │ └── simple_console.o
>> │ ├── SConscript
>> │ └── test
>> │ ├── common
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── console
>> │ │ ├── hello.c
>> │ │ ├── hello.o
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── example
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── libc
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── linker
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── log
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ └── setjmp
>> │ ├── SConscript
>> │ ├── test2.c
>> │ ├── test2.o
>> │ ├── test.c
>> │ └── test.o
>> ├── libriscv_example.a
>> ├── link.ld
>> ├── riscv_example.bin
>> ├── riscv_example.elf
>> ├── riscv_example.ibi_bin
>> ├── riscv_example.map
>> └── riscv_example.S
>>
>> 36 directories, 113 files
>>
>>
>>
>>
>> On Sat, Feb 17, 2018 at 10:15 PM, Bill Deegan <bill at baddogconsulting.com> wrote:
>> > Likely you don't need a SConscript for every directory in your case?
>> > Are you wrapping a complete linux build tree?
>> >
>> > Is this a build tree you can share for us to look at?
>> >
>> > -Bill
>> >
>> > On Sat, Feb 17, 2018 at 3:30 PM, Hua Yanghao <huayanghao at gmail.com> wrote:
>> >>
>> >> I see everything passed into env.SConscript(default_import=some_obj),
>> >> if some_obj is a string it can pass through but if some_obj is a
>> >> function object it end up as None in Script/SConscript.py/SConscript()
>> >> function.
>> >>
>> >> I could have used a string and then import the right function however
>> >> my function to be passed is a dynamically generated partial function
>> >> ...
>> >>
>> >> Could anyone please give me some hint how could this be best implemented?
>> >>
>> >> Best Regards,
>> >> Yanghao
>> >>
>> >> On Sat, Feb 17, 2018 at 7:53 PM, Hua Yanghao <huayanghao at gmail.com> wrote:
>> >> > @Bill, I have managed to hack the Frame class and the
>> >> > BuildDefaultGlobals() function to have some thing imported into
>> >> > SConscript by default.
>> >> > However I don't quite understand when the "default_import" keyword
>> >> > argument passed down, somewhere somehow it end up as a set.
>> >> >
>> >> > Any documentation to explain what is going on from Base class into
>> >> > SConscript() calls regarding parameter handling?
>> >> >
>> >> > Thanks,
>> >> > Yanghao
>> >> >
>> >> > On Sat, Feb 17, 2018 at 7:45 PM, Hua Yanghao <huayanghao at gmail.com>
>> >> > wrote:
>> >> >> It varies a lot, ranging from 0 (yes, no source files at all) to a few
>> >> >> dozen (didn't really count, see statistics below).
>> >> >>
>> >> >> Here is an example Linux kernel driver sub-module Makefile:
>> >> >>
>> >> >> linux/drivers/sn/Makefile:
>> >> >> 1 #
>> >> >> 2 # Makefile for the Altix device drivers.
>> >> >> 3 #
>> >> >> 4 #
>> >> >> 5
>> >> >> 6 obj-$(CONFIG_SGI_IOC3) += ioc3.o
>> >> >>
>> >> >> I want to replicate this kind of simplicity in scons.
>> >> >>
>> >> >> I actually have implemented almost the same thing in scons, but all
>> >> >> those SConscript files have to start with Import("my_func") ... which
>> >> >> is pretty annoying. I know I need it in every single SConscript, and I
>> >> >> have to repeat it ... and this is so anti-pattern.
>> >> >> I have get rid of the Return() part already, but getting rid of the
>> >> >> Import() part seems not that straightforward.
>> >> >>
>> >> >> hua at huyangha-mobl:~/git/usw $ find . -iname "SConscript" | wc
>> >> >> 121 121 5953
>> >> >>
>> >> >> At the moment the project supports 4 different CPU architectures, a
>> >> >> little less than 1000 C files, 400K+ lines of code.
>> >> >>
>> >> >> And this is no small problem (at least for me). If I start working
>> >> >> around it with too much python code, I might end up bypassing a
>> >> >> significant portion of core scons and eventually might find it easier
>> >> >> to just wrap around Makefiles.
>> >> >>
>> >> >> Someone thought I was criticizing scons, I actually love scons, but
>> >> >> that doesn't stop me criticizing scons. ;-) Just don't see me as an
>> >> >> enemy I am a serious user of scons since 10 years ago when I was still
>> >> >> in college.
>> >> >>
>> >> >> Best Regards,
>> >> >> Yanghao
>> >> >>
>> >> >> On Sat, Feb 17, 2018 at 4:46 PM, Bill Deegan
>> >> >> <bill at baddogconsulting.com> wrote:
>> >> >>> How many source files per SConscript?
>> >> >>> That's a lot of sconscripts.
>> >> >>> Is this all one project?
>> >> >>> Are all the SConscripts identical?
>> >> >>>
>> >> >>> On Sat, Feb 17, 2018 at 3:05 AM, Hua Yanghao <huayanghao at gmail.com>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> Actually I have one and only one import for hundreds of SConscript :)
>> >> >>>>
>> >> >>>> On Sat, Feb 17, 2018 at 01:19 Jonathon Reinhart
>> >> >>>> <jonathon.reinhart at gmail.com> wrote:
>> >> >>>>>
>> >> >>>>> Hua,
>> >> >>>>>
>> >> >>>>> If you are importing lots of variables into SConscripts, then you're
>> >> >>>>> using SCons wrong. Most of your variables should probably be
>> >> >>>>> construction
>> >> >>>>> variables (in the environment).
>> >> >>>>>
>> >> >>>>> I've written many, many SConscripts and I've never used more than 3
>> >> >>>>> or 4
>> >> >>>>> Import() statements in a single SConscript. Before you critique a
>> >> >>>>> long-established framework, you might want to consider if you're
>> >> >>>>> using it
>> >> >>>>> inappropriately.
>> >> >>>>>
>> >> >>>>> Jonathon
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Fri, Feb 16, 2018 at 6:02 PM, Hua Yanghao <huayanghao at gmail.com>
>> >> >>>>> wrote:
>> >> >>>>>>
>> >> >>>>>> Will do. It is really not good if a particular framework force user
>> >> >>>>>> to
>> >> >>>>>> create repeated code. In comparison kbuild can easily create a one
>> >> >>>>>> line make
>> >> >>>>>> file for a module.
>> >> >>>>>>
>> >> >>>>>> I am using localized scons anyway if not accepted will just use it
>> >> >>>>>> on my
>> >> >>>>>> own.
>> >> >>>>>>
>> >> >>>>>> On Fri, Feb 16, 2018 at 22:46 Bill Deegan
>> >> >>>>>> <bill at baddogconsulting.com>
>> >> >>>>>> wrote:
>> >> >>>>>>>
>> >> >>>>>>> Feel free to make a pull request, though I can't promise that it
>> >> >>>>>>> will
>> >> >>>>>>> be accepted.
>> >> >>>>>>>
>> >> >>>>>>> Is it really that much of an inconvenience?
>> >> >>>>>>> -Bill
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> On Fri, Feb 16, 2018 at 3:20 PM, Hua Yanghao
>> >> >>>>>>> <huayanghao at gmail.com>
>> >> >>>>>>> wrote:
>> >> >>>>>>>>
>> >> >>>>>>>> Hi Bill,
>> >> >>>>>>>> Do you think if it is useful to add a new keyword for
>> >> >>>>>>>> env.SConscript(default_imports=[xyz,abc,...])?
>> >> >>>>>>>> For return I think I can abuse some global variables. This can
>> >> >>>>>>>> really
>> >> >>>>>>>> make the SConscript clean/minimal.
>> >> >>>>>>>>
>> >> >>>>>>>> Best Regards,
>> >> >>>>>>>> Yanghao
>> >> >>>>>>>>
>> >> >>>>>>>> On Fri, Feb 16, 2018 at 9:09 PM, Bill Deegan
>> >> >>>>>>>> <bill at baddogconsulting.com> wrote:
>> >> >>>>>>>> > No, there is no way without hacking the core code.
>> >> >>>>>>>> >
>> >> >>>>>>>> > It is the way it is by design.
>> >> >>>>>>>> >
>> >> >>>>>>>> > -Bill
>> >> >>>>>>>> > SCons Project Co-Manager
>> >> >>>>>>>> >
>> >> >>>>>>>> > On Fri, Feb 16, 2018 at 2:28 PM, Hua Yanghao
>> >> >>>>>>>> > <huayanghao at gmail.com>
>> >> >>>>>>>> > wrote:
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Does anyone know in scons there are ways to implicitly make a
>> >> >>>>>>>> >> object
>> >> >>>>>>>> >> (e.g. xyz) available for SConscript without explicity
>> >> >>>>>>>> >> Import(xyz)?
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Just want to eliminate the annoying repeated Import(xyz) in
>> >> >>>>>>>> >> every
>> >> >>>>>>>> >> single SConscript.
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> And what about implicit Return()? e.g. xyz() returns a list of
>> >> >>>>>>>> >> object,
>> >> >>>>>>>> >> then in every SConscript if the top level SConstruct want to
>> >> >>>>>>>> >> collect
>> >> >>>>>>>> >> the complete list of object returned from every single
>> >> >>>>>>>> >> SConscript.
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> For example:
>> >> >>>>>>>> >> SConscript-1:
>> >> >>>>>>>> >> Import("xyz")
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> ret = xyz(...)
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Return("ret")
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Would ideally become:
>> >> >>>>>>>> >> SConscript-2:
>> >> >>>>>>>> >> xyz(...)
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Thanks,
>> >> >>>>>>>> >> Yanghao
>> >> >>>>>>>> >> _______________________________________________
>> >> >>>>>>>> >> Scons-users mailing list
>> >> >>>>>>>> >> Scons-users at scons.org
>> >> >>>>>>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>>> >
>> >> >>>>>>>> >
>> >> >>>>>>>> >
>> >> >>>>>>>> > _______________________________________________
>> >> >>>>>>>> > Scons-users mailing list
>> >> >>>>>>>> > Scons-users at scons.org
>> >> >>>>>>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>>> >
>> >> >>>>>>>> _______________________________________________
>> >> >>>>>>>> Scons-users mailing list
>> >> >>>>>>>> Scons-users at scons.org
>> >> >>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> _______________________________________________
>> >> >>>>>>> Scons-users mailing list
>> >> >>>>>>> Scons-users at scons.org
>> >> >>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> _______________________________________________
>> >> >>>>>> Scons-users mailing list
>> >> >>>>>> Scons-users at scons.org
>> >> >>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>> _______________________________________________
>> >> >>>>> Scons-users mailing list
>> >> >>>>> Scons-users at scons.org
>> >> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>
>> >> >>>>
>> >> >>>> _______________________________________________
>> >> >>>> Scons-users mailing list
>> >> >>>> Scons-users at scons.org
>> >> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> Scons-users mailing list
>> >> >>> Scons-users at scons.org
>> >> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>
>> >> _______________________________________________
>> >> Scons-users mailing list
>> >> Scons-users at scons.org
>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >
>> >
>> >
>> > _______________________________________________
>> > Scons-users mailing list
>> > Scons-users at scons.org
>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180217/437de1a9/attachment-0001.html>
More information about the Scons-users
mailing list