[Scons-users] Provide default Import to SConscript
Bill Deegan
bill at baddogconsulting.com
Sun Feb 18 09:54:06 EST 2018
Spencer,
So your build info source are these .ini files?
And your SConscript files just process them?
-Bill
On Sat, Feb 17, 2018 at 10:54 PM, Spencer Yost <syost at triad.rr.com> wrote:
> Just as an offhand comment, that is sort of related to this. First, I
> don't have to import a lot.
>
> We just converted our build system to SCons(~1600+ packages, ~15000 source
> files and tens of millions of lines of code). Because python expertise
> ranges from completely lacking to extremely poor :-) in our c/c++ staff;
> the powers that be indicated that developers will not maintain SConscript
> files. They will maintain the original build configuration files that we
> had(something that approximates a win.ini file). That means I have to
> rebuild SConscript files for every original configuration file we had.
> Trying to figure out a way to programmatically consolidate them to minimize
> the number of SConscript files was nearly impossible. So we have about 1600
> SConscript files. Stuff like library dependencies vary between
> "packages" very frequently too. In addition the staff has to be able to
> build an individual "package" or the whole tree. I have a builder that
> creates the SConscript file whenever this original build configuration file
> changes. So it's not a big deal. But consolidating them was just way too
> big of a fish to fry in the time and budget allowed.
>
> Sometimes you just have to have a bunch of them.
>
> PS: SCons is working great for us. Dependencies are much more robust,
> speed is marginally better and the entire arc from source code change to
> Debian package creation is flawless. Being able to create command line
> options has been a godsend for us. Previously we would have to create
> build macros to emulate what command line options do.
>
> Spencer Yost
>
> On Feb 17, 2018, at 6:17 PM, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
> Hua,
>
> From the tree you've shared, I'm going to guess you have way more
> SConscripts than you need.
>
> For example, you're set of test directories could have a single SConscript
> instead of one per.
> Which no doubt are just a few lines each.
>
> In some cases it looks like you have a SConscript in a directory which
> only has a header file (or none) and then a directory (or more) of source
> code. Skip the SConscript in the basically no-op directorie(s).
>
> Another suggestion, use variant dirs so you're not creating build files in
> your source directory.
>
> If you find yourself with a ton of very small SConscripts, it's a sign
> that you may be doing something non-scons'ian.
>
> Of course it also depends on how working on each group of code is split up
> between different developers.
>
> There is some art in structuring your build system (and your code) such
> that engineers rarely end up causing merge conflicts with each other.
>
> I can see some value in default_import, but it would be something
> requiring some debate on getting accepted.
> As far as I can recall, your's is the first request for something like
> this in the history of SCons (or at least as long as I've been involved).
>
> Philosophically SCons is about being explicit in the build requirements
> and also about being correct by design. Yielding reproducible builds
> regardless of the individual developers shell and machine setup.
>
>
> On Sat, Feb 17, 2018 at 5:31 PM, Hua Yanghao <huayanghao at gmail.com> wrote:
>
>> Hi Bill,
>> I do not create a SConscript for every directory, only for directly
>> which either provide a set of include file or having source code to
>> build.
>> I will try to create a minimal version of the build system to show the
>> idea.
>>
>> Not sure if the below tree will give you some idea what I am trying to
>> implement, at least you should be able to see where I am having a
>> SConscript file.
>> build/configs/riscv_example/riscv32-qemu-virt-main/
>> ├── cmdlist.txt
>> ├── config
>> │ └── config.h
>> ├── firmware
>> │ ├── arch
>> │ │ ├── include
>> │ │ │ └── arch
>> │ │ │ ├── cache.h
>> │ │ │ ├── irq.h
>> │ │ │ ├── misc.h
>> │ │ │ ├── mmu.h
>> │ │ │ ├── smp.h
>> │ │ │ ├── spinlock.h
>> │ │ │ └── timer.h
>> │ │ ├── riscv
>> │ │ │ ├── helper.c
>> │ │ │ ├── helper.o
>> │ │ │ ├── libdefault.o
>> │ │ │ ├── main.c
>> │ │ │ ├── main.o
>> │ │ │ ├── SConscript
>> │ │ │ ├── start.o
>> │ │ │ └── start.S
>> │ │ ├── SConscript
>> │ │ ├── weak.c
>> │ │ └── weak.o
>> │ ├── boards
>> │ │ ├── include
>> │ │ │ └── board.h
>> │ │ ├── qemu-riscv32-virt
>> │ │ │ ├── board.c
>> │ │ │ ├── board.o
>> │ │ │ ├── SConscript
>> │ │ │ ├── uart16550_device.c
>> │ │ │ └── uart16550_device.o
>> │ │ └── SConscript
>> │ ├── common
>> │ │ ├── cpu.c
>> │ │ ├── cpu.o
>> │ │ ├── include
>> │ │ │ └── common
>> │ │ │ ├── common.h
>> │ │ │ ├── cpu.h
>> │ │ │ └── log.h
>> │ │ ├── libdefault.o
>> │ │ ├── log.c
>> │ │ ├── log.o
>> │ │ ├── main.c
>> │ │ ├── main.o
>> │ │ ├── SConscript
>> │ │ ├── weak.c
>> │ │ └── weak.o
>> │ ├── drivers
>> │ │ └── uart16550
>> │ │ ├── fd_ops.c
>> │ │ ├── fd_ops.o
>> │ │ ├── include
>> │ │ │ ├── uart16550_api.h
>> │ │ │ └── uart16550_device.h
>> │ │ ├── libdefault.o
>> │ │ ├── SConscript
>> │ │ ├── uart16550.c
>> │ │ └── uart16550.o
>> │ ├── include
>> │ │ ├── bootstage.h
>> │ │ ├── compiler.h
>> │ │ ├── debug.h
>> │ │ └── linux
>> │ │ └── typecheck.h
>> │ ├── lib
>> │ │ ├── cmd
>> │ │ │ ├── cmd.c
>> │ │ │ ├── cmd.o
>> │ │ │ ├── include
>> │ │ │ │ └── cmd
>> │ │ │ │ └── cmd.h
>> │ │ │ ├── SConscript
>> │ │ │ ├── test.c
>> │ │ │ └── test.o
>> │ │ ├── fio
>> │ │ │ ├── cmd.c
>> │ │ │ ├── cmd.o
>> │ │ │ ├── include
>> │ │ │ │ └── usw_device_public.h
>> │ │ │ ├── libdefault.o
>> │ │ │ ├── newlib.c
>> │ │ │ ├── newlib.o
>> │ │ │ ├── SConscript
>> │ │ │ ├── usw_device.c
>> │ │ │ ├── usw_device.o
>> │ │ │ └── usw_device_private.h
>> │ │ ├── pipe
>> │ │ │ ├── include
>> │ │ │ │ └── pipe
>> │ │ │ │ ├── pipe.h
>> │ │ │ │ └── status.h
>> │ │ │ ├── libdefault.o
>> │ │ │ ├── pipe.c
>> │ │ │ ├── pipe.o
>> │ │ │ └── SConscript
>> │ │ └── simple_console
>> │ │ ├── include
>> │ │ │ └── simple_console.h
>> │ │ ├── libdefault.o
>> │ │ ├── print_color.h
>> │ │ ├── SConscript
>> │ │ ├── simple_console.c
>> │ │ └── simple_console.o
>> │ ├── SConscript
>> │ └── test
>> │ ├── common
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── console
>> │ │ ├── hello.c
>> │ │ ├── hello.o
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── example
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── libc
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── linker
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ ├── log
>> │ │ ├── SConscript
>> │ │ ├── test.c
>> │ │ └── test.o
>> │ └── setjmp
>> │ ├── SConscript
>> │ ├── test2.c
>> │ ├── test2.o
>> │ ├── test.c
>> │ └── test.o
>> ├── libriscv_example.a
>> ├── link.ld
>> ├── riscv_example.bin
>> ├── riscv_example.elf
>> ├── riscv_example.ibi_bin
>> ├── riscv_example.map
>> └── riscv_example.S
>>
>> 36 directories, 113 files
>>
>>
>>
>>
>> On Sat, Feb 17, 2018 at 10:15 PM, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>> > Likely you don't need a SConscript for every directory in your case?
>> > Are you wrapping a complete linux build tree?
>> >
>> > Is this a build tree you can share for us to look at?
>> >
>> > -Bill
>> >
>> > On Sat, Feb 17, 2018 at 3:30 PM, Hua Yanghao <huayanghao at gmail.com>
>> wrote:
>> >>
>> >> I see everything passed into env.SConscript(default_import=some_obj),
>> >> if some_obj is a string it can pass through but if some_obj is a
>> >> function object it end up as None in Script/SConscript.py/SConscript()
>> >> function.
>> >>
>> >> I could have used a string and then import the right function however
>> >> my function to be passed is a dynamically generated partial function
>> >> ...
>> >>
>> >> Could anyone please give me some hint how could this be best
>> implemented?
>> >>
>> >> Best Regards,
>> >> Yanghao
>> >>
>> >> On Sat, Feb 17, 2018 at 7:53 PM, Hua Yanghao <huayanghao at gmail.com>
>> wrote:
>> >> > @Bill, I have managed to hack the Frame class and the
>> >> > BuildDefaultGlobals() function to have some thing imported into
>> >> > SConscript by default.
>> >> > However I don't quite understand when the "default_import" keyword
>> >> > argument passed down, somewhere somehow it end up as a set.
>> >> >
>> >> > Any documentation to explain what is going on from Base class into
>> >> > SConscript() calls regarding parameter handling?
>> >> >
>> >> > Thanks,
>> >> > Yanghao
>> >> >
>> >> > On Sat, Feb 17, 2018 at 7:45 PM, Hua Yanghao <huayanghao at gmail.com>
>> >> > wrote:
>> >> >> It varies a lot, ranging from 0 (yes, no source files at all) to a
>> few
>> >> >> dozen (didn't really count, see statistics below).
>> >> >>
>> >> >> Here is an example Linux kernel driver sub-module Makefile:
>> >> >>
>> >> >> linux/drivers/sn/Makefile:
>> >> >> 1 #
>> >> >> 2 # Makefile for the Altix device drivers.
>> >> >> 3 #
>> >> >> 4 #
>> >> >> 5
>> >> >> 6 obj-$(CONFIG_SGI_IOC3) += ioc3.o
>> >> >>
>> >> >> I want to replicate this kind of simplicity in scons.
>> >> >>
>> >> >> I actually have implemented almost the same thing in scons, but all
>> >> >> those SConscript files have to start with Import("my_func") ...
>> which
>> >> >> is pretty annoying. I know I need it in every single SConscript,
>> and I
>> >> >> have to repeat it ... and this is so anti-pattern.
>> >> >> I have get rid of the Return() part already, but getting rid of the
>> >> >> Import() part seems not that straightforward.
>> >> >>
>> >> >> hua at huyangha-mobl:~/git/usw $ find . -iname "SConscript" | wc
>> >> >> 121 121 5953
>> >> >>
>> >> >> At the moment the project supports 4 different CPU architectures, a
>> >> >> little less than 1000 C files, 400K+ lines of code.
>> >> >>
>> >> >> And this is no small problem (at least for me). If I start working
>> >> >> around it with too much python code, I might end up bypassing a
>> >> >> significant portion of core scons and eventually might find it
>> easier
>> >> >> to just wrap around Makefiles.
>> >> >>
>> >> >> Someone thought I was criticizing scons, I actually love scons, but
>> >> >> that doesn't stop me criticizing scons. ;-) Just don't see me as an
>> >> >> enemy I am a serious user of scons since 10 years ago when I was
>> still
>> >> >> in college.
>> >> >>
>> >> >> Best Regards,
>> >> >> Yanghao
>> >> >>
>> >> >> On Sat, Feb 17, 2018 at 4:46 PM, Bill Deegan
>> >> >> <bill at baddogconsulting.com> wrote:
>> >> >>> How many source files per SConscript?
>> >> >>> That's a lot of sconscripts.
>> >> >>> Is this all one project?
>> >> >>> Are all the SConscripts identical?
>> >> >>>
>> >> >>> On Sat, Feb 17, 2018 at 3:05 AM, Hua Yanghao <huayanghao at gmail.com
>> >
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> Actually I have one and only one import for hundreds of
>> SConscript :)
>> >> >>>>
>> >> >>>> On Sat, Feb 17, 2018 at 01:19 Jonathon Reinhart
>> >> >>>> <jonathon.reinhart at gmail.com> wrote:
>> >> >>>>>
>> >> >>>>> Hua,
>> >> >>>>>
>> >> >>>>> If you are importing lots of variables into SConscripts, then
>> you're
>> >> >>>>> using SCons wrong. Most of your variables should probably be
>> >> >>>>> construction
>> >> >>>>> variables (in the environment).
>> >> >>>>>
>> >> >>>>> I've written many, many SConscripts and I've never used more
>> than 3
>> >> >>>>> or 4
>> >> >>>>> Import() statements in a single SConscript. Before you critique a
>> >> >>>>> long-established framework, you might want to consider if you're
>> >> >>>>> using it
>> >> >>>>> inappropriately.
>> >> >>>>>
>> >> >>>>> Jonathon
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Fri, Feb 16, 2018 at 6:02 PM, Hua Yanghao <
>> huayanghao at gmail.com>
>> >> >>>>> wrote:
>> >> >>>>>>
>> >> >>>>>> Will do. It is really not good if a particular framework force
>> user
>> >> >>>>>> to
>> >> >>>>>> create repeated code. In comparison kbuild can easily create a
>> one
>> >> >>>>>> line make
>> >> >>>>>> file for a module.
>> >> >>>>>>
>> >> >>>>>> I am using localized scons anyway if not accepted will just use
>> it
>> >> >>>>>> on my
>> >> >>>>>> own.
>> >> >>>>>>
>> >> >>>>>> On Fri, Feb 16, 2018 at 22:46 Bill Deegan
>> >> >>>>>> <bill at baddogconsulting.com>
>> >> >>>>>> wrote:
>> >> >>>>>>>
>> >> >>>>>>> Feel free to make a pull request, though I can't promise that
>> it
>> >> >>>>>>> will
>> >> >>>>>>> be accepted.
>> >> >>>>>>>
>> >> >>>>>>> Is it really that much of an inconvenience?
>> >> >>>>>>> -Bill
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> On Fri, Feb 16, 2018 at 3:20 PM, Hua Yanghao
>> >> >>>>>>> <huayanghao at gmail.com>
>> >> >>>>>>> wrote:
>> >> >>>>>>>>
>> >> >>>>>>>> Hi Bill,
>> >> >>>>>>>> Do you think if it is useful to add a new keyword for
>> >> >>>>>>>> env.SConscript(default_imports=[xyz,abc,...])?
>> >> >>>>>>>> For return I think I can abuse some global variables. This can
>> >> >>>>>>>> really
>> >> >>>>>>>> make the SConscript clean/minimal.
>> >> >>>>>>>>
>> >> >>>>>>>> Best Regards,
>> >> >>>>>>>> Yanghao
>> >> >>>>>>>>
>> >> >>>>>>>> On Fri, Feb 16, 2018 at 9:09 PM, Bill Deegan
>> >> >>>>>>>> <bill at baddogconsulting.com> wrote:
>> >> >>>>>>>> > No, there is no way without hacking the core code.
>> >> >>>>>>>> >
>> >> >>>>>>>> > It is the way it is by design.
>> >> >>>>>>>> >
>> >> >>>>>>>> > -Bill
>> >> >>>>>>>> > SCons Project Co-Manager
>> >> >>>>>>>> >
>> >> >>>>>>>> > On Fri, Feb 16, 2018 at 2:28 PM, Hua Yanghao
>> >> >>>>>>>> > <huayanghao at gmail.com>
>> >> >>>>>>>> > wrote:
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Does anyone know in scons there are ways to implicitly
>> make a
>> >> >>>>>>>> >> object
>> >> >>>>>>>> >> (e.g. xyz) available for SConscript without explicity
>> >> >>>>>>>> >> Import(xyz)?
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Just want to eliminate the annoying repeated Import(xyz) in
>> >> >>>>>>>> >> every
>> >> >>>>>>>> >> single SConscript.
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> And what about implicit Return()? e.g. xyz() returns a
>> list of
>> >> >>>>>>>> >> object,
>> >> >>>>>>>> >> then in every SConscript if the top level SConstruct want
>> to
>> >> >>>>>>>> >> collect
>> >> >>>>>>>> >> the complete list of object returned from every single
>> >> >>>>>>>> >> SConscript.
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> For example:
>> >> >>>>>>>> >> SConscript-1:
>> >> >>>>>>>> >> Import("xyz")
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> ret = xyz(...)
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Return("ret")
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Would ideally become:
>> >> >>>>>>>> >> SConscript-2:
>> >> >>>>>>>> >> xyz(...)
>> >> >>>>>>>> >>
>> >> >>>>>>>> >> Thanks,
>> >> >>>>>>>> >> Yanghao
>> >> >>>>>>>> >> _______________________________________________
>> >> >>>>>>>> >> Scons-users mailing list
>> >> >>>>>>>> >> Scons-users at scons.org
>> >> >>>>>>>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>>> >
>> >> >>>>>>>> >
>> >> >>>>>>>> >
>> >> >>>>>>>> > _______________________________________________
>> >> >>>>>>>> > Scons-users mailing list
>> >> >>>>>>>> > Scons-users at scons.org
>> >> >>>>>>>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>>> >
>> >> >>>>>>>> _______________________________________________
>> >> >>>>>>>> Scons-users mailing list
>> >> >>>>>>>> Scons-users at scons.org
>> >> >>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> _______________________________________________
>> >> >>>>>>> Scons-users mailing list
>> >> >>>>>>> Scons-users at scons.org
>> >> >>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> _______________________________________________
>> >> >>>>>> Scons-users mailing list
>> >> >>>>>> Scons-users at scons.org
>> >> >>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>> _______________________________________________
>> >> >>>>> Scons-users mailing list
>> >> >>>>> Scons-users at scons.org
>> >> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>
>> >> >>>>
>> >> >>>> _______________________________________________
>> >> >>>> Scons-users mailing list
>> >> >>>> Scons-users at scons.org
>> >> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> Scons-users mailing list
>> >> >>> Scons-users at scons.org
>> >> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >> >>>
>> >> _______________________________________________
>> >> Scons-users mailing list
>> >> Scons-users at scons.org
>> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >
>> >
>> >
>> > _______________________________________________
>> > Scons-users mailing list
>> > Scons-users at scons.org
>> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> >
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180218/9c751b77/attachment-0001.html>
More information about the Scons-users
mailing list