[Scons-users] Building object files in multiple environments

[Hua Yanghao]

Hi Dan,
The problem with building everything into a library and only linking
in the project specific part is that you assume all your project are
using the same CPU architecture/same compiler.
If you want to cover all different types of CPUs/Project in a single
repo ... building a separate library does not make too much sense.

FYI the SoC I am working on have 4 different CPU architecture on a
single project.

Regarding how it works it is really designing quite some python
modules and use scons as a library to handle the dependency and job
dispatching.
It is currently less than 4K lines of code but it involves quite some
design. However, everything boils down to import module SConscript and
adding necessary include path for that module.

Basically this is how it works:
* SConstruct is the entry to read in the config file and import
necessary modules
* Each module at is level only specifies the source file, and return a
list of source file to the toplevel SConstruct file.
* Top level SConstruct file will then build each module source file
into object files as well as a library (lib_files/obj_files).
* SConstruct then takes all the library and object files and create
the final binary (using variant_dir for SConscript in order NOT to
build in place).
* The configuration file can be used as: 1. configuration file (of
course) 2. trigger the build (it invokes scons) 3. execute the target
(run it in emulator or real board).

Best Regards,
Yanghao

On Wed, Nov 8, 2017 at 12:38 PM, Dan Čermák
<dan.cermak at cgc-instruments.com> wrote:
> Thanks a lot for your extensive answer!
>
> Do I understand it correctly that for each "device"/"project" you infer
> all modules from the config file and then build them for that specific
> project?
>
> I actually wanted it kind of the other way around: all modules are build
> once and then linked for each project, but the project specific stuff
> should be build with an additional include path.
>
> I must admit that I like your approach very much and I think I could use
> something like that, too. Could you give me maybe a hint how your
> configuration system roughly works (only if you can do that without
> getting into trouble with your employer)?
>
>
> Cheers,
>
> Dan
>
>
> Hua Yanghao <huayanghao at gmail.com> writes:
>
>> I think it is much easier to introduce something called "configuration".
>> Take a look at the Linux kernel or u-boot Kconfig system, where
>> everything is considered equally as source code and then the
>> configuration file selects which pieces you want to compile and link
>> together.
>>
>> My current project folder structure is something like this:
>> hua at grass:~/git/usw $ tree -L 1
>> .
>> ├── arch
>> ├── boards
>> ├── build_list.txt
>> ├── common
>> ├── configs
>> ├── doc
>> ├── drivers
>> ├── external
>> ├── include
>> ├── lib
>> ├── Makefile
>> ├── output
>> ├── README.md
>> ├── SConstruct
>> ├── site_scons
>> ├── soc
>> ├── test
>> ├── TODO.md
>> └── tools
>>
>> 14 directories, 5 files
>> hua at grass:~/git/usw $
>>
>> where in configs/ you have all the configuration file written in
>> python, for example configs/qemu_arm_vexpress_bios.py: 1
>> #!/usr/bin/env python
>>   2 '''
>>   3 To execute:
>>   4 export QEMU_AUDIO_DRV="none"
>>   5 qemu-system-arm -M vexpress-a9 -nographic -kernel
>> output/armv7_gcc_full/usw.elf
>>   6 '''
>>   7 from common import ConfigMeta
>>   8 from compiler import gcc_arm_none_eabi
>>   9
>>  10 class Config(object):
>>  11     __metaclass__ = ConfigMeta
>>  12     COMPILER = gcc_arm_none_eabi.Compiler()
>>  13
>>  14     CONFIG_ARCH = "armv7"
>>  15     CONFIG_NO_STACK = 1
>>  16     CONFIG_BIOS_START = 1
>>  17     CONFIG_BOOT_ENTRY = 0x60000200
>>  18     CONFIG_LINK_SCRIPT = "arch/armv7/link.ld"
>>  19     CONFIG_TEXT_START = 0x00000000
>>  20     CONFIG_HEAP_SIZE = 0x100000
>>  21     CONFIG_ARM_BASE = 0x1e000000
>>  22     CONFIG_UART_BASE = 0x10009000
>>  23     CONFIG_PRINT_COLOR = 1
>>  24
>>  25     CONFIG_IBI_HEADER = 1
>>  26     CONFIG_IBI_SIZELIMIT = 0x64000
>>  27     # SMP
>>  28     CONFIG_CPU_NUM = 4
>>  29     CONFIG_IRQ_STACK_SIZE = 0x10000
>>  30     CONFIG_FIQ_STACK_SIZE = 0x10000
>>  31
>>  32     # Module List
>>  33     MODULE_LIST = [
>>  34         "arch",
>>  35         "arch/armv7",
>>  36     ]
>>
>> This is one of my simplest config file, where it selects which module
>> (e.g. each folder that have a SConscript is considered a module) is
>> specified in the MODULE_LIST and then in SConstruct the SConscript
>> file is automatically imported.
>> All build that is corresponding to a particular config file has a
>> dedicated output folder:
>> hua at grass:~/git/usw $ ll output/
>> total 28
>> drwxr-xr-x  9 hua hua 4096 Nov  8 10:30 linux64_full
>> drwxr-xr-x 10 hua hua 4096 Nov  8 10:29 qemu_arm_vexpress
>> drwxr-xr-x  4 hua hua 4096 Nov  8 10:30 qemu_arm_vexpress_bios
>> drwxr-xr-x 11 hua hua 4096 Nov  8 10:30 xmm7xx0
>> drwxr-xr-x 11 hua hua 4096 Nov  8 10:30 xmm7xx0_xxx
>> drwxr-xr-x  4 hua hua 4096 Nov  8 10:30 xmm7xx0_xxx_lmu
>> drwxr-xr-x  8 hua hua 4096 Nov  8 10:30 xmm7xx0_xxx_yyy
>> hua at grass:~/git/usw $
>>
>> So even for a single device, you could have have different
>> configuration, e.g. one for boot loader/bios, one for verification SW,
>> one for production SW etc.
>>
>> The best part I love SCons is that in the output folder there is a
>> copy of the file that is actually being selected/used from the
>> configuration system so you know exactly what is compiled.
>>
>> For each individual module the SConscript is extremely simple and
>> straightforward:
>>
>> hua at grass:~/git/usw $ cat boards/vexpress/SConscript
>> name = "usw_lib"
>>
>> Import('cs')
>> Import('usw_files')
>>
>> obj_files = [
>>     "board.c",
>>     "pl011_device.c",
>>     "test_device.c",
>>     "pipe_device.c",
>> ]
>>
>> lib_files = [
>> ]
>>
>> if hasattr(cs, "CONFIG_LOAD_BINARY"):
>>     obj_files.append(("binary.c", ["cmd.txt", "page_table.bin"]))
>>
>> ret = usw_files(name, lib_files, obj_files)
>>
>> Return('ret')
>> hua at grass:~/git/usw $
>>
>> There you just provide a obj_files / lib_files and you can even
>> specify dependency files (where you don't really want a Scons parser),
>> and even specify customized compilation flags for individual files
>> (Kconfig/Kbuild can also do this, but not the explicit dependency
>> specification).
>>
>> I am trying to make this framework (basically Kconfig/Kbuild reduced
>> feature set implemented using SCons) open source but company process
>> is lengthy ...
>> Or maybe it even makes sense to make this kind of feature as part of
>> SCons itself?
>>
>> I hope this helps a little bit.
>>
>> Best Regards,
>> Hua Yanghao
>>
>> On Wed, Nov 8, 2017 at 10:48 AM, Dan Čermák
>> <dan.cermak at cgc-instruments.com> wrote:
>>> Hi Folks,
>>>
>>> I am currently using SCons for a big mono-repo C++ Firmware, where I
>>> have lots of common source files and then a directory where the actual
>>> firmware for each device is (this is just a .cpp file with device
>>> specific configurations and the appropriate high-level logic).
>>>
>>> For illustration purposes, the directory structure looks something like
>>> this:
>>> .
>>> ├── devices
>>> │   ├── Device_A
>>> │   │   ├── main.cpp
>>> │   │   ├── SConscript
>>> │   │   ├── uart.cpp
>>> │   │   └── uart.hpp
>>> │   ├── Device_B
>>> │   │   ├── main.cpp
>>> │   │   └── SConscript
>>> │   └── SConscript
>>> ├── spi
>>> │   └── config.cpp
>>> ├── uart
>>> │   └── uart.cpp
>>> ├── util
>>> │   └── endian.c
>>> └── SConstruct
>>>
>>> The idea behind this is that everything outside of 'devices' is
>>> considered as common files and build into object files. Every
>>> subdirectory of devices should have a SConscript that creates a single
>>> binary file (this is the firmware for the specific device) that is
>>> linked with the common object files.
>>>
>>> I have achieved this by creating object files for every cpp file not in
>>> devices/ and passing them to the SConscript in devices/. However, there
>>> is a catch: I would like to build everything under devices with an
>>> additional include path (the top level directory of the project) which
>>> should not be propagated to the common object files. I therefore tried
>>> creating a clone of the environment for each device in
>>> devices/SConscript and then doing the following in
>>> devices/Device_A/SConscript:
>>>
>>> Import('env_clone', 'obj')
>>>
>>> local_obj = env_clone.Object(Glob('*.cpp'))
>>> prog = env_clone.Program('Device_A_bin', local_obj + obj)
>>>
>>> Return('prog')
>>>
>>> where env_clone is the cloned environment with the additional include
>>> path and obj the list of object files that are common to all
>>> devices (which have been created in the environment env from which
>>> env_clone was cloned). This however causes SCons to complain, that there
>>> are object files in multiple environments with the same build command.
>>>
>>>
>>> My guess is, that my solution of cloning environments is not the correct
>>> way. Does someone have an idea how to achieve this with SCons?
>>>
>>>
>>> Thanks in advance,
>>>
>>> Dan
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users