[Scons-users] SCons Python Builder not building in parallel?

[Hua Yanghao]

Thanks Bill.

I was thinking why scons does not fork some process to handle the
python-based buliders. But now I realized that this cannot be done
reliably as arbitrary python code could be executed and change the
current python execution context and this can only be done in the same
address space e.g. only threads can be used.

It is not so easy to move the logic to a independent script based
builder due to exactly the same reason ... because I do not have
control of what users might have add in their configuration python
code. So I decided to add a few more prints to ease the feeling of a
none-response period ... until I got some time to really bench marking
my configuration code and find out the hot spots ... :-)
On Sat, Sep 22, 2018 at 10:01 PM Bill Deegan <bill at baddogconsulting.com> wrote:
>
> So you have a builder which is implemented in python and does non-trivial amount of work in the builder logic itself?
> If so, then yes move the logic to a script and have your builder call the script with the appropriate arguments.
>
> When scons is running your builders python logic the GIL comes into play.
>
> -Bill
>
> On Sat, Sep 22, 2018 at 1:16 PM Hua Yanghao <huayanghao at gmail.com> wrote:
>>
>> Dear Scons Users,
>> I recently noticed that with -jN my CPU utilization is less than 100%
>> (4 cores) during some phases (e.g. when executing a python builder)
>> while reaches 400% when compiling C codes. I have a python builder
>> that generates quite some .h files, which all the C files depend on.
>> Those .h files has no relationship/dependencies among themselves, and
>> running with --debug=explain gives below information:
>>
>> ./run configs/qemu_arm_vexpress.py -j8 --debug=explain
>> Running : ./tools/scons -Q config=configs/qemu_arm_vexpress.py -j8
>> --debug=explain
>> USW INFO: usw_config completed in 1.60 seconds.
>> scons: rebuilding
>> `build/configs/qemu_arm_vexpress/bios_bin/config/config.h' because
>> AlwaysBuild() is specified
>> __CONFIG build/configs/qemu_arm_vexpress/bios_bin/config/config.h
>> scons: rebuilding
>> `build/configs/qemu_arm_vexpress/bios_elf/config/config.h' because
>> AlwaysBuild() is specified
>> __CONFIG build/configs/qemu_arm_vexpress/bios_elf/config/config.h
>> scons: rebuilding
>> `build/configs/qemu_arm_vexpress/core0/config/config.h' because
>> AlwaysBuild() is specified
>> __CONFIG build/configs/qemu_arm_vexpress/core0/config/config.h
>> scons: rebuilding
>> `build/configs/qemu_arm_vexpress/core1/config/config.h' because
>> AlwaysBuild() is specified
>> __CONFIG build/configs/qemu_arm_vexpress/core1/config/config.h
>> scons: rebuilding
>> `build/configs/qemu_arm_vexpress/core2/config/config.h' because
>> AlwaysBuild() is specified
>> __CONFIG build/configs/qemu_arm_vexpress/core2/config/config.h
>> scons: rebuilding
>> `build/configs/qemu_arm_vexpress/core3/config/config.h' because
>> AlwaysBuild() is specified
>> __CONFIG build/configs/qemu_arm_vexpress/core3/config/config.h
>>
>> Any hints why these are not building in parallel? Is it because scons
>> builds python with multithreads where the GIL eats all the CPU
>> intensive workloads performance? Or should I re-implement the builder
>> in such a way that it is not using scons tools but rather running as a
>> dedicated executable and then wrapped into scons tools?
>>
>> Thanks,
>> Yanghao Hua
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users