[Scons-users] Scons none-deterministic behavior for incremental build
Hua Yanghao
huayanghao at gmail.com
Wed Dec 5 03:23:35 EST 2018
It is impossible to run on python2.7 within a reasonable time as a
huge list of python3-only syntax/modules needs to be adapted ...
On Wed, Dec 5, 2018 at 12:15 AM Bill Deegan <bill at baddogconsulting.com> wrote:
>
> Does this happen if you use python 2.7?
>
> On Tue, Dec 4, 2018 at 6:00 PM Hua Yanghao <huayanghao at gmail.com> wrote:
>>
>> Apparently the MD5 signature is taking *turns* some how.
>>
>> You can see below the old md5 and new md5 checksum is somehow
>> alternating between two fixed value ...
>> How is scons calculating the md5 value? is it only depends on the file content?
>> The md5sum of the "arch_init.c" is:
>> md5sum firmware/arch/armv7/arch_init.c
>> db624fbb365812d7ba58c87d49f44b6f firmware/arch/armv7/arch_init.c
>>
>> First build:
>> e494890ea1b6e36acf275f3698a3f3f9
>> c706a016770a7c85ac42f16f79456797
>> Returning True
>> in self.changed ... 2
>> return not r: False
>> ['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
>> <SCons.Script.Main.BuildTask object at 0x7fdc934c7fd0>
>> ______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o
>>
>> Second build:
>> c706a016770a7c85ac42f16f79456797
>> e494890ea1b6e36acf275f3698a3f3f9
>> Returning True
>> in self.changed ... 2
>> return not r: False
>> ['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o']
>> <SCons.Script.Main.BuildTask object at 0x7f3fca5e4fd0>
>> ______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o
>> On Tue, Dec 4, 2018 at 11:54 PM Hua Yanghao <huayanghao at gmail.com> wrote:
>> >
>> > That is generated by sconstruct, not from shell. Any potential issue
>> > here? (It was working for a very long time before and only breaks
>> > recently when I added some new features).
>> >
>> > In the meantime I have isolated the issue to be in Node.Node.changed()
>> > call, where scons old & new signature for the C files seems not
>> > matching (print("###### 3.1") is executed in error case):
>> > 1463 if self.has_builder():
>> > |3
>> > 1464 print("####### 3")
>> > |3
>> > 1465 import SCons.Util
>> > |3
>> > 1466 newsig = SCons.Util.MD5signature(contents)
>> > |3
>> > 1467 if bi.bactsig != newsig:
>> > |3
>> > 1468 if t: Trace(': bactsig %s != newsig %s' %
>> > (bi.bactsig, newsig)) |3
>> > 1469 print("####### 3.1")
>> > |3
>> > 1470 result = True
>> > On Tue, Dec 4, 2018 at 11:24 PM Bill Deegan <bill at baddogconsulting.com> wrote:
>> > >
>> > > Generated by logic in your sconstruct/sconscripts? or do you shell out and run a script/program to generate it.
>> > >
>> > >
>> > > On Tue, Dec 4, 2018 at 4:37 PM Hua Yanghao <huayanghao at gmail.com> wrote:
>> > >>
>> > >> Hi Bill,
>> > >> config.h is generated file, not checked in, it is generated in the
>> > >> build folder only.
>> > >> Well at least I see scons is very odering-sensitive. For example, if
>> > >> the order of SConscript processing changes (in the case of a set()),
>> > >> even if there is no dependency actually changed, scons thinks the
>> > >> target need to be rebuilt.
>> > >>
>> > >> any progress on these similar issues already? :)
>> > >>
>> > >> Best Regards,
>> > >> Yanghao
>> > >> On Tue, Dec 4, 2018 at 10:15 PM Bill Deegan <bill at baddogconsulting.com> wrote:
>> > >> >
>> > >> > Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in?
>> > >> >
>> > >> > It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase.
>> > >> >
>> > >> > We're actively debugging similar issue.
>> > >> >
>> > >> > -Bill
>> > >> >
>> > >> > On Tue, Dec 4, 2018 at 3:23 PM Hua Yanghao <huayanghao at gmail.com> wrote:
>> > >> >>
>> > >> >> I does that a lot ... but a quick try to make sure no
>> > >> >> LIBS/LIBPATH/CPPPATH etc. is touched after Object() is being called,
>> > >> >> still shows the same 50~70% rebuild of the entire project.
>> > >> >> I am not using any parallel build, and somehow scons is having a very
>> > >> >> none-deterministic behavior.
>> > >> >>
>> > >> >> Actually I fixed partially the issue (e.g. in simpler build the
>> > >> >> incremental builds now works fine), due to python set() is having a
>> > >> >> random order for every run. I am thinking maybe scons is somehow also
>> > >> >> depends on the ordering of a set()? And when it happens to be the same
>> > >> >> order then no rebuild?
>> > >> >> On Tue, Dec 4, 2018 at 9:00 PM Oscar Anson <oscar at solidangle.com> wrote:
>> > >> >> >
>> > >> >> > Hi,
>> > >> >> >
>> > >> >> > Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library.
>> > >> >> >
>> > >> >> > El mar., 4 dic. 2018 20:55, Hua Yanghao <huayanghao at gmail.com> escribió:
>> > >> >> >>
>> > >> >> >> Dear Scons Experts,
>> > >> >> >> I recently hit on a very strange problem where scons incremental
>> > >> >> >> builds sometimes OK, sometimes the entire project is re-built.
>> > >> >> >>
>> > >> >> >> scons: 3.0.1
>> > >> >> >> python: 3.6.7
>> > >> >> >>
>> > >> >> >> --debug=explain gives below example trace:
>> > >> >> >> scons: rebuilding
>> > >> >> >> `build/configs/qemu_arm_vexpress/arm3/config/config.h' because
>> > >> >> >> AlwaysBuild() is specified
>> > >> >> >> __CONFIG build/configs/qemu_arm_vexpress/arm3/config/config.h
>> > >> >> >> scons: rebuilding
>> > >> >> >> `build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o'
>> > >> >> >> because:
>> > >> >> >> `firmware/arch/armv7/arch_init.c' is no longer a dependency
>> > >> >> >> `firmware/arch/armv7/arch_init.c' is a new dependency
>> > >> >> >>
>> > >> >> >> It seems some how the C file itself is removed and re-added as the
>> > >> >> >> Object() dependency, and causes rebuild. However this is *NOT*
>> > >> >> >> *always* the case. Sometime the rebuild doesn't happen.
>> > >> >> >>
>> > >> >> >> I failed to create a minimal example that can show this issue so far,
>> > >> >> >> but in the production environment it is about 50% full-rebuild and 50%
>> > >> >> >> incremental re-build.
>> > >> >> >>
>> > >> >> >> Any thoughts or hints?
>> > >> >> >>
>> > >> >> >> Thanks,
>> > >> >> >> Yanghao
>> > >> >> >> _______________________________________________
>> > >> >> >> Scons-users mailing list
>> > >> >> >> Scons-users at scons.org
>> > >> >> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> > >> >> >
>> > >> >> > _______________________________________________
>> > >> >> > Scons-users mailing list
>> > >> >> > Scons-users at scons.org
>> > >> >> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> > >> >> _______________________________________________
>> > >> >> Scons-users mailing list
>> > >> >> Scons-users at scons.org
>> > >> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> > >> >
>> > >> > _______________________________________________
>> > >> > Scons-users mailing list
>> > >> > Scons-users at scons.org
>> > >> > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> > >> _______________________________________________
>> > >> Scons-users mailing list
>> > >> Scons-users at scons.org
>> > >> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> > >
>> > > _______________________________________________
>> > > Scons-users mailing list
>> > > Scons-users at scons.org
>> > > https://pairlist4.pair.net/mailman/listinfo/scons-users
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
More information about the Scons-users
mailing list