[Scons-users] SCons working with Ninja

Thu Apr 15 19:29:47 EDT 2021

Don,

Pre py3, it wasn't uncommon that file operations done by in process python
actions could yield race conditions (esp on windows) where the file wasn't
fully closed before the next process would try to open in.
I'm fairly certain py3 improved file handling a bit and that issue is past.

-Bill

On Thu, Apr 15, 2021 at 4:17 PM Bill Deegan <bill at baddogconsulting.com>
wrote:

> Don,
>
> If you'd like to just try it and report back how broken it is.. that could
> be useful too.
> Just as long as the expectation is that it might take a while before it's
> handled.
>
> If you need external help for a refactor, I can probably recommend a
> consultant for such... ;)
>
> -Bill
>
> On Thu, Apr 15, 2021 at 2:29 PM Don Baldwin <donb at qti.qualcomm.com> wrote:
>
>> Thanks for the feedback Bill.  I think you’re right about the amount of
>> processing we’re doing in our SConscript files.  I’m going to propose that
>> we start considering a re-design, and if you don’t mind, I’ll use your
>> email below as an argument.  Your point about Python not having true
>> multi-threading is also a good argument against using Python functions, so
>> thanks for that as well.
>>
>>
>>
>> Our build scripts are very distributed, and no one person knows
>> everything that is happening in them.  Therefore, you’re probably right as
>> well about us not being a good candidate for alpha testing the Ninja
>> generator.
>>
>>
>>
>> Thanks for your feedback.
>>
>>
>>
>> -Don
>>
>>
>>
>>
>>
>> *From:* Scons-users <scons-users-bounces at scons.org> * On Behalf Of *Bill
>> Deegan
>> *Sent:* Thursday, April 15, 2021 1:54 PM
>> *To:* SCons users mailing list <scons-users at scons.org>
>> *Subject:* [EXT] Re: [Scons-users] SCons working with Ninja
>>
>>
>>
>> Don,
>>
>>
>>
>> But if you're willing to give it a shot and report back your issues we
>> can put them into the queue..
>>
>>
>>
>> -Bill
>>
>>
>>
>> On Thu, Apr 15, 2021 at 1:52 PM Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>> Don,
>>
>>
>>
>> If processing your sconscript files is taking a significant amount of
>> time, you're almost always doing something wrong.
>>
>> Have you run with --debug=time? What output do you get?
>>
>>
>>
>> Also using heavyweight function actions will slow down the build, There's
>> no real threading in python due to GIL.
>>
>> Another reason why it's best to move such out of process if possible.
>>
>> I think while this would work with scons/ninja, it would be vastly slower
>> than moving that logic into scripts..
>>
>>
>>
>> This is IMHO one of the most common mistakes users make.
>>
>> SCons is python great, let me make a big blob of python logic in process
>> to do this thing.
>>
>> That would not be something you could do with most other build systems,
>> and for most such, it's not something  you should do.
>>
>>
>>
>> Honestly it's more like alpha tester..
>>
>> You're probably not a great candidate for the initial test.
>>
>>
>>
>> I'd prefer a more or less vanilla c/c++ build.
>>
>>
>>
>>
>>
>> On Thu, Apr 15, 2021 at 9:56 AM Don Baldwin <donb at qti.qualcomm.com>
>> wrote:
>>
>> Hi Bill,
>>
>>
>>
>> I think it would be great to be a beta tester of this, but will need to
>> check with my leads to make sure we can spare the resourced.  I’ll get back
>> to you on that.  Also, it will obviously only be if we can actually use the
>> feature, which we may not be able to if it doesn’t support function actions.
>>
>>
>>
>> I’m not sure of the details of the post-link processing, but they are
>> mostly doing file munging with Python scripts.  We have spoken with the
>> teams to convert their tools to a command line form, but they’re fairly
>> reluctant to do so.  Having the ability to call Python functions as actions
>> is a great feature, and no one wants to give that up.
>>
>>
>>
>> In our build system, parsing through our SConscript files takes a
>> significant amount of time, and that’s part of what we were hoping to avoid
>> with the Ninja builds.  I think we’d need a way to save the environments
>> for each action, if we’re to have to call back into SCons.  Is that the
>> approach you guys are taking here?
>>
>>
>>
>> Thanks,
>>
>> Don
>>
>>
>>
>>
>>
>> *From:* Scons-users <scons-users-bounces at scons.org> *On Behalf Of *Bill
>> Deegan
>> *Sent:* Wednesday, April 14, 2021 8:14 PM
>> *To:* SCons users mailing list <scons-users at scons.org>
>> *Subject:* [EXT] Re: [Scons-users] SCons working with Ninja
>>
>>
>>
>> The work in progress is generously contributed and upstreaming sponsored
>> by MongoDB, so the reference to someone's packaging of such would probably
>> be an earlier version of that code.
>>
>>
>>
>> As I understand it many of the build steps as possible are exported to
>> the ninja build file, the remainder will call back into SCons to build.
>>
>>
>>
>> Don - I'd be curious if you could share what post link steps have to be
>> implemented in python? Are they generating command lines? or doing file
>> munging in python?
>>
>>
>>
>> The plan is to add it to the next (4.1.1 or 4.2.0.. I need to review the
>> included changes since 4.1.0), it will NOT be enabled by default.
>>
>> As the code has only been used in only one production environment, I
>> expect there'll be a lot of bumps in the road before it's ready for general
>> usage.
>>
>>
>>
>> It's possible the implementation may be significantly changed before it's
>> enabled by default.
>>
>>
>>
>> If you're willing to be a PATIENT "beta" user and even contribute fixes..
>> , then we'd be glad to have the help.
>>
>> If not, then you're better waiting until it's had more usage to try it
>> out.
>>
>>
>>
>> Further, if you already have an implementation (based on the mongodb
>> work, or not) and can share any lessoned and code learned, that'd be great.
>>
>>
>>
>> We don't really have the resources to fully support this at the moment,
>> thus the "experimental feature" approach was chosen as a way to expose it
>> to motivated parties who might help us move it forward to production
>> quality.
>>
>>
>>
>> -Bill
>>
>> SCons Project Co-Manager.
>>
>>
>>
>>
>>
>> On Wed, Apr 14, 2021 at 1:56 PM Don Baldwin <donb at qti.qualcomm.com>
>> wrote:
>>
>> Thanks Gary.  However, in our case, we have several post-link steps that
>> are carried out as Python functions, so they are an unavoidable part of the
>> build process.
>>
>> -Don
>>
>> -----Original Message-----
>> From: Scons-users <scons-users-bounces at scons.org> On Behalf Of Gary
>> Granger
>> Sent: Wednesday, April 14, 2021 1:22 PM
>> To: SCons users mailing list <scons-users at scons.org>
>> Subject: [EXT] Re: [Scons-users] SCons working with Ninja
>>
>> Hello Don,
>>
>> For whatever it's worth, we have a tool to generate ninja files from
>> SCons, but I did not try to solve the problem of running python functions
>> from ninja.  Instead, the tool generates a ninja build file from the nodes,
>> and then it lets scons build just the nodes which do not use command line
>> actions.  Most of our python function builders generate source files and
>> version header files, so they are the leaves of the dependency graph and it
>> doesn't matter if ninja cannot build them. Building the few python actions
>> establishes a 'build baseline', then ninja builds the rest.  Obviously not
>> complete, since someone has to remember to rebuild the ninja file if the
>> python dependencies change. Although I suppose the tool could be modified
>> to list all the python targets with a rule to re-run scons to build just
>> those.  And maybe this does not help you, if most of your actions are not
>> command lines.
>>
>> That said, we don't use this much because in practice it's easier to just
>> stick with scons for everything.
>>
>> https://github.com/NCAR/eol_scons/blob/master/eol_scons/tools/ninja.py
>>
>> The ninja support mentioned by Mats is surprising news, I'll be
>> interested to see how that works.
>>
>> Gary
>>
>> On 4/14/21 1:42 PM, Don Baldwin wrote:
>> > Thanks Mats.  Will this be implemented in SCons 3.x as well as 4.x?  We
>> are planning on moving to 4.x fairly soon, but we aren't there yet, and
>> will probably not move our older projects to 4.x.  Also, do you have an ETA
>> of when this will be available?
>> >
>> > Also, what about existing tools that claim to support generating Ninja
>> scripts from SCons builds, such as:
>> >
>> >     * https://github.com/RedBeard0531/mongo_module_ninja
>> >
>> > Does anyone in this group know if that supports function actions?
>> >
>> > Lastly, we've been able to implement our own home-grown tools for doing
>> a build "play-back" similar to what Ninja does, but we've only gotten it to
>> work for command line actions.  I think the main hurdle for function
>> actions is reproducing the SCons environment.  We've tried pickling it
>> using both the native pickle library and 'dill', but have encountered
>> infinite loop errors and errors trying to pickle class objects.  Has anyone
>> else attempted this and gotten past these errors?  This is using SCons
>> 3.1.2.
>> >
>> > Thanks,
>> > Don
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210415/24963ddb/attachment-0001.htm>