[Scons-users] SCons working with Ninja

Thu Apr 15 12:56:10 EDT 2021

Hi Bill,

I think it would be great to be a beta tester of this, but will need to check with my leads to make sure we can spare the resourced.  I’ll get back to you on that.  Also, it will obviously only be if we can actually use the feature, which we may not be able to if it doesn’t support function actions.

I’m not sure of the details of the post-link processing, but they are mostly doing file munging with Python scripts.  We have spoken with the teams to convert their tools to a command line form, but they’re fairly reluctant to do so.  Having the ability to call Python functions as actions is a great feature, and no one wants to give that up.

In our build system, parsing through our SConscript files takes a significant amount of time, and that’s part of what we were hoping to avoid with the Ninja builds.  I think we’d need a way to save the environments for each action, if we’re to have to call back into SCons.  Is that the approach you guys are taking here?

Thanks,
Don

From: Scons-users <scons-users-bounces at scons.org> On Behalf Of Bill Deegan
Sent: Wednesday, April 14, 2021 8:14 PM
To: SCons users mailing list <scons-users at scons.org>
Subject: [EXT] Re: [Scons-users] SCons working with Ninja

The work in progress is generously contributed and upstreaming sponsored  by MongoDB, so the reference to someone's packaging of such would probably be an earlier version of that code.

As I understand it many of the build steps as possible are exported to the ninja build file, the remainder will call back into SCons to build.

Don - I'd be curious if you could share what post link steps have to be implemented in python? Are they generating command lines? or doing file munging in python?

The plan is to add it to the next (4.1.1 or 4.2.0.. I need to review the included changes since 4.1.0), it will NOT be enabled by default.
As the code has only been used in only one production environment, I expect there'll be a lot of bumps in the road before it's ready for general usage.

It's possible the implementation may be significantly changed before it's enabled by default.

If you're willing to be a PATIENT "beta" user and even contribute fixes.. , then we'd be glad to have the help.
If not, then you're better waiting until it's had more usage to try it out.

Further, if you already have an implementation (based on the mongodb work, or not) and can share any lessoned and code learned, that'd be great.

We don't really have the resources to fully support this at the moment, thus the "experimental feature" approach was chosen as a way to expose it to motivated parties who might help us move it forward to production quality.

-Bill
SCons Project Co-Manager.

On Wed, Apr 14, 2021 at 1:56 PM Don Baldwin <donb at qti.qualcomm.com<mailto:donb at qti.qualcomm.com>> wrote:
Thanks Gary.  However, in our case, we have several post-link steps that are carried out as Python functions, so they are an unavoidable part of the build process.

-Don

-----Original Message-----
From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> On Behalf Of Gary Granger
Sent: Wednesday, April 14, 2021 1:22 PM
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: [EXT] Re: [Scons-users] SCons working with Ninja

Hello Don,

For whatever it's worth, we have a tool to generate ninja files from SCons, but I did not try to solve the problem of running python functions from ninja.  Instead, the tool generates a ninja build file from the nodes, and then it lets scons build just the nodes which do not use command line actions.  Most of our python function builders generate source files and version header files, so they are the leaves of the dependency graph and it doesn't matter if ninja cannot build them. Building the few python actions establishes a 'build baseline', then ninja builds the rest.  Obviously not complete, since someone has to remember to rebuild the ninja file if the python dependencies change. Although I suppose the tool could be modified to list all the python targets with a rule to re-run scons to build just those.  And maybe this does not help you, if most of your actions are not command lines.

That said, we don't use this much because in practice it's easier to just stick with scons for everything.

https://github.com/NCAR/eol_scons/blob/master/eol_scons/tools/ninja.py

The ninja support mentioned by Mats is surprising news, I'll be interested to see how that works.

Gary

On 4/14/21 1:42 PM, Don Baldwin wrote:
> Thanks Mats.  Will this be implemented in SCons 3.x as well as 4.x?  We are planning on moving to 4.x fairly soon, but we aren't there yet, and will probably not move our older projects to 4.x.  Also, do you have an ETA of when this will be available?
>
> Also, what about existing tools that claim to support generating Ninja scripts from SCons builds, such as:
>
>     * https://github.com/RedBeard0531/mongo_module_ninja
>
> Does anyone in this group know if that supports function actions?
>
> Lastly, we've been able to implement our own home-grown tools for doing a build "play-back" similar to what Ninja does, but we've only gotten it to work for command line actions.  I think the main hurdle for function actions is reproducing the SCons environment.  We've tried pickling it using both the native pickle library and 'dill', but have encountered infinite loop errors and errors trying to pickle class objects.  Has anyone else attempted this and gotten past these errors?  This is using SCons 3.1.2.
>
> Thanks,
> Don

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210415/281968d2/attachment-0001.htm>