[Scons-users] SCons working with Ninja

Don Baldwin donb at qti.qualcomm.com
Thu Apr 15 20:04:50 EDT 2021


Thanks for the offer Bill, and the info.  I’ll let you know if we decide to take you up on it.  😊

-Don

From: Scons-users <scons-users-bounces at scons.org> On Behalf Of Bill Deegan
Sent: Thursday, April 15, 2021 4:30 PM
To: SCons users mailing list <scons-users at scons.org>
Subject: [EXT] Re: [Scons-users] SCons working with Ninja

Don,

Pre py3, it wasn't uncommon that file operations done by in process python actions could yield race conditions (esp on windows) where the file wasn't fully closed before the next process would try to open in.
I'm fairly certain py3 improved file handling a bit and that issue is past.

-Bill

On Thu, Apr 15, 2021 at 4:17 PM Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:
Don,

If you'd like to just try it and report back how broken it is.. that could be useful too.
Just as long as the expectation is that it might take a while before it's handled.

If you need external help for a refactor, I can probably recommend a consultant for such... ;)

-Bill

On Thu, Apr 15, 2021 at 2:29 PM Don Baldwin <donb at qti.qualcomm.com<mailto:donb at qti.qualcomm.com>> wrote:
Thanks for the feedback Bill.  I think you’re right about the amount of processing we’re doing in our SConscript files.  I’m going to propose that we start considering a re-design, and if you don’t mind, I’ll use your email below as an argument.  Your point about Python not having true multi-threading is also a good argument against using Python functions, so thanks for that as well.

Our build scripts are very distributed, and no one person knows everything that is happening in them.  Therefore, you’re probably right as well about us not being a good candidate for alpha testing the Ninja generator.

Thanks for your feedback.

-Don


From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> On Behalf Of Bill Deegan
Sent: Thursday, April 15, 2021 1:54 PM
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: [EXT] Re: [Scons-users] SCons working with Ninja

Don,

But if you're willing to give it a shot and report back your issues we can put them into the queue..

-Bill

On Thu, Apr 15, 2021 at 1:52 PM Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>> wrote:
Don,

If processing your sconscript files is taking a significant amount of time, you're almost always doing something wrong.
Have you run with --debug=time? What output do you get?

Also using heavyweight function actions will slow down the build, There's no real threading in python due to GIL.
Another reason why it's best to move such out of process if possible.
I think while this would work with scons/ninja, it would be vastly slower than moving that logic into scripts..

This is IMHO one of the most common mistakes users make.
SCons is python great, let me make a big blob of python logic in process to do this thing.
That would not be something you could do with most other build systems, and for most such, it's not something  you should do.

Honestly it's more like alpha tester..
You're probably not a great candidate for the initial test.

I'd prefer a more or less vanilla c/c++ build.


On Thu, Apr 15, 2021 at 9:56 AM Don Baldwin <donb at qti.qualcomm.com<mailto:donb at qti.qualcomm.com>> wrote:
Hi Bill,

I think it would be great to be a beta tester of this, but will need to check with my leads to make sure we can spare the resourced.  I’ll get back to you on that.  Also, it will obviously only be if we can actually use the feature, which we may not be able to if it doesn’t support function actions.

I’m not sure of the details of the post-link processing, but they are mostly doing file munging with Python scripts.  We have spoken with the teams to convert their tools to a command line form, but they’re fairly reluctant to do so.  Having the ability to call Python functions as actions is a great feature, and no one wants to give that up.

In our build system, parsing through our SConscript files takes a significant amount of time, and that’s part of what we were hoping to avoid with the Ninja builds.  I think we’d need a way to save the environments for each action, if we’re to have to call back into SCons.  Is that the approach you guys are taking here?

Thanks,
Don


From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> On Behalf Of Bill Deegan
Sent: Wednesday, April 14, 2021 8:14 PM
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: [EXT] Re: [Scons-users] SCons working with Ninja

The work in progress is generously contributed and upstreaming sponsored  by MongoDB, so the reference to someone's packaging of such would probably be an earlier version of that code.

As I understand it many of the build steps as possible are exported to the ninja build file, the remainder will call back into SCons to build.

Don - I'd be curious if you could share what post link steps have to be implemented in python? Are they generating command lines? or doing file munging in python?

The plan is to add it to the next (4.1.1 or 4.2.0.. I need to review the included changes since 4.1.0), it will NOT be enabled by default.
As the code has only been used in only one production environment, I expect there'll be a lot of bumps in the road before it's ready for general usage.

It's possible the implementation may be significantly changed before it's enabled by default.

If you're willing to be a PATIENT "beta" user and even contribute fixes.. , then we'd be glad to have the help.
If not, then you're better waiting until it's had more usage to try it out.

Further, if you already have an implementation (based on the mongodb work, or not) and can share any lessoned and code learned, that'd be great.

We don't really have the resources to fully support this at the moment, thus the "experimental feature" approach was chosen as a way to expose it to motivated parties who might help us move it forward to production quality.

-Bill
SCons Project Co-Manager.


On Wed, Apr 14, 2021 at 1:56 PM Don Baldwin <donb at qti.qualcomm.com<mailto:donb at qti.qualcomm.com>> wrote:
Thanks Gary.  However, in our case, we have several post-link steps that are carried out as Python functions, so they are an unavoidable part of the build process.

-Don

-----Original Message-----
From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> On Behalf Of Gary Granger
Sent: Wednesday, April 14, 2021 1:22 PM
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: [EXT] Re: [Scons-users] SCons working with Ninja

Hello Don,

For whatever it's worth, we have a tool to generate ninja files from SCons, but I did not try to solve the problem of running python functions from ninja.  Instead, the tool generates a ninja build file from the nodes, and then it lets scons build just the nodes which do not use command line actions.  Most of our python function builders generate source files and version header files, so they are the leaves of the dependency graph and it doesn't matter if ninja cannot build them. Building the few python actions establishes a 'build baseline', then ninja builds the rest.  Obviously not complete, since someone has to remember to rebuild the ninja file if the python dependencies change. Although I suppose the tool could be modified to list all the python targets with a rule to re-run scons to build just those.  And maybe this does not help you, if most of your actions are not command lines.

That said, we don't use this much because in practice it's easier to just stick with scons for everything.

https://github.com/NCAR/eol_scons/blob/master/eol_scons/tools/ninja.py

The ninja support mentioned by Mats is surprising news, I'll be interested to see how that works.

Gary

On 4/14/21 1:42 PM, Don Baldwin wrote:
> Thanks Mats.  Will this be implemented in SCons 3.x as well as 4.x?  We are planning on moving to 4.x fairly soon, but we aren't there yet, and will probably not move our older projects to 4.x.  Also, do you have an ETA of when this will be available?
>
> Also, what about existing tools that claim to support generating Ninja scripts from SCons builds, such as:
>
>     * https://github.com/RedBeard0531/mongo_module_ninja
>
> Does anyone in this group know if that supports function actions?
>
> Lastly, we've been able to implement our own home-grown tools for doing a build "play-back" similar to what Ninja does, but we've only gotten it to work for command line actions.  I think the main hurdle for function actions is reproducing the SCons environment.  We've tried pickling it using both the native pickle library and 'dill', but have encountered infinite loop errors and errors trying to pickle class objects.  Has anyone else attempted this and gotten past these errors?  This is using SCons 3.1.2.
>
> Thanks,
> Don

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210416/270303d7/attachment-0001.htm>


More information about the Scons-users mailing list