[Scons-users] managing dependencies

[Arvid Rosén]

Hi again!

Regarding:
“BTW.. One way to which might work for you is to have your upload processes declare a common sideffect.
This will prevent SCons from running more than one of those steps at the same time.
That could be simpler than a large number of Depends()”

We do that too, but in most cases it doesn’t replace the need for scheduling large parallel builds with “Depends” or “Requires”, as it doesn’t help SCons to realize that it needs to focus on some targets early on to be able to start uploading the first finished artifact as soon as possible.

Sorry for the OT. But maybe this is interesting for someone else too. I have sometimes thought about adding scheduling as a feature request, but I don’t know if it would be practical to describe these kind of problems in a way in the SConscript file that would be clear and easy to understand.

Cheers,
Arvid


From: Scons-users <scons-users-bounces at scons.org> on behalf of Bill Deegan <bill at baddogconsulting.com>
Reply to: SCons users mailing list <scons-users at scons.org>
Date: Tuesday, 10 December 2019 at 17:13
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] managing dependencies

Sure in that case you don't have to.
By "have to" I mean, your missing dependencies and your build will fail because of that.

In your case it's for reasons external to the build itself.

BTW.. One way to which might work for you is to have your upload processes declare a common sideffect.
This will prevent SCons from running more than one of those steps at the same time.
That could be simpler than a large number of Depends()

On Mon, Dec 9, 2019 at 5:29 PM Arvid Rosén <arvid at softube.com<mailto:arvid at softube.com>> wrote:
We don’t “have to” do this, but we do it to speed up multi-hour builds. If we don’t do it (the real life project is 1000x more complex naturally), we end up with 150 products finishing roughly at the same time, which will max out the bandwidth available for uploading the finished artifacts to some server for example.

Actually, I think it is rather nice that you can do this in SCons, in a rather straight forward and clean way.

Cheers,
Arvid


From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> on behalf of Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>>
Reply to: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Date: Monday, 9 December 2019 at 16:55
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] managing dependencies

That said if you have to do this, you're likely doing something wrong.

If you have a custom builder, perhaps you're not adding all the output files in the emitter or in the emitter call.
SCons can't order dependencies it's not aware of.

-Bill

On Mon, Dec 9, 2019 at 3:26 AM Arvid Rosén <arvid at softube.com<mailto:arvid at softube.com>> wrote:
Well,

In SCons, there is no need to use an extra file for this purpose. You can declare dependencies without having files too.

I mean something like this:

A = Program(“a.cpp”)
B = Program(“b.cpp”)
C = Program(“c.cpp”)
D = Program(“d.cpp”)

Depends(C, A)
Depends(D, [A, B])

Builds of A and B starts immediately. C starts when A is finished. D starts when A and B have finished.

Being a python script, this can be made more general using for loops and stuff, and we typically have a variable somewhere to control how many products that can be in-flight simultaneously. Btw, this is a typical use case of “Requires” rather than “Depends” I guess, but I chose the latter for simplicity here.

Cheers,
Arvid



From: daggs <daggs at gmx.com<mailto:daggs at gmx.com>>
Date: Monday, 9 December 2019 at 09:04
To: Arvid Rosén <arvid at softube.com<mailto:arvid at softube.com>>
Cc: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] managing dependencies

Greetings,

something like for all short deps, depend them on a file which I can use touch to create?
will the same file for all of them do the job?

Thank,

Dagg.


Sent: Monday, December 09, 2019 at 9:38 AM
From: "Arvid Rosén" <arvid at softube.com<mailto:arvid at softube.com>>
To: "SCons users mailing list" <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] managing dependencies
Hi,

Maybe, but I certainly wouldn’t rely on that.

We do very massive builds with hundreds of targets in parallel. To avoid all targets finishing almost at the same time (competing with each other for the same limited resources), we add “fake” dependencies so to schedule the builds so that only a couple of targets can be worked on at the same time, but still 20 threads or so can be used. This is very easy to do in SCons.

Cheers,
Arvid

From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> on behalf of daggs <daggs at gmx.com<mailto:daggs at gmx.com>>
Reply to: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Date: Monday, 9 December 2019 at 08:06
To: "scons-users at scons.org<mailto:scons-users at scons.org>" <scons-users at scons.org<mailto:scons-users at scons.org>>
Cc: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] managing dependencies

Greetings,

does the order of dependecy declaration affects the order?

Dagg.


Sent: Monday, December 09, 2019 at 1:06 AM
From: "Bill Deegan" <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>>
To: "SCons users mailing list" <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] managing dependencies
Nope.
You specify the dependencies, SCons figures out the order.
If you need one thing to happen before another then that's a dependency you must explicitly list.

-Bill

On Sun, Dec 8, 2019 at 4:49 AM daggs <daggs at gmx.com<mailto:daggs at gmx.com>> wrote:
Greetings,

I wanted to ask about scons dependencies management, I have a target with large number of dependencies, all the dependencies don't have dependencies of their own.
some of the dependencies take more time to generate than the other ones.
I've noticed that not all the dependencies are handled at the start (I have a print at the start of each dependency handling).
is it possible to instruct scons to give an higher priority to a specific dependency?

Thanks,

Dagg.
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________ Scons-users mailing list Scons-users at scons.org<mailto:Scons-users at scons.org> https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________ Scons-users mailing list Scons-users at scons.org<mailto:Scons-users at scons.org> https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20191211/e8bf2b05/attachment-0001.html>