[Scons-users] Prioritization of builds

Wed Jan 20 09:07:42 EST 2016

That's an interesting idea to use a heuristic, but I'm not sure how well it
would work in practice. The real issue isn't the Node cost, but the length
of the source -> sink path I think because those targets must be built is
serial.

Possible issues:
1. You assume that target rebuild cost is always ~same as the last rebuild;
may be reasonable for compilation, but may not be reasonable for modelling.
Sometimes the change of a single parameter can change the performance
greatly.
2. A full rebuild is still retarded, since the heuristic h(n) = 0.

When trying to optimize builds in the past, especially in highly parallel
contexts (40+ cores), that the build performance boils down to the slowest
individual Node in the build tree. What end up happening is that the
longest "build" node almost always becomes a bottleneck holding up some
library construction or other important dependency because not all
dependencies are equal in build priority.

When I was working on a legacy system where developers were still using
highly functional programming, some of the files had 10's of thousands of
line of code because they had no sense of organization. This lead to some
files taking 30+ seconds to compile. I found that by breaking down a dozen
of these file(s) into parts that it reduced the wallclock time of my build
by almost 50%.

I think that building the models in parallel is far more important than
trying to shuffle them to the front; without seeing a build profile, I'm
still not sure that his models weren't executing as soon as they were
available. We need to know how long his total build takes versus how long
it takes to build all the models in serial + the time it takes to build the
model dependencies. I expect that we will find that the vast majority of
time is a serial problem.

V/R,
William

On Wed, Jan 20, 2016 at 12:46 PM, Michael Potter <michael at potter.name>
wrote:

> What if SCons tracked how long each step took, then optimized the
> order on subsequent builds?
>
> On Sat, Jan 16, 2016 at 8:53 AM, William Blevins <wblevins001 at gmail.com>
> wrote:
> > It looks like ModelSim does provide a FlexNet (previously FlexLM) license
> > solution. It might be worth investigating. If you are using hardware
> > security key, I am surprised that you cannot run more than one instance.
> I
> > always thought the point was to tie to software to a particular
> machine...
> >
> > On Sat, Jan 16, 2016 at 1:25 PM, Brian Cody <brian.j.cody at gmail.com>
> wrote:
> >>
> >> The tool in question is Modelsim. We have license dongles--no server. If
> >> Modelsim is currently running a simulation on a given computer and
> another
> >> instance is started on the same computer, the command will fail with an
> >> error return status.
> >>
> >> Side effect seemed to be appropriate in this case!
> >>
> >> On Jan 15, 2016 10:40 PM, "Bill Deegan" <bill at baddogconsulting.com>
> wrote:
> >>>
> >>> Most flexlm licensed tools can enable this. Though it's up to the
> >>> developer to do so.
> >>> (I've deployed flexlm at a number of clients for their software
> licensing
> >>> (on the software producer side))
> >>>
> >>> -Bill
> >>>
> >>> On Fri, Jan 15, 2016 at 5:57 PM, William Blevins <
> wblevins001 at gmail.com>
> >>> wrote:
> >>>>
> >>>> I have used tools like this before and did not require a side-effect
> >>>> file. For example, Intel compilers require a license server. Even
> with a
> >>>> single floating license, I can run jobs in parallel without an issue
> because
> >>>> license requests are queued by the license server, so this is handled
> >>>> outside of SCons. What tool are you using?
> >>>>
> >>>> V/R,
> >>>> William
> >>>>
> >>>> On Fri, Jan 15, 2016 at 6:02 PM, Brian Cody <brian.j.cody at gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hello fellow sconsians,
> >>>>>
> >>>>> Let's say I have 1000 targets to build. There are some dependencies
> >>>>> between them but over all there's a considerable amount of
> parallelism
> >>>>> possible. Now let's say 10 of those targets require the use of a
> licensed
> >>>>> tool which can only run one at a time. I use a side effect to
> enforce that
> >>>>> with those 10 targets. I typically run with -j12 or so, one for each
> virtual
> >>>>> core available. This works, and I'm happy.
> >>>>>
> >>>>> Except.. Those 10 builds I mentioned just happen to be very slow. The
> >>>>> final target built will always be one of those ten. That's fine.
> However,
> >>>>> the build will take much longer than necessary if it doesn't happen
> to start
> >>>>> building those targets early. And doing that would require that their
> >>>>> dependencies are also built early. However for some reason, these
> targets
> >>>>> seem to always get started towards the end of the build.
> >>>>>
> >>>>> Today this is a minor annoyance. It may be adding 30 seconds to a 1
> >>>>> minute 30 second build. Looking in my crystal ball this problem will
> be
> >>>>> getting much much worse with some of the new targets that are coming.
> >>>>>
> >>>>> Is there a known way to influence the order at which targets are
> built
> >>>>> without introducing phony dependencies or side effects? So if A and
> B are
> >>>>> both ready to built right now, and one job-doer becomes available,
> is there
> >>>>> any way I can cause SCons to pick them up in a particular order?
> Maybe by
> >>>>> influencing a container holding onto targets?
> >>>>>
> >>>>> Thank you!
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Scons-users mailing list
> >>>>> Scons-users at scons.org
> >>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Scons-users mailing list
> >>>> Scons-users at scons.org
> >>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Scons-users mailing list
> >>> Scons-users at scons.org
> >>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>>
> >>
> >> _______________________________________________
> >> Scons-users mailing list
> >> Scons-users at scons.org
> >> https://pairlist4.pair.net/mailman/listinfo/scons-users
> >>
> >
> >
> > _______________________________________________
> > Scons-users mailing list
> > Scons-users at scons.org
> > https://pairlist4.pair.net/mailman/listinfo/scons-users
> >
>
>
>
> --
> Michael Potter
>   Tapp Solutions, LLC
>    www.tappsolutions.com
> +1 770 815 6142  ** Atlanta ** michael at potter.name  **
> www.linkedin.com/in/michaelpotter
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20160120/ea38cc72/attachment-0001.html>