[Scons-users] Batching behavior of the Parallel class in Jobs.py

[Andrew C. Morrow]

I noticed something a little odd, maybe, in the Parallel class:

https://bitbucket.org/scons/scons/src/ea74a7f7a61561bebfbbfd5e7e70c4c346dd8bdb/src/engine/SCons/Job.py?at=default&fileviewer=file-view-default#Job.py-403:426

The way I read this, Jobs are processed in 'batches', but this doesn't seem
to be necessary. To experiment with this, I added prints at lines 396 and
407 to print the number of jobs. The current behavior when I do a -j12
build looks like:

('XXX acm', 1)
('XXX acm', 2)
('XXX acm', 3)
('XXX acm', 4)
('XXX acm', 5)
('XXX acm', 6)
('XXX acm', 7)
('XXX acm', 8)
('XXX acm', 9)
('XXX acm', 10)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 10)
('XXX acm', 9)
('XXX acm', 8)
('XXX acm', 7)
('XXX acm', 6)
('XXX acm', 5)
('XXX acm', 4)
('XXX acm', 3)
('XXX acm', 2)
('XXX acm', 3)
('XXX acm', 4)
('XXX acm', 5)
('XXX acm', 6)
('XXX acm', 7)
('XXX acm', 8)
('XXX acm', 9)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 10)
('XXX acm', 9)
('XXX acm', 8)
('XXX acm', 7)
('XXX acm', 6)
('XXX acm', 5)
('XXX acm', 4)
('XXX acm', 3)
('XXX acm', 2)
('XXX acm', 3)
('XXX acm', 5)
('XXX acm', 6)
('XXX acm', 7)
('XXX acm', 8)
('XXX acm', 9)
('XXX acm', 10)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 10)
('XXX acm', 9)
('XXX acm', 8)
('XXX acm', 7)
('XXX acm', 6)
('XXX acm', 5)
('XXX acm', 4)
('XXX acm', 3)
('XXX acm', 2)
('XXX acm', 1)
('XXX acm', 2)
('XXX acm', 3)
^C('XXX acm', 2)
('XXX acm', 1)
('XXX acm', 0)

Or something like that. This seems to suggest that even during a -j12
build, there are frequently well fewer than 12 jobs in flight.

Chaning line 425 to say "if True", so that we always execute the break on
426, and therefore always enqueue a new job if on is available as soon as
any one job completes, results in a pattern like:

'XXX acm', 1)
('XXX acm', 2)
('XXX acm', 3)
('XXX acm', 4)
('XXX acm', 5)
('XXX acm', 6)
('XXX acm', 7)
('XXX acm', 8)
('XXX acm', 9)
('XXX acm', 10)
('XXX acm', 11)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
'XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
'XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
'XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
('XXX acm', 12)
('XXX acm', 11)
'XXX acm', 12)
('XXX acm', 11)
^C('XXX acm', 10)
('XXX acm', 9)
('XXX acm', 8)
('XXX acm', 7)
('XXX acm', 6)
('XXX acm', 5)
('XXX acm', 4)
('XXX acm', 3)
('XXX acm', 2)
('XXX acm', 1)
('XXX acm', 0)

That at least seems like it would be better, to me. I haven't tried to
measure any sort of throughput change, yet.

Can anyone provide some background on why the Parallel class works this
way? Is there any reason not to just remove the loop on line 405 and deal
with the single job completion then go back to the well for more?

Thanks,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161005/949b494f/attachment.html>