[Scons-users] modifying/extending SCons to run jobs on Torque cluster

[Thomas Lippincott]

That's basically what I'm doing at the moment, and this works for
command-line actions, with one SCons instance taking care of the build
information.  But this doesn't work for non-command-line actions,
because they have no command-line corollary other than running SCons
itself. Right now, the setup also requires a separate thread to monitor
each submitted job, which is OK for when you have, say, 8 independent,
long-running actions, but not when you have several hundred
medium-length actions, deep dependency trees, etc.

Torque has functionality for making jobs depend on one another, so
ideally it would be possible to map the SCons dependency tree into
Torque dependencies, submit them along with one job that depends on all
the leaf dependencies, and just monitor that.  Then no multithreading
would be necessary at the submit-node.  In general, access to the
dependency graph would open up possibilities for a lot of
experimentation with different approaches, but it sounds like that part
of the code base isn't great to work with.

-Tom

On 10/04/2014 09:24 PM, Bill Deegan wrote:
> Thomas,
> 
> Can't torque submit jobs and wait or their completion?
> If so the master SCons's process could maintain enough information and the
> need to merge the sconsigns would be removed..
> 
> -Bill
> 
> On Sat, Oct 4, 2014 at 1:43 PM, Thomas Lippincott <tom.lippincott at gmail.com>
> wrote:
> 
>> This would be OK for my purposes, if "global" means a given invocation
>> of the primary scons script. When scons is invoked on the submit node,
>> it might decide to build the tree by submitting three jobs, which run
>> "scons job1-target1 job1-target2...", "scons job2-target1
>> job2-target2...", etc, on worker nodes.  The submit-node's scons
>> instance copies ".sconsign.db" to ".sconsign.JOB-1", ".sconsign.JOB-2",
>> etc, then submits the jobs to the scheduler and starts waiting for them
>> to complete.  Meanwhile, worker nodes execute the jobs, but when running
>> on a worker node the scons script first calls
>> SConsignFile(".sconsign.JOB-ID").  Back at the submit-node's scons
>> instance, once the worker nodes have finished, scons needs to read all
>> the ".sconsign.JOB-ID" files and merge them back into ".sconsign.db". It
>> sounds like this final merging functionality is the main work (in theory).
>> -Tom
>>
>>
>> On 10/04/2014 02:13 PM, Bill Deegan wrote:
>>> The SConsignFile() is global right?
>>> I don't think you can do that per environment or per builder.
>>>
>>> -Bill
>>>
>>> On Sat, Oct 4, 2014 at 1:46 AM, Dirk Bächle <tshortik at gmx.de> wrote:
>>>
>>>>  Tom,
>>>>
>>>> On 04.10.2014 06:42, Bill Deegan wrote:
>>>>
>>>>  Is there functionality for directing the scons command to
>>>> use a different .sconsign.db file, and an API for
>>>> reading/merging/writing entries from different files?
>>>>
>>>>  No. (Unless of course you write it.. :)
>>>>
>>>>
>>>> there is the SConsignFile() method for redirecting the writing/reading
>> of
>>>> entries to a different filename. The rest of the current API can be
>> found
>>>> in "engine/SCons/SConsign.py"...
>>>>
>>>> Dirk
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> 
> 
> 
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>