[Scons-users] Using scons with SGE

[Peter Kerpedjiev]

On 11/30/14, 8:26 PM, William Blevins wrote:
>
> I may misunderstand, but as long as your Action is blocking and the 
> sources from the complete dependency lists to your targets, then it 
> should work.
>
Right, it does work. It just spawns a huge number of threads and I end 
up with errors like this:

scons: *** [tests/1803/add.fasta] Resource temporarily unavailable


Which presumably have something to do with NFS. They don't occur if I 
lower the number of threads, but then I don't end up fully utilizing the 
cluster.

I guess I was asking if anybody had tried integrating it with SGE's job 
dependency scheduling. I.e. having cmd2 wait on cmd1, so that I'm not 
trying to run a very large number of threads on an incapable computer.

-Peter

> On Nov 30, 2014 2:14 PM, "Peter Kerpedjiev" <pkerpedjiev at gmail.com 
> <mailto:pkerpedjiev at gmail.com>> wrote:
>
>     Hi,
>
>     I would really like to use scons with the Sun Grid Engine and I'd
>     like to ask you all what the best way to do so is. The issue is
>     chaining build steps. For the most part, I use a series of
>     Commands to create a pipeline for data analysis. So in then end I
>     would have something like this:
>
>     inputs = ['in1.txt', 'in2.txt', 'in3.txt', 'in4.txt']
>
>     outputs = ['out1.txt', 'out2.txt', 'out3.txt', 'out4.txt']
>
>     for input,output in zip(inputs, outputs):
>
>         Command(target=output, source=input, action='blah $SOURCE >
>     $TARGET')
>
>     Command(target='sum.txt', source=outputs, action='blah2 $SOURCES >
>     $TARGET')
>
>
>     So the easiest way to run this on the SGE is to prefix the command
>     'blah' with the submit command (i.e. 'qsub blah $SOURCE >
>     $TARGET'). This creates a problem, however, since 'qsub' is
>     asynchronous and returns immediately. We can make it synchronous
>     using the '-s' option, but then we end up with the same problem as
>     having not used it in the first place, namely that we have to wait
>     for it to finish. The final option is to make it synchronous and
>     run scons with some number of threads roughly equal to how many
>     spots are available to the grid engine. This more or less works,
>     but seems a little bit hacky.
>
>     My question is, is there a preferred way of submitting jobs to a
>     grid engine with scons such that dependent steps wait on each
>     other and parallel steps are submitted at once?
>
>     -Peter
>     _______________________________________________
>     Scons-users mailing list
>     Scons-users at scons.org <mailto:Scons-users at scons.org>
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141130/04273688/attachment.html>