[Scons-users] Problems with SCons parallel build

[Robert Zeigler]

I think your success with -j 4 after removing the cd is a red herring. cd in a command works with parallel builds, and I have access to resources where the parallelization allows for dozens (and even hundreds) of parallel jobs.

Robert

On Mar 21, 2014, at 3/215:38 PM , Brady Johnson <bradyallenjohnson at gmail.com> wrote:


> Sorry guys, I spoke too soon. It worked with "-j 4", but then I got greedy and tried with "-j 12" and "-j 8" and it failed again, saying source files dont exist, even though they do indeed exist, and they are not generated. Seems like the "cd" in a Command() was one problem, because it wouldnt work before with -j 2 nor -j 4. Now there's something else.

> 

> Regards,

> 

> Brady

> 

> 

> 

> On Fri, Mar 21, 2014 at 11:30 PM, Brady Johnson <bradyallenjohnson at gmail.com> wrote:

> Wow, I fixed it!! Actually a few emails ago when I reported that removing all the calls to cd from the Command() builders failed, it failed because of what I commented out, later parts of the build needed these files. I changed all the calls to "cd /some/dest/dir ; jar xf /some/file" with "unzip -d /some/dest/dir /some/file" (no cd's anywhere) and now it works great with "-j 4"

> 

> So then either Command() isnt creating a sub process, or the child process is somehow affecting the parent. Either way, calling "cd" in a Command() causes parallel building to not work.

> 

> Thanks for your help Gary, Dirk, and Evan!!

> 

> Regards,

> 

> Brady

> 

> 

> 

> On Fri, Mar 21, 2014 at 10:56 PM, Evan Driscoll <driscoll at cs.wisc.edu> wrote:

> On 2014-03-21 16:50, Brady Johnson wrote:

> I commented out all the cd's from the Command() builders, and its

> still failing.

> 

> Something else that I just considered: I build several RPMs by

> calling os.system('rpmbuild...') directly instead of calling the

> SCons rpm builder. I wonder if rpmbuild changes directories,

> subsequently changing it for the SCons process. Anybody know if this

> could cause problems?

> 

> A child process can't change the directory of the parent (or, maybe there's a way if you try really hard and jump through hoops or something). That shouldn't be the cause.

> 

> One thought you could try: change env['ENV']['PATH'] so that x86_64-dx-linux-gnu-g++ is a script you write that outputs the current directory before running. Would at least tell you if this is a wild goose chase. :-)

> 

> Evan

> 

> 

> 

> Thanks,

> 

> Brady

> 

> On Fri, Mar 21, 2014 at 10:30 PM, Brady Johnson

> <bradyallenjohnson at gmail.com> wrote:

> 

> The only cd in the entire project is in the aforementioned call to

> Command(), chdir isn't used anywhere, neither via os nor via

> builders.

> 

> Regards,

> 

> Brady

> 

> On Mar 21, 2014 10:26 PM, "Evan Driscoll" <driscoll at cs.wisc.edu>

> wrote:

> On 2014-03-21 16:04, Gary Oberbrunner wrote:

> "cd" in a command is OK, since that runs in a subprocess.  But a

> python chdir() would be very bad, since the whole process has only

> one

> current dir, so you're changing it for all threads if you do that.

> 

> Don't forget chdir arguments to builders too.

> 

> In other words, when using a parallel build:

> 

> 1) Command("cd blah && whatever") is OK

> 2) AnyBuilder(..., chdir=1) or AnyBuilder(..., chdir='blah') is NOT

> 3) os.chdir() from a Python function called by a builder is NOT

> 

> Evan

> 

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users [1]

> 

> 

> 

> Links:

> ------

> [1] http://four.pairlist.net/mailman/listinfo/scons-users

> 

> 

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

> 

> 

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users