[Scons-users] Problems with SCons parallel build

[Brady Johnson]

At first I thought this problem was caused by the RPM generation that Im
doing by calling os.system('rpmbuild ...') and perhaps 'rpmbuild' could be
changing the directory, since the rpm spec files have lots of cd, mkdir,
etc.  But I realized that cant be it, since os.system() spawns a
subprocess. Upon further investigation, I found the problem to be
attributed to shutil.make_archive(). Looking at the python docs for the
utility: http://docs.python.org/2/library/shutil.html I saw the following:

shutil.make_archive(*base_name*, *format*[, *root_dir*[, *base_dir*[,

> *verbose*[, *dry_run*[, *owner*[, *group*[, *logger*]]]]]]])

>

> *...**root_dir* is a directory that will be the root directory of the

> archive; ie. we typically chdir into *root_dir* before creating the

> archive.

> ...


<http://docs.python.org/2/library/shutil.html#shutil.make_archive>
After changing "shutil.make_archive()" with "os.system('tar czf theFile.tgz
-C dir files)" it works great!! Ive tested it several times with "-j 12"
with no problems.

Is there someway we could document this? It would be nice to help people
avoid this problem in the future.

Regards,

Brady


On Sat, Mar 22, 2014 at 1:00 AM, Robert Zeigler <robert.zeigler at gmail.com>wrote:


> I think your success with -j 4 after removing the cd is a red herring. cd

> in a command works with parallel builds, and I have access to resources

> where the parallelization allows for dozens (and even hundreds) of parallel

> jobs.

>

> Robert

>

> On Mar 21, 2014, at 3/215:38 PM , Brady Johnson <

> bradyallenjohnson at gmail.com> wrote:

>

> > Sorry guys, I spoke too soon. It worked with "-j 4", but then I got

> greedy and tried with "-j 12" and "-j 8" and it failed again, saying source

> files dont exist, even though they do indeed exist, and they are not

> generated. Seems like the "cd" in a Command() was one problem, because it

> wouldnt work before with -j 2 nor -j 4. Now there's something else.

> >

> > Regards,

> >

> > Brady

> >

> >

> >

> > On Fri, Mar 21, 2014 at 11:30 PM, Brady Johnson <

> bradyallenjohnson at gmail.com> wrote:

> > Wow, I fixed it!! Actually a few emails ago when I reported that

> removing all the calls to cd from the Command() builders failed, it failed

> because of what I commented out, later parts of the build needed these

> files. I changed all the calls to "cd /some/dest/dir ; jar xf /some/file"

> with "unzip -d /some/dest/dir /some/file" (no cd's anywhere) and now it

> works great with "-j 4"

> >

> > So then either Command() isnt creating a sub process, or the child

> process is somehow affecting the parent. Either way, calling "cd" in a

> Command() causes parallel building to not work.

> >

> > Thanks for your help Gary, Dirk, and Evan!!

> >

> > Regards,

> >

> > Brady

> >

> >

> >

> > On Fri, Mar 21, 2014 at 10:56 PM, Evan Driscoll <driscoll at cs.wisc.edu>

> wrote:

> > On 2014-03-21 16:50, Brady Johnson wrote:

> > I commented out all the cd's from the Command() builders, and its

> > still failing.

> >

> > Something else that I just considered: I build several RPMs by

> > calling os.system('rpmbuild...') directly instead of calling the

> > SCons rpm builder. I wonder if rpmbuild changes directories,

> > subsequently changing it for the SCons process. Anybody know if this

> > could cause problems?

> >

> > A child process can't change the directory of the parent (or, maybe

> there's a way if you try really hard and jump through hoops or something).

> That shouldn't be the cause.

> >

> > One thought you could try: change env['ENV']['PATH'] so that

> x86_64-dx-linux-gnu-g++ is a script you write that outputs the current

> directory before running. Would at least tell you if this is a wild goose

> chase. :-)

> >

> > Evan

> >

> >

> >

> > Thanks,

> >

> > Brady

> >

> > On Fri, Mar 21, 2014 at 10:30 PM, Brady Johnson

> > <bradyallenjohnson at gmail.com> wrote:

> >

> > The only cd in the entire project is in the aforementioned call to

> > Command(), chdir isn't used anywhere, neither via os nor via

> > builders.

> >

> > Regards,

> >

> > Brady

> >

> > On Mar 21, 2014 10:26 PM, "Evan Driscoll" <driscoll at cs.wisc.edu>

> > wrote:

> > On 2014-03-21 16:04, Gary Oberbrunner wrote:

> > "cd" in a command is OK, since that runs in a subprocess.  But a

> > python chdir() would be very bad, since the whole process has only

> > one

> > current dir, so you're changing it for all threads if you do that.

> >

> > Don't forget chdir arguments to builders too.

> >

> > In other words, when using a parallel build:

> >

> > 1) Command("cd blah && whatever") is OK

> > 2) AnyBuilder(..., chdir=1) or AnyBuilder(..., chdir='blah') is NOT

> > 3) os.chdir() from a Python function called by a builder is NOT

> >

> > Evan

> >

> > _______________________________________________

> > Scons-users mailing list

> > Scons-users at scons.org

> > http://four.pairlist.net/mailman/listinfo/scons-users [1]

> >

> >

> >

> > Links:

> > ------

> > [1] http://four.pairlist.net/mailman/listinfo/scons-users

> >

> >

> > _______________________________________________

> > Scons-users mailing list

> > Scons-users at scons.org

> > http://four.pairlist.net/mailman/listinfo/scons-users

> > _______________________________________________

> > Scons-users mailing list

> > Scons-users at scons.org

> > http://four.pairlist.net/mailman/listinfo/scons-users

> >

> >

> > _______________________________________________

> > Scons-users mailing list

> > Scons-users at scons.org

> > http://four.pairlist.net/mailman/listinfo/scons-users

>

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://four.pairlist.net/pipermail/scons-users/attachments/20140322/45e100d0/attachment-0001.htm