[Scons-users] Memory problem with Integration tests

Thu Aug 14 19:33:21 EDT 2014

>
> I’m not sure how to confirm or deny the question of:
> “If not, can you confirm my assumption that the out of memory exception is
> not a stack overflow?”
> I have put the error and stack trace into the attachment, maybe it will
> tell you more than me.

No stack overflow based on the stack trace.  Thanks for the additional
information.  A memory profile for -j2 would still be helpful.

> With the ‘J1’ option, the memory was about 400-500MB.
> With the ‘J2’ option, I saw it grow up to 1200MB (than I needed to go) and
> after I was back it had crashed.

That exceeds my worst case projection. I'm puzzled at the growth rate.  One
of the senior devs might have more insight here.

When I would do a complete clean and give the command: ‘scons scripttests’.
> SCons will start building, compiling, linking, archiving and run the tests
> on a VMWare image.

Oh, so your build is happening on a VM?  How much RAM is allocated to your
VM and is that VM 32-bit?

> What I usually to is first a ‘scons testApp’ command. This will do the
> building, compiling and linking of the test app.
> After this, I usually do the ‘scons scripttests’ command, so scons skips
> the compile, link and archive steps and starts the tests directly.

If your earlier statement regarding 'scons scripttests' in true and you
have build dependencies, then 'scons scripttests' still does the overhead
of checking that those dependencies are up-to-date regardless, so scons
isn't really "skipping" the compile steps.

V/R,
William

On Thu, Aug 14, 2014 at 11:06 AM, Rob Deckers <Rob.Deckers at vanderlande.com>
wrote:

> Hi,
>
>
>
> There is no functional reason for using Python 32bits.
>
>
>
> The memory profile will have to wait until the weekend before I have time.
>
>
>
> I’m not sure how to confirm or deny the question of:
>
> “If not, can you confirm my assumption that the out of memory exception
> is not a stack overflow?”
>
> I have put the error and stack trace into the attachment, maybe it will
> tell you more than me.
>
>
>
> With the ‘J1’ option, the memory was about 400-500MB.
>
> With the ‘J2’ option, I saw it grow up to 1200MB (than I needed to go) and
> after I was back it had crashed.
>
> When I would do a complete clean and give the command: ‘scons scripttests’.
>
> SCons will start building, compiling, linking, archiving and run the tests
> on a VMWare image.
>
>
>
> What I usually to is first a ‘scons testApp’ command. This will do the
> building, compiling and linking of the test app.
>
> After this, I usually do the ‘scons scripttests’ command, so scons skips
> the compile, link and archive steps and starts the tests directly.
>
>
>
> I hope this gives more insight.
>
> Kind regards,
>
> Rob Deckers
>
>
>
> *From:* Scons-users [mailto:scons-users-bounces at scons.org] *On Behalf Of *William
> Blevins
> *Sent:* dinsdag 12 augustus 2014 18:28
> *To:* SCons users mailing list
>
> *Subject:* Re: [Scons-users] Memory problem with Integration tests
>
>
>
> Rob,
>
> My machine (Windows 7) has 14GB of RAM available.
>
> Python (being 32bits) seems to run out while coming close to the 2GB
> boundary.
>
>
>
> Is there a reason to use 32-bit python on a 64-bit operating system?
>
>
>
> (watched it grow and grow during the run).
> It especially seemed to grow between after finishing one of the
> integration-test-targets and before starting a second.
>
>
>
> It's strange that it would run out of memory around the 2GB (~1/2 the max
> for 32-bit) mark unless SCons was trying to allocate a very large
> contiguous chunk of memory.  Maybe an array or a string?  Generally,
> containers of that nature double in size when increasing capacity.  In most
> languages, the array size limit is well below the maximum.  In Java, this
> in 1GB for a 32-bit jvm.
>
>
>
> Would it be possible to get a memory usage profile for your setup?  Maybe
> via something like https://pypi.python.org/pypi/memory_profiler or your
> preferred tool?
>
>
>
> If not, can you confirm my assumption that the out of memory exception is
> not a stack overflow?
>
>
>
> What I had noticed already was that this in between time was VERY long
> (single to several minutes).
> Because the tests cannot be run in parallel, and I was trying to optimize,
> I tried:
> scons scripttests --max-drift=1 --implicit-cache -j 1
> and
> env.Decider('MD5-timestamp')
> The "-j 1" really made a HUGE improvement. With this option the memory is
> around 400-500MB (what you had guessed)
> Also the very long waits between the integration-test-targets are almost
> completely gone.
>
> It seems to work, but I don't know why and I would not like to be
> obligated to use the "-j 1"
>
>
>
> SCons Job pipeline handles -j1 and -jX for X>1 differently [if I'm not
> mistaken].
>
>
>
> I imagine each parallel task has a memory overhead.  I don't know if that
> overhead grows proportionally to the number of jobs.
>
>
>
> How many processors were you running with before?  What happens if you do
> -j2 instead of -jNUM_PROCS?
>
>
>
> Does the scons run in question build software or just run test scripts?
>  The configuration I use at work does both, and I haven't ran into this
> issue.
>
>
> -William
>
>
>
> On Tue, Aug 12, 2014 at 3:49 AM, Rob Deckers <Rob.Deckers at vanderlande.com>
> wrote:
>
> Hi,
>
> Thanks for the first comments.
>
> It's nice to hear that the overall size and the basic setup doesn't sound
> alarming to you.
>
> My machine (Windows 7) has 14GB of RAM available.
> Python (being 32bits) seems to run out while coming close to the 2GB
> boundary.
> (watched it grow and grow during the run).
> It especially seemed to grow between after finishing one of the
> integration-test-targets and before starting a second.
>
> What I had noticed already was that this in between time was VERY long
> (single to several minutes).
> Because the tests cannot be run in parallel, and I was trying to optimize,
> I tried:
>
> scons scripttests --max-drift=1 --implicit-cache -j 1
> and
> env.Decider('MD5-timestamp')
>
> The "-j 1" really made a HUGE improvement. With this option the memory is
> around 400-500MB (what you had guessed)
> Also the very long waits between the integration-test-targets are almost
> completely gone.
>
> It seems to work, but I don't know why and I would not like to be
> obligated to use the "-j 1"
>
> Kind regards,
> Rob Deckers
>
>
>
> -----Original Message-----
> From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of
> Dirk Bächle
> Sent: maandag 11 augustus 2014 17:35
> To: scons-users at scons.org
> Subject: Re: [Scons-users] Memory problem with Integration tests
>
> Hi Rob,
>
> it's difficult to guess what's wrong without seeing the
> SConstructs/SConscripts. Please find a few first comments below.
>
> On 11.08.2014 09:21, Rob Deckers wrote:
> > Hi All,
> >
> > I have SCons working including integration tests on one archive and now
> I'm integrating the integration tests on a second (slightly different)
> archive.
> >
> > The previous archive had about 800 scripttests and this one has 1200.
> > The 1200 tests are devided over about 100 directories and each directory
> is it's own target.
> >
> > Each scripttest uses the same single executable.
> > The executable is dependend on about 8.000 header and cpp files
> Given this number of input files, a memory consumption of 400-500MB (as
> your test results indicate), looks quite normal to me. This depends on the
> project's structure of course, but it's not an alarming value.
>
> > To make a all_scripttests target, each individual scripttest target is
> added like:
> > env.Alias(FscScriptTestTool['scripttest_alias'], target)
> >
> > When I first do a scons run to build the executable and then an
> all_scripttests run.
> > I see the memory growing with each scripttest (a rate of about 1MB every
> 4 seconds).
> >
> > Eventually, I get a memory error triggered from some copy command.
> > When I disabled that single line, I got the error on some other line.
> >
> > Is it possible my alias construction is to blame?
> I don't think so.
> > What does scons (beneath the surface) do with 100 targets all dependent
> on one executable with 8K files ?
> > Does it create 100 times the entire tree for the executable?
> No, it shouldn't do this. The dependencies are internally stored in a
> graph, which allows several targets to have the same source node(s) as
> input.
>
> > In the attachment is the output from an up-to-date run for the
> executable. I used:
> > scons executable --debug=memory --debug=count I have a hard time
> > really interpreting those results.
> The first thing that catches my eye is the high number of "Override"
> Environments. This means you're using things like
>
>    env.Program(target, source, SET_A_VARIABLE="something_special")
>
> a lot. Try to avoid that, and setup the Environments in your top-level
> SConstruct for the different cases you need. Then simply pass them around,
> but don't recreate them in all the places.
> If you think you *have* to do it like this, let's discuss the underlying
> problem and how to solve it. There should be a way around it...
>
> How much memory does your system have?
> Have you tried to watch memory consumption with a "top/vmstat" in parallel
> to running SCons? The "scons --debug=memory" isn't necessarily
> comprehensive.
> By how much does the overall memory consumption of the "scons" process
> grow when you run it with one test target, versus running with two targets?
>
> Best regards,
>
> Dirk
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> http://four.pairlist.net/mailman/listinfo/scons-users
>
>
> ------------------------------------------------------------------------------
>
> ** Disclaimer **
>
> This e-mail, including any attachments, may include proprietary and
> confidential information of Vanderlande and may only be read by the person
> or those persons to whom it is addressed.
> This document is forwarded to you in such a form (e-mail) that Vanderlande
> cannot guarantee the completeness and/or correctness of its contents and
> information.
> If you have received this e-mail message in error, please notify us
> immediately. Please also delete this document from your computer.
> This document may not be reproduced, copied, distributed, published,
> modified, or furnished to third parties, without the prior written consent
> of Vanderlande.
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> http://four.pairlist.net/mailman/listinfo/scons-users
>
>
> ------------------------------
>
> ** Disclaimer **
>
> This e-mail, including any attachments, may include proprietary and
> confidential information of Vanderlande and may only be read by the person
> or those persons to whom it is addressed.
> This document is forwarded to you in such a form (e-mail) that Vanderlande
> cannot guarantee the completeness and/or correctness of its contents and
> information.
> If you have received this e-mail message in error, please notify us
> immediately. Please also delete this document from your computer.
> This document may not be reproduced, copied, distributed, published,
> modified, or furnished to third parties, without the prior written consent
> of Vanderlande.
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> http://four.pairlist.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20140814/94f0ab0e/attachment-0001.html>