[Scons-users] Memory problem with Integration tests

Sun Aug 17 03:33:36 EDT 2014

For the memory profile, I need to decorate functions. Are there any special function that I should decorate to get insights?

I have tried all the functions from our scripttest tool.
But I got: RuntimeError: maximum recursion depth exceeded in cmp:
Before the first test really started.
See attachment 1, maybe this provides a clue

When I minimalised the decorated functions to just 1 the scripttests started, ran and gave a memory error.
This was our function: _runscripttest_action_generator ##  Action generator for running a script test on TESTTARGET
The profile result is in attachtment 2.

The attachments 1 & 2 are made with J<plenty>.

I also tried to profiele _test_dependency_action_generator.
This action generator is used to copy some shared libraries to the VMWare.
This ended in the same RuntimeError: maximum recursion depth exceeded in cmp:

Kind regards,
Rob Deckers

From: Scons-users [mailto:scons-users-bounces at scons.org] On Behalf Of William Blevins
Sent: vrijdag 15 augustus 2014 1:33
To: SCons users mailing list
Subject: Re: [Scons-users] Memory problem with Integration tests

I’m not sure how to confirm or deny the question of:
“If not, can you confirm my assumption that the out of memory exception is not a stack overflow?”
I have put the error and stack trace into the attachment, maybe it will tell you more than me.

No stack overflow based on the stack trace.  Thanks for the additional information.  A memory profile for -j2 would still be helpful.

With the ‘J1’ option, the memory was about 400-500MB.
With the ‘J2’ option, I saw it grow up to 1200MB (than I needed to go) and after I was back it had crashed.

That exceeds my worst case projection. I'm puzzled at the growth rate.  One of the senior devs might have more insight here.

When I would do a complete clean and give the command: ‘scons scripttests’.
SCons will start building, compiling, linking, archiving and run the tests on a VMWare image.

Oh, so your build is happening on a VM?  How much RAM is allocated to your VM and is that VM 32-bit?

What I usually to is first a ‘scons testApp’ command. This will do the building, compiling and linking of the test app.
After this, I usually do the ‘scons scripttests’ command, so scons skips the compile, link and archive steps and starts the tests directly.

If your earlier statement regarding 'scons scripttests' in true and you have build dependencies, then 'scons scripttests' still does the overhead of checking that those dependencies are up-to-date regardless, so scons isn't really "skipping" the compile steps.

V/R,
William

On Thu, Aug 14, 2014 at 11:06 AM, Rob Deckers <Rob.Deckers at vanderlande.com<mailto:Rob.Deckers at vanderlande.com>> wrote:
Hi,

There is no functional reason for using Python 32bits.

The memory profile will have to wait until the weekend before I have time.

I’m not sure how to confirm or deny the question of:
“If not, can you confirm my assumption that the out of memory exception is not a stack overflow?”
I have put the error and stack trace into the attachment, maybe it will tell you more than me.

With the ‘J1’ option, the memory was about 400-500MB.
With the ‘J2’ option, I saw it grow up to 1200MB (than I needed to go) and after I was back it had crashed.
When I would do a complete clean and give the command: ‘scons scripttests’.
SCons will start building, compiling, linking, archiving and run the tests on a VMWare image.

What I usually to is first a ‘scons testApp’ command. This will do the building, compiling and linking of the test app.
After this, I usually do the ‘scons scripttests’ command, so scons skips the compile, link and archive steps and starts the tests directly.

I hope this gives more insight.
Kind regards,
Rob Deckers

From: Scons-users [mailto:scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>] On Behalf Of William Blevins
Sent: dinsdag 12 augustus 2014 18:28
To: SCons users mailing list

Subject: Re: [Scons-users] Memory problem with Integration tests

Rob,
My machine (Windows 7) has 14GB of RAM available.
Python (being 32bits) seems to run out while coming close to the 2GB boundary.

Is there a reason to use 32-bit python on a 64-bit operating system?

(watched it grow and grow during the run).
It especially seemed to grow between after finishing one of the integration-test-targets and before starting a second.

It's strange that it would run out of memory around the 2GB (~1/2 the max for 32-bit) mark unless SCons was trying to allocate a very large contiguous chunk of memory.  Maybe an array or a string?  Generally, containers of that nature double in size when increasing capacity.  In most languages, the array size limit is well below the maximum.  In Java, this in 1GB for a 32-bit jvm.

Would it be possible to get a memory usage profile for your setup?  Maybe via something like https://pypi.python.org/pypi/memory_profiler or your preferred tool?

If not, can you confirm my assumption that the out of memory exception is not a stack overflow?

What I had noticed already was that this in between time was VERY long (single to several minutes).
Because the tests cannot be run in parallel, and I was trying to optimize, I tried:
scons scripttests --max-drift=1 --implicit-cache -j 1
and
env.Decider('MD5-timestamp')
The "-j 1" really made a HUGE improvement. With this option the memory is around 400-500MB (what you had guessed)
Also the very long waits between the integration-test-targets are almost completely gone.
It seems to work, but I don't know why and I would not like to be obligated to use the "-j 1"

SCons Job pipeline handles -j1 and -jX for X>1 differently [if I'm not mistaken].

I imagine each parallel task has a memory overhead.  I don't know if that overhead grows proportionally to the number of jobs.

How many processors were you running with before?  What happens if you do -j2 instead of -jNUM_PROCS?

Does the scons run in question build software or just run test scripts?  The configuration I use at work does both, and I haven't ran into this issue.

-William

On Tue, Aug 12, 2014 at 3:49 AM, Rob Deckers <Rob.Deckers at vanderlande.com<mailto:Rob.Deckers at vanderlande.com>> wrote:
Hi,

Thanks for the first comments.

It's nice to hear that the overall size and the basic setup doesn't sound alarming to you.

My machine (Windows 7) has 14GB of RAM available.
Python (being 32bits) seems to run out while coming close to the 2GB boundary.
(watched it grow and grow during the run).
It especially seemed to grow between after finishing one of the integration-test-targets and before starting a second.

What I had noticed already was that this in between time was VERY long (single to several minutes).
Because the tests cannot be run in parallel, and I was trying to optimize, I tried:

scons scripttests --max-drift=1 --implicit-cache -j 1
and
env.Decider('MD5-timestamp')

The "-j 1" really made a HUGE improvement. With this option the memory is around 400-500MB (what you had guessed)
Also the very long waits between the integration-test-targets are almost completely gone.

It seems to work, but I don't know why and I would not like to be obligated to use the "-j 1"

Kind regards,
Rob Deckers

-----Original Message-----
From: Scons-users [mailto:scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>] On Behalf Of Dirk Bächle
Sent: maandag 11 augustus 2014 17:35
To: scons-users at scons.org<mailto:scons-users at scons.org>
Subject: Re: [Scons-users] Memory problem with Integration tests

Hi Rob,

it's difficult to guess what's wrong without seeing the SConstructs/SConscripts. Please find a few first comments below.

On 11.08.2014 09:21, Rob Deckers wrote:
> Hi All,
>
> I have SCons working including integration tests on one archive and now I'm integrating the integration tests on a second (slightly different) archive.
>
> The previous archive had about 800 scripttests and this one has 1200.
> The 1200 tests are devided over about 100 directories and each directory is it's own target.
>
> Each scripttest uses the same single executable.
> The executable is dependend on about 8.000 header and cpp files
Given this number of input files, a memory consumption of 400-500MB (as your test results indicate), looks quite normal to me. This depends on the project's structure of course, but it's not an alarming value.

> To make a all_scripttests target, each individual scripttest target is added like:
> env.Alias(FscScriptTestTool['scripttest_alias'], target)
>
> When I first do a scons run to build the executable and then an all_scripttests run.
> I see the memory growing with each scripttest (a rate of about 1MB every 4 seconds).
>
> Eventually, I get a memory error triggered from some copy command.
> When I disabled that single line, I got the error on some other line.
>
> Is it possible my alias construction is to blame?
I don't think so.
> What does scons (beneath the surface) do with 100 targets all dependent on one executable with 8K files ?
> Does it create 100 times the entire tree for the executable?
No, it shouldn't do this. The dependencies are internally stored in a graph, which allows several targets to have the same source node(s) as input.

> In the attachment is the output from an up-to-date run for the executable. I used:
> scons executable --debug=memory --debug=count I have a hard time
> really interpreting those results.
The first thing that catches my eye is the high number of "Override"
Environments. This means you're using things like

   env.Program(target, source, SET_A_VARIABLE="something_special")

a lot. Try to avoid that, and setup the Environments in your top-level SConstruct for the different cases you need. Then simply pass them around, but don't recreate them in all the places.
If you think you *have* to do it like this, let's discuss the underlying problem and how to solve it. There should be a way around it...

How much memory does your system have?
Have you tried to watch memory consumption with a "top/vmstat" in parallel to running SCons? The "scons --debug=memory" isn't necessarily comprehensive.
By how much does the overall memory consumption of the "scons" process grow when you run it with one test target, versus running with two targets?

Best regards,

Dirk

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
http://four.pairlist.net/mailman/listinfo/scons-users
------------------------------------------------------------------------------

** Disclaimer **

This e-mail, including any attachments, may include proprietary and confidential information of Vanderlande and may only be read by the person or those persons to whom it is addressed.
This document is forwarded to you in such a form (e-mail) that Vanderlande cannot guarantee the completeness and/or correctness of its contents and information.
If you have received this e-mail message in error, please notify us immediately. Please also delete this document from your computer.
This document may not be reproduced, copied, distributed, published, modified, or furnished to third parties, without the prior written consent of Vanderlande.
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
http://four.pairlist.net/mailman/listinfo/scons-users

________________________________

** Disclaimer **

This e-mail, including any attachments, may include proprietary and confidential information of Vanderlande and may only be read by the person or those persons to whom it is addressed.
This document is forwarded to you in such a form (e-mail) that Vanderlande cannot guarantee the completeness and/or correctness of its contents and information.
If you have received this e-mail message in error, please notify us immediately. Please also delete this document from your computer.
This document may not be reproduced, copied, distributed, published, modified, or furnished to third parties, without the prior written consent of Vanderlande.

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
http://four.pairlist.net/mailman/listinfo/scons-users

------------------------------------------------------------------------------

** Disclaimer **

This e-mail, including any attachments, may include proprietary and confidential information of Vanderlande and may only be read by the person or those persons to whom it is addressed.
This document is forwarded to you in such a form (e-mail) that Vanderlande cannot guarantee the completeness and/or correctness of its contents and information.
If you have received this e-mail message in error, please notify us immediately. Please also delete this document from your computer.
This document may not be reproduced, copied, distributed, published, modified, or furnished to third parties, without the prior written consent of Vanderlande.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20140817/3e216ea8/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: attachment2.txt
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20140817/3e216ea8/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: attachment1.txt
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20140817/3e216ea8/attachment-0003.txt>