[Scons-users] SCONS -j12, console output is disordered
Gary Granger
granger at ucar.edu
Thu Jan 4 12:32:54 EST 2024
For whatever it's worth, I had an example to provide, but then after
looking at it I decided it needed some work. So now it's a little more
self-contained, and I think it's a reasonable example of a SPAWN
override. We use it to save full test output into log files while
filtering the console output to the more meaningful lines. I expect it
could be modified easily to cache the output from a single command and
write the output to the console all at once. Perhaps that would avoid
your interleaved output problem. Our implementation can be seen here:
https://github.com/NCAR/eol_scons/blob/master/eol_scons/spawner.py
Good luck,
Gary
On 12/25/23 13:36, Bill Deegan wrote:
> Please respond to the individual message, not the digest of messages.
>
> You can find information on SPAWN here:
> https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN
>
> I'm not aware of any easy to find examples, but you should be able to
> figure it out based on the manpage
> <https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN>
> contents: (copied here)
>
> A command interpreter function that will be called to execute
> command line strings. The function must accept five arguments:
>
> def spawn(shell, escape, cmd, args, env):
>
> |shell|is a string naming the shell program to use,|escape|is a
> function that can be called to escape shell special characters in
> the command line,|cmd|is the path to the command to be
> executed,|args|holds the arguments to the command and|env|is a
> dictionary of environment variables defining the execution
> environment in which the command should be executed.
>
> Here's some related stack overflow questions:
> https://stackoverflow.com/questions/11139519/scons-save-redirect-gcc-textual-output-warnings
>
> Please let us know if you come up with a good solution.
>
> -Bill
>
> On Sun, Dec 24, 2023 at 11:51 PM Yu,Zhiwei (YFI,Shanghai,CN) via
> Scons-users <scons-users at scons.org> wrote:
>
> Hi Guys，
>
> Thanks Mats, Duane and Bill, Thanks for your replay.
>
> I find something new. My compiler is IAR, and default, IAR
> will output warning and error messages to stderr, normal message
> will be put into stdout.
> So, if there are warnings in my code, normal msg and
> warning will be intertwined in words.
> Luckily, Param --only_stdout can make IAR output stream to
> stdout.
>
> But I have a new problem. The warnings msg is output by
> multi lines. So they are intertwined in line. For example:
> file0 warnings line 0
> file0 warnings line 1
> file1 normal msg line 0
> file0 warnings line 2
> file0 warnings line 3
> Actually, I want the lines is:
> file0 warnings line 0
> file0 warnings line 1
> file0 warnings line 2
> file0 warnings line 3
> file1 normal msg line 0
>
> I try to add mutex for print(), but no effection.
> Maybe, I should set env['SPAWN'], but I don't know how to
> use it.
>
> I have two questions:
> Where can I find a sample about env['SPAWN']?
> How can I print msg when scons building complete?
>
>
> Best Regards!
>
> Software Engineer
> ------------------------------------
> zhiwei_yu at yanfeng.com
>
>
>
> -----邮件原件-----
> 发件人: Scons-users <scons-users-bounces at scons.org> 代表
> scons-users-request at scons.org
> 发送时间: 2023年12月24日 3:29
> 收件人: scons-users at scons.org
> 主题: Scons-users Digest, Vol 147, Issue 2
>
> CAUTION: This email originated from outside of the organization.
> DO NOT click links or open attachments unless you recognize the
> sender and know the content is safe.
>
>
> Send Scons-users mailing list submissions to
> scons-users at scons.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://pairlist4.pair.net/mailman/listinfo/scons-users
> or, via email, send a message with subject or body 'help' to
> scons-users-request at scons.org
>
> You can reach the person managing the list at
> scons-users-owner at scons.org
>
> When replying, please edit your Subject line so it is more
> specific than "Re: Contents of Scons-users digest..."
>
>
> Today's Topics:
>
> 1. Re: SCONS -j12, console output is disordered (Duane Ellis)
> 2. Re: SCONS -j12, console output is disordered (Mats Wichmann)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 23 Dec 2023 18:33:20 +0000
> From: Duane Ellis <duane at duaneellis.com>
> To: SCons users mailing list <scons-users at scons.org>
> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
> Message-ID:
>
> <PH0PR05MB84789EC7A7CE775049F19AD3CA9BA at PH0PR05MB8478.namprd05.prod.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> SCONS could ?
> Capture the output of each sub process in the
> standard way.
> Ie: use ?POPEN()? and COMMUNICATE with the sub
> process.
> Thus the subprocess output goes to SCONS and
> SCONS collects it in a buffer.
> You might want to put a timeout on all sub
> processes, ie: 5 minutes?
> In case they output an error and want sometype
> of input from the human to continue
>
> Then once the subprocess has completed, (Exit
> success or Exit Error)
> The responsible runner thread would:
> Lock the console (mutex) so other SCONS
> threads cannot use it.
> output stdout and stderr of the sub process
> Unlock the console so other SCONS threads
> can use it.
>
> From: Scons-users <scons-users-bounces at scons.org> on behalf of
> Mats Wichmann <mats at wichmann.us>
> Date: Saturday, December 23, 2023 at 8:51 AM
> To: scons-users at scons.org <scons-users at scons.org>
> Subject: Re: [Scons-users] SCONS -j12, console output is
> disordered On 12/23/23 08:39, Yu,Zhiwei (YFI,Shanghai,CN) via
> Scons-users wrote:
> > Hi Guys?
> >
> > ??????I add -j12 param, then the console output is disordered.
> > ??????I wonder that the reason is python's print function is not
> > thread-safe or atomic operation, right?
> > ??????How could I fix the problem when I use SCONS.
>
> There are two sources of output:
>
> (1) scons-generated (by default, the command line, modifiable by
> setting the various COMSTR variables, or setting up a
> PRINT_CMD_LINE_FUNC).
> This text should not have any particular interleave problems.
> (2) command-issued output. This you can't do much about, because
> SCons isn't really designed for that. As jobs are classified as
> ready to run, they're put on the queue; meanwhile when a job
> runner slot opens up, it's assigned a job from the queue. Those
> are run as subprocesses, and run independently; SCons doesn't
> collect their output or any other information besides the exit
> status. So you're just seeing the effect of having 12 job runner
> slots all firing off jobs as they become available - if those
> generate their own output that can well interleave.
>
> To not have interleaved command output, you'd need to convince
> SCons to collect the output of the subprocesses rather than going
> to the default stdout/stderr. That is possible, but beyond the
> scope of a first answer.
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://pairlist4.pair.net/pipermail/scons-users/attachments/20231223/4c7188b9/attachment-0001.htm>
>
> ------------------------------
>
> Message: 2
> Date: Sat, 23 Dec 2023 12:29:06 -0700
> From: Mats Wichmann <mats at wichmann.us>
> To: scons-users at scons.org
> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
> Message-ID: <2ec58418-041d-43b1-9e0a-90a9506c174c at wichmann.us>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> On 12/23/23 11:33, Duane Ellis wrote:
> > SCONS could ?
> >
> > ??????????????? Capture the output of each sub process in the
> standard way.
> >
> > ??????????????? Ie: use ?POPEN()? and COMMUNICATE with the sub
> process.
> >
> > ???????????????????Thus the subprocess output goes to SCONS and
> SCONS
> > collects it in a buffer.
>
> Yes, it could. It even does so now in one very specific situation
> - while evaluating a Configure context. However, even playing the
> game with pushing in the value of the PSPAWN construction variable
> in place of the SPAWN variable which is what the configure code
> does is not a full solution, as scons would actually have to be
> prepared to gather and do something with the output, which it
> currently isn't.
>
> >
> > ??????????????????? You might want to put a timeout on all sub
> > processes, ie: 5 minutes?
> >
> > ??????????????????? In case they output an error and want
> sometype of
> > input from the human to continue
> >
> > ??????????????? Then once the subprocess has completed, (Exit
> success
> > or Exit Error)
> >
> > ?????????????? The responsible runner thread would:
> >
> > ????????????????????? Lock the console (mutex) so other SCONS
> threads
> > cannot use it.
> >
> > ????????????????????? output stdout and stderr of the sub process
> >
> > ????????????????????? Unlock the console so other SCONS threads
> can use it.
>
> This is the method the test runner uses - queues tests, has a pool
> of runners to run them, and when one finished it takes a lock
> before writing the collected output. But that's a different
> usecase... the test runner *knows* it needs to collect output,
> while SCons itself really isn't structured that way. Using logging
> could also do this, as it handles locking between threads
> internally (there's an experimental version of the test runner
> that does that).
>
> It *can* be done, but would require some effort.
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> ------------------------------
>
> End of Scons-users Digest, Vol 147, Issue 2
> *******************************************
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20240104/3c58e881/attachment-0001.htm>
More information about the Scons-users
mailing list