[Scons-users] SCONS -j12, console output is disordered
Bill Deegan
bill at baddogconsulting.com
Mon Dec 25 15:36:49 EST 2023
Please respond to the individual message, not the digest of messages.
You can find information on SPAWN here:
https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN
I'm not aware of any easy to find examples, but you should be able to
figure it out based on the manpage
<https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN> contents:
(copied here)
A command interpreter function that will be called to execute command line
strings. The function must accept five arguments:
def spawn(shell, escape, cmd, args, env):
shell is a string naming the shell program to use, escape is a function
that can be called to escape shell special characters in the command line,
cmd is the path to the command to be executed, args holds the arguments to
the command and env is a dictionary of environment variables defining the
execution environment in which the command should be executed.
Here's some related stack overflow questions:
https://stackoverflow.com/questions/11139519/scons-save-redirect-gcc-textual-output-warnings
Please let us know if you come up with a good solution.
-Bill
On Sun, Dec 24, 2023 at 11:51 PM Yu,Zhiwei (YFI,Shanghai,CN) via
Scons-users <scons-users at scons.org> wrote:
> Hi Guys，
>
> Thanks Mats, Duane and Bill, Thanks for your replay.
>
> I find something new. My compiler is IAR, and default, IAR will
> output warning and error messages to stderr, normal message will be put
> into stdout.
> So, if there are warnings in my code, normal msg and warning will
> be intertwined in words.
> Luckily, Param --only_stdout can make IAR output stream to stdout.
>
> But I have a new problem. The warnings msg is output by multi
> lines. So they are intertwined in line. For example:
> file0 warnings line 0
> file0 warnings line 1
> file1 normal msg line 0
> file0 warnings line 2
> file0 warnings line 3
> Actually, I want the lines is:
> file0 warnings line 0
> file0 warnings line 1
> file0 warnings line 2
> file0 warnings line 3
> file1 normal msg line 0
>
> I try to add mutex for print(), but no effection.
> Maybe, I should set env['SPAWN'], but I don't know how to use it.
>
> I have two questions:
> Where can I find a sample about env['SPAWN']?
> How can I print msg when scons building complete?
>
>
> Best Regards!
>
> Software Engineer
> ------------------------------------
> zhiwei_yu at yanfeng.com
>
>
>
> -----邮件原件-----
> 发件人: Scons-users <scons-users-bounces at scons.org> 代表
> scons-users-request at scons.org
> 发送时间: 2023年12月24日 3:29
> 收件人: scons-users at scons.org
> 主题: Scons-users Digest, Vol 147, Issue 2
>
> CAUTION: This email originated from outside of the organization. DO NOT
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> Send Scons-users mailing list submissions to
> scons-users at scons.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://pairlist4.pair.net/mailman/listinfo/scons-users
> or, via email, send a message with subject or body 'help' to
> scons-users-request at scons.org
>
> You can reach the person managing the list at
> scons-users-owner at scons.org
>
> When replying, please edit your Subject line so it is more specific than
> "Re: Contents of Scons-users digest..."
>
>
> Today's Topics:
>
> 1. Re: SCONS -j12, console output is disordered (Duane Ellis)
> 2. Re: SCONS -j12, console output is disordered (Mats Wichmann)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 23 Dec 2023 18:33:20 +0000
> From: Duane Ellis <duane at duaneellis.com>
> To: SCons users mailing list <scons-users at scons.org>
> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
> Message-ID:
> <
> PH0PR05MB84789EC7A7CE775049F19AD3CA9BA at PH0PR05MB8478.namprd05.prod.outlook.com
> >
>
> Content-Type: text/plain; charset="utf-8"
>
> SCONS could ?
> Capture the output of each sub process in the standard way.
> Ie: use ?POPEN()? and COMMUNICATE with the sub process.
> Thus the subprocess output goes to SCONS and SCONS
> collects it in a buffer.
> You might want to put a timeout on all sub processes,
> ie: 5 minutes?
> In case they output an error and want sometype of
> input from the human to continue
>
> Then once the subprocess has completed, (Exit success or
> Exit Error)
> The responsible runner thread would:
> Lock the console (mutex) so other SCONS threads
> cannot use it.
> output stdout and stderr of the sub process
> Unlock the console so other SCONS threads can use it.
>
> From: Scons-users <scons-users-bounces at scons.org> on behalf of Mats
> Wichmann <mats at wichmann.us>
> Date: Saturday, December 23, 2023 at 8:51 AM
> To: scons-users at scons.org <scons-users at scons.org>
> Subject: Re: [Scons-users] SCONS -j12, console output is disordered On
> 12/23/23 08:39, Yu,Zhiwei (YFI,Shanghai,CN) via Scons-users wrote:
> > Hi Guys?
> >
> > ??????I add -j12 param, then the console output is disordered.
> > ??????I wonder that the reason is python's print function is not
> > thread-safe or atomic operation, right?
> > ??????How could I fix the problem when I use SCONS.
>
> There are two sources of output:
>
> (1) scons-generated (by default, the command line, modifiable by setting
> the various COMSTR variables, or setting up a PRINT_CMD_LINE_FUNC).
> This text should not have any particular interleave problems.
> (2) command-issued output. This you can't do much about, because SCons
> isn't really designed for that. As jobs are classified as ready to run,
> they're put on the queue; meanwhile when a job runner slot opens up, it's
> assigned a job from the queue. Those are run as subprocesses, and run
> independently; SCons doesn't collect their output or any other information
> besides the exit status. So you're just seeing the effect of having 12 job
> runner slots all firing off jobs as they become available - if those
> generate their own output that can well interleave.
>
> To not have interleaved command output, you'd need to convince SCons to
> collect the output of the subprocesses rather than going to the default
> stdout/stderr. That is possible, but beyond the scope of a first answer.
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://pairlist4.pair.net/pipermail/scons-users/attachments/20231223/4c7188b9/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 2
> Date: Sat, 23 Dec 2023 12:29:06 -0700
> From: Mats Wichmann <mats at wichmann.us>
> To: scons-users at scons.org
> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
> Message-ID: <2ec58418-041d-43b1-9e0a-90a9506c174c at wichmann.us>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> On 12/23/23 11:33, Duane Ellis wrote:
> > SCONS could ?
> >
> > ??????????????? Capture the output of each sub process in the standard
> way.
> >
> > ??????????????? Ie: use ?POPEN()? and COMMUNICATE with the sub process.
> >
> > ???????????????????Thus the subprocess output goes to SCONS and SCONS
> > collects it in a buffer.
>
> Yes, it could. It even does so now in one very specific situation - while
> evaluating a Configure context. However, even playing the game with
> pushing in the value of the PSPAWN construction variable in place of the
> SPAWN variable which is what the configure code does is not a full
> solution, as scons would actually have to be prepared to gather and do
> something with the output, which it currently isn't.
>
> >
> > ??????????????????? You might want to put a timeout on all sub
> > processes, ie: 5 minutes?
> >
> > ??????????????????? In case they output an error and want sometype of
> > input from the human to continue
> >
> > ??????????????? Then once the subprocess has completed, (Exit success
> > or Exit Error)
> >
> > ?????????????? The responsible runner thread would:
> >
> > ????????????????????? Lock the console (mutex) so other SCONS threads
> > cannot use it.
> >
> > ????????????????????? output stdout and stderr of the sub process
> >
> > ????????????????????? Unlock the console so other SCONS threads can use
> it.
>
> This is the method the test runner uses - queues tests, has a pool of
> runners to run them, and when one finished it takes a lock before writing
> the collected output. But that's a different usecase... the test runner
> *knows* it needs to collect output, while SCons itself really isn't
> structured that way. Using logging could also do this, as it handles
> locking between threads internally (there's an experimental version of the
> test runner that does that).
>
> It *can* be done, but would require some effort.
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> ------------------------------
>
> End of Scons-users Digest, Vol 147, Issue 2
> *******************************************
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20231225/c23dd73b/attachment-0001.htm>
More information about the Scons-users
mailing list