[Scons-users] SCONS -j12, console output is disordered

Mon Dec 25 02:51:30 EST 2023

Hi Guys，

	Thanks Mats, Duane and Bill, Thanks for your replay.

	I find something new. My compiler is IAR, and default, IAR will output warning and error messages to stderr, normal message will be put into stdout.
	So, if there are warnings in my code, normal msg and warning will be intertwined in words.
	Luckily, Param --only_stdout can make IAR output stream to stdout.

	But I have a new problem. The warnings msg is output by multi lines. So they are intertwined in line. For example:
		file0 warnings line 0
		file0 warnings line 1
		file1 normal msg line 0
		file0 warnings line 2
		file0 warnings line 3
	Actually, I want the lines is:
		file0 warnings line 0
		file0 warnings line 1
		file0 warnings line 2
		file0 warnings line 3
		file1 normal msg line 0

	I try to add mutex for print(), but no effection.
	Maybe, I should set env['SPAWN'], but I don't know how to use it.

	I have two questions:
	Where can I find a sample about env['SPAWN']?
	How can I print msg when scons building complete?

Best Regards!

Software Engineer
------------------------------------
zhiwei_yu at yanfeng.com

-----邮件原件-----
发件人: Scons-users <scons-users-bounces at scons.org> 代表 scons-users-request at scons.org
发送时间: 2023年12月24日 3:29
收件人: scons-users at scons.org
主题: Scons-users Digest, Vol 147, Issue 2

CAUTION: This email originated from outside of the organization. DO NOT click links or open attachments unless you recognize the sender and know the content is safe.

Send Scons-users mailing list submissions to
        scons-users at scons.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://pairlist4.pair.net/mailman/listinfo/scons-users
or, via email, send a message with subject or body 'help' to
        scons-users-request at scons.org

You can reach the person managing the list at
        scons-users-owner at scons.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of Scons-users digest..."

Today's Topics:

   1. Re: SCONS -j12, console output is disordered (Duane Ellis)
   2. Re: SCONS -j12, console output is disordered (Mats Wichmann)

----------------------------------------------------------------------

Message: 1
Date: Sat, 23 Dec 2023 18:33:20 +0000
From: Duane Ellis <duane at duaneellis.com>
To: SCons users mailing list <scons-users at scons.org>
Subject: Re: [Scons-users] SCONS -j12, console output is disordered
Message-ID:
        <PH0PR05MB84789EC7A7CE775049F19AD3CA9BA at PH0PR05MB8478.namprd05.prod.outlook.com>

Content-Type: text/plain; charset="utf-8"

SCONS could ?
                Capture the output of each sub process in the standard way.
                Ie: use ?POPEN()? and COMMUNICATE with the sub process.
                   Thus the subprocess output goes to SCONS and SCONS collects it in a buffer.
                    You might want to put a timeout on all sub processes, ie: 5 minutes?
                    In case they output an error and want sometype of input from the human to continue

                Then once the subprocess has completed, (Exit success or Exit Error)
               The responsible runner thread would:
                      Lock the console (mutex) so other SCONS threads cannot use it.
                      output stdout and stderr of the sub process
                      Unlock the console so other SCONS threads can use it.

From: Scons-users <scons-users-bounces at scons.org> on behalf of Mats Wichmann <mats at wichmann.us>
Date: Saturday, December 23, 2023 at 8:51 AM
To: scons-users at scons.org <scons-users at scons.org>
Subject: Re: [Scons-users] SCONS -j12, console output is disordered On 12/23/23 08:39, Yu,Zhiwei (YFI,Shanghai,CN) via Scons-users wrote:
> Hi Guys?
>
> ??????I add -j12 param, then the console output is disordered.
> ??????I wonder that the reason is python's print function is not 
> thread-safe or atomic operation, right?
> ??????How could I fix the problem when I use SCONS.

There are two sources of output:

(1) scons-generated (by default, the command line, modifiable by setting the various COMSTR variables, or setting up a PRINT_CMD_LINE_FUNC).
This text should not have any particular interleave problems.
(2) command-issued output.  This you can't do much about, because SCons isn't really designed for that.  As jobs are classified as ready to run, they're put on the queue; meanwhile when a job runner slot opens up, it's assigned a job from the queue. Those are run as subprocesses, and run independently; SCons doesn't collect their output or any other information besides the exit status.  So you're just seeing the effect of having 12 job runner slots all firing off jobs as they become available - if those generate their own output that can well interleave.

To not have interleaved command output, you'd need to convince SCons to collect the output of the subprocesses rather than going to the default stdout/stderr.  That is possible, but beyond the scope of a first answer.

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20231223/4c7188b9/attachment-0001.htm>

------------------------------

Message: 2
Date: Sat, 23 Dec 2023 12:29:06 -0700
From: Mats Wichmann <mats at wichmann.us>
To: scons-users at scons.org
Subject: Re: [Scons-users] SCONS -j12, console output is disordered
Message-ID: <2ec58418-041d-43b1-9e0a-90a9506c174c at wichmann.us>
Content-Type: text/plain; charset=UTF-8; format=flowed

On 12/23/23 11:33, Duane Ellis wrote:
> SCONS could ?
>
>  ??????????????? Capture the output of each sub process in the standard way.
>
>  ??????????????? Ie: use ?POPEN()? and COMMUNICATE with the sub process.
>
>  ???????????????????Thus the subprocess output goes to SCONS and SCONS 
> collects it in a buffer.

Yes, it could.  It even does so now in one very specific situation - while evaluating a Configure context.  However, even playing the game with pushing in the value of the PSPAWN construction variable in place of the SPAWN variable which is what the configure code does is not a full solution, as scons would actually have to be prepared to gather and do something with the output, which it currently isn't.

>
>  ??????????????????? You might want to put a timeout on all sub 
> processes, ie: 5 minutes?
>
>  ??????????????????? In case they output an error and want sometype of 
> input from the human to continue
>
>  ??????????????? Then once the subprocess has completed, (Exit success 
> or Exit Error)
>
>  ?????????????? The responsible runner thread would:
>
>  ????????????????????? Lock the console (mutex) so other SCONS threads 
> cannot use it.
>
>  ????????????????????? output stdout and stderr of the sub process
>
>  ????????????????????? Unlock the console so other SCONS threads can use it.

This is the method the test runner uses - queues tests, has a pool of runners to run them, and when one finished it takes a lock before writing the collected output. But that's a different usecase... the test runner *knows* it needs to collect output, while SCons itself really isn't structured that way. Using logging could also do this, as it handles locking between threads internally (there's an experimental version of the test runner that does that).

It *can* be done, but would require some effort.

------------------------------

Subject: Digest Footer

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

------------------------------

End of Scons-users Digest, Vol 147, Issue 2
*******************************************