[Scons-users] SCONS -j12, console output is disordered
Bill Deegan
bill at baddogconsulting.com
Thu Jan 4 12:39:21 EST 2024
Gary,
Any chance that repo could be relicensed, or dual licensed MIT?
That would allow us to possibly pull in some of your logic to core scons..
-Bill
On Thu, Jan 4, 2024 at 9:33 AM Gary Granger <granger at ucar.edu> wrote:
> For whatever it's worth, I had an example to provide, but then after
> looking at it I decided it needed some work. So now it's a little more
> self-contained, and I think it's a reasonable example of a SPAWN override.
> We use it to save full test output into log files while filtering the
> console output to the more meaningful lines. I expect it could be modified
> easily to cache the output from a single command and write the output to
> the console all at once. Perhaps that would avoid your interleaved output
> problem. Our implementation can be seen here:
>
> https://github.com/NCAR/eol_scons/blob/master/eol_scons/spawner.py
>
> Good luck,
> Gary
>
> On 12/25/23 13:36, Bill Deegan wrote:
>
> Please respond to the individual message, not the digest of messages.
>
> You can find information on SPAWN here:
> https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN
>
> I'm not aware of any easy to find examples, but you should be able to
> figure it out based on the manpage
> <https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN> contents:
> (copied here)
>
> A command interpreter function that will be called to execute command line
> strings. The function must accept five arguments:
>
> def spawn(shell, escape, cmd, args, env):
>
> shell is a string naming the shell program to use, escape is a function
> that can be called to escape shell special characters in the command line,
> cmd is the path to the command to be executed, args holds the arguments
> to the command and env is a dictionary of environment variables defining
> the execution environment in which the command should be executed.
> Here's some related stack overflow questions:
> https://stackoverflow.com/questions/11139519/scons-save-redirect-gcc-textual-output-warnings
> Please let us know if you come up with a good solution.
>
> -Bill
>
> On Sun, Dec 24, 2023 at 11:51 PM Yu,Zhiwei (YFI,Shanghai,CN) via
> Scons-users <scons-users at scons.org> wrote:
>
>> Hi Guys，
>>
>> Thanks Mats, Duane and Bill, Thanks for your replay.
>>
>> I find something new. My compiler is IAR, and default, IAR will
>> output warning and error messages to stderr, normal message will be put
>> into stdout.
>> So, if there are warnings in my code, normal msg and warning will
>> be intertwined in words.
>> Luckily, Param --only_stdout can make IAR output stream to stdout.
>>
>> But I have a new problem. The warnings msg is output by multi
>> lines. So they are intertwined in line. For example:
>> file0 warnings line 0
>> file0 warnings line 1
>> file1 normal msg line 0
>> file0 warnings line 2
>> file0 warnings line 3
>> Actually, I want the lines is:
>> file0 warnings line 0
>> file0 warnings line 1
>> file0 warnings line 2
>> file0 warnings line 3
>> file1 normal msg line 0
>>
>> I try to add mutex for print(), but no effection.
>> Maybe, I should set env['SPAWN'], but I don't know how to use it.
>>
>> I have two questions:
>> Where can I find a sample about env['SPAWN']?
>> How can I print msg when scons building complete?
>>
>>
>> Best Regards!
>>
>> Software Engineer
>> ------------------------------------
>> zhiwei_yu at yanfeng.com
>>
>>
>>
>> -----邮件原件-----
>> 发件人: Scons-users <scons-users-bounces at scons.org> 代表
>> scons-users-request at scons.org
>> 发送时间: 2023年12月24日 3:29
>> 收件人: scons-users at scons.org
>> 主题: Scons-users Digest, Vol 147, Issue 2
>>
>> CAUTION: This email originated from outside of the organization. DO NOT
>> click links or open attachments unless you recognize the sender and know
>> the content is safe.
>>
>>
>> Send Scons-users mailing list submissions to
>> scons-users at scons.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> or, via email, send a message with subject or body 'help' to
>> scons-users-request at scons.org
>>
>> You can reach the person managing the list at
>> scons-users-owner at scons.org
>>
>> When replying, please edit your Subject line so it is more specific than
>> "Re: Contents of Scons-users digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: SCONS -j12, console output is disordered (Duane Ellis)
>> 2. Re: SCONS -j12, console output is disordered (Mats Wichmann)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sat, 23 Dec 2023 18:33:20 +0000
>> From: Duane Ellis <duane at duaneellis.com>
>> To: SCons users mailing list <scons-users at scons.org>
>> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
>> Message-ID:
>> <
>> PH0PR05MB84789EC7A7CE775049F19AD3CA9BA at PH0PR05MB8478.namprd05.prod.outlook.com
>> >
>>
>> Content-Type: text/plain; charset="utf-8"
>>
>> SCONS could ?
>> Capture the output of each sub process in the standard
>> way.
>> Ie: use ?POPEN()? and COMMUNICATE with the sub process.
>> Thus the subprocess output goes to SCONS and SCONS
>> collects it in a buffer.
>> You might want to put a timeout on all sub processes,
>> ie: 5 minutes?
>> In case they output an error and want sometype of
>> input from the human to continue
>>
>> Then once the subprocess has completed, (Exit success or
>> Exit Error)
>> The responsible runner thread would:
>> Lock the console (mutex) so other SCONS threads
>> cannot use it.
>> output stdout and stderr of the sub process
>> Unlock the console so other SCONS threads can use
>> it.
>>
>> From: Scons-users <scons-users-bounces at scons.org> on behalf of Mats
>> Wichmann <mats at wichmann.us>
>> Date: Saturday, December 23, 2023 at 8:51 AM
>> To: scons-users at scons.org <scons-users at scons.org>
>> Subject: Re: [Scons-users] SCONS -j12, console output is disordered On
>> 12/23/23 08:39, Yu,Zhiwei (YFI,Shanghai,CN) via Scons-users wrote:
>> > Hi Guys?
>> >
>> > ??????I add -j12 param, then the console output is disordered.
>> > ??????I wonder that the reason is python's print function is not
>> > thread-safe or atomic operation, right?
>> > ??????How could I fix the problem when I use SCONS.
>>
>> There are two sources of output:
>>
>> (1) scons-generated (by default, the command line, modifiable by setting
>> the various COMSTR variables, or setting up a PRINT_CMD_LINE_FUNC).
>> This text should not have any particular interleave problems.
>> (2) command-issued output. This you can't do much about, because SCons
>> isn't really designed for that. As jobs are classified as ready to run,
>> they're put on the queue; meanwhile when a job runner slot opens up, it's
>> assigned a job from the queue. Those are run as subprocesses, and run
>> independently; SCons doesn't collect their output or any other information
>> besides the exit status. So you're just seeing the effect of having 12 job
>> runner slots all firing off jobs as they become available - if those
>> generate their own output that can well interleave.
>>
>> To not have interleaved command output, you'd need to convince SCons to
>> collect the output of the subprocesses rather than going to the default
>> stdout/stderr. That is possible, but beyond the scope of a first answer.
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://pairlist4.pair.net/pipermail/scons-users/attachments/20231223/4c7188b9/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sat, 23 Dec 2023 12:29:06 -0700
>> From: Mats Wichmann <mats at wichmann.us>
>> To: scons-users at scons.org
>> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
>> Message-ID: <2ec58418-041d-43b1-9e0a-90a9506c174c at wichmann.us>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>> On 12/23/23 11:33, Duane Ellis wrote:
>> > SCONS could ?
>> >
>> > ??????????????? Capture the output of each sub process in the standard
>> way.
>> >
>> > ??????????????? Ie: use ?POPEN()? and COMMUNICATE with the sub process.
>> >
>> > ???????????????????Thus the subprocess output goes to SCONS and SCONS
>> > collects it in a buffer.
>>
>> Yes, it could. It even does so now in one very specific situation -
>> while evaluating a Configure context. However, even playing the game with
>> pushing in the value of the PSPAWN construction variable in place of the
>> SPAWN variable which is what the configure code does is not a full
>> solution, as scons would actually have to be prepared to gather and do
>> something with the output, which it currently isn't.
>>
>> >
>> > ??????????????????? You might want to put a timeout on all sub
>> > processes, ie: 5 minutes?
>> >
>> > ??????????????????? In case they output an error and want sometype of
>> > input from the human to continue
>> >
>> > ??????????????? Then once the subprocess has completed, (Exit success
>> > or Exit Error)
>> >
>> > ?????????????? The responsible runner thread would:
>> >
>> > ????????????????????? Lock the console (mutex) so other SCONS threads
>> > cannot use it.
>> >
>> > ????????????????????? output stdout and stderr of the sub process
>> >
>> > ????????????????????? Unlock the console so other SCONS threads can
>> use it.
>>
>> This is the method the test runner uses - queues tests, has a pool of
>> runners to run them, and when one finished it takes a lock before writing
>> the collected output. But that's a different usecase... the test runner
>> *knows* it needs to collect output, while SCons itself really isn't
>> structured that way. Using logging could also do this, as it handles
>> locking between threads internally (there's an experimental version of the
>> test runner that does that).
>>
>> It *can* be done, but would require some effort.
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>> ------------------------------
>>
>> End of Scons-users Digest, Vol 147, Issue 2
>> *******************************************
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
> _______________________________________________
> Scons-users mailing listScons-users at scons.orghttps://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20240104/dba14c5a/attachment-0001.htm>
More information about the Scons-users
mailing list