[Scons-users] Surprising behavior with pipes

[Gary Granger]

On 10/01/2017 09:46 AM, Andrew C. Morrow wrote:
>
> That does assume that SHELL is set to bash though, right? I'd be OK
> limiting it to POSIX, but going so far as to mandate one particular
> shell implementation feels pretty fragile.
I suppose so, but in this case I knew that I only needed it to work on bash.

> I guess the right solution is to not use pipes, but instead redirect
> the output to an intermediate target and then consume that, making a
> two step process.
>
> Somewhat unfortunate in my case as I really don't need the
> intermediate state and would prefer to avoid the IO cost. Perhaps the
> intermediate could be a tempfile of some sort. Basically, re-implement
> the pipe by hand.

I have used an approach which is close to implementing the pipe by hand,
so maybe it's relevant to you.  For running test programs, I wanted to
be able to filter the stdout and also save all the output to a log file,
while not having to resort to a complicated pipe with tee and grep.  So
I wrote a special Action subclass which would override the SCons SPAWN
variable with a custom spawn implementation.  That way I could execute
the action command with my own subprocess and get access to the output
pipe.  Maybe you could write a spawner which did the same thing, except
connected the pipes between subprocesses?  Or if you know the
intermediate output is small enough, you could read it all into memory
and then feed that as the input to the next process in the pipe, without
using disk space, and still being able to check the exit status of each
process in the chain.  If I ever need to run a pipe on one of our
Windows builds, without a bash shell, I guess I'd have to try an
approach like that.

In case it's useful, the code I'm talking about is the LogAction and
_SpawnerLogger classes in this tool:

   
https://github.com/ncareol/eol_scons/blob/master/eol_scons/tools/testing.py

The doc string may not be up to date, so user beware.

Gary