[Scons-users] Surprising behavior with pipes

[Andrew C. Morrow]

Thanks all for your suggestions.

Tom your points about the fragility of a pipeline and IO redirections are
well taken. I think for now I will probably just take the IO hit and write
the abidw output to disk via its --outfile argument (seems to be about
250MB on disk for my build), and then investigate Gary's suggestion as a
later optimization.

Once I have it in a state I consider good I hope to offer it to the SCons
contrib repo. Short-circuiting dynamic library relinks when the ABI hasn't
changed can be a huge win for developer builds.

On Wed, Oct 4, 2017 at 8:04 PM, Gary Granger <granger at ucar.edu> wrote:

> On 10/01/2017 09:46 AM, Andrew C. Morrow wrote:
> >
> > That does assume that SHELL is set to bash though, right? I'd be OK
> > limiting it to POSIX, but going so far as to mandate one particular
> > shell implementation feels pretty fragile.
> I suppose so, but in this case I knew that I only needed it to work on
> bash.
>
> > I guess the right solution is to not use pipes, but instead redirect
> > the output to an intermediate target and then consume that, making a
> > two step process.
> >
> > Somewhat unfortunate in my case as I really don't need the
> > intermediate state and would prefer to avoid the IO cost. Perhaps the
> > intermediate could be a tempfile of some sort. Basically, re-implement
> > the pipe by hand.
>
> I have used an approach which is close to implementing the pipe by hand,
> so maybe it's relevant to you.  For running test programs, I wanted to
> be able to filter the stdout and also save all the output to a log file,
> while not having to resort to a complicated pipe with tee and grep.  So
> I wrote a special Action subclass which would override the SCons SPAWN
> variable with a custom spawn implementation.  That way I could execute
> the action command with my own subprocess and get access to the output
> pipe.  Maybe you could write a spawner which did the same thing, except
> connected the pipes between subprocesses?  Or if you know the
> intermediate output is small enough, you could read it all into memory
> and then feed that as the input to the next process in the pipe, without
> using disk space, and still being able to check the exit status of each
> process in the chain.  If I ever need to run a pipe on one of our
> Windows builds, without a bash shell, I guess I'd have to try an
> approach like that.
>
> In case it's useful, the code I'm talking about is the LogAction and
> _SpawnerLogger classes in this tool:
>
>
> https://github.com/ncareol/eol_scons/blob/master/eol_
> scons/tools/testing.py
>
> The doc string may not be up to date, so user beware.
>
> Gary
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20171005/334e95ed/attachment-0001.html>