[Scons-users] TempFileMunge conflicts with ccache

[Mathew Robinson]

Hey Daniel,

MongoDB actually fixed this but using $() which tells subst to not use the CCACHE as part of the temp file content. 

You can see it here:

https://github.com/mongodb/mongo/blob/master/site_scons/site_tools/ccache.py

Sent from my iPhone

> On 11 May 2020, at 18:03, Daniel Moody <dmoody256 at gmail.com> wrote:
> 
> 
> I was recently messing with ccache and ran into this, and I noticed a github issue was never created (or I couldn't find one:  https://github.com/SCons/scons/issues?q=is%3Aissue+ccache)
> 
> So this happens if you set the "compiler" (e.g. $CXX) to a two word statement like for example:
> 
> env['CXX'] = 'ccache ' +  env['CXX'] 
> 
> The problem with this approach is that this gets wrapped into the tempfile command itself, for example in the msvc tool:
> https://github.com/SCons/scons/blob/5a438e635699dfee672f0bcbbef79b57e79b4bcd/SCons/Tool/msvc.py#L254  
> 
> Seems the tempfile code is expecting the "compiler" to be a single word argument from this line where it takes only the first arg based off spaces in the cmd (it seems to assume that cmd[0] is equal to $CXX in the above example):
> https://github.com/SCons/scons/blob/5a438e635699dfee672f0bcbbef79b57e79b4bcd/SCons/Platform/__init__.py#L252  
> 
> There is a question here if setting a "compiler" to a two word statement is allowable? I didn't find any documentation acknowledging it or forbidding it, but it seems to happen to work in some cases.
> 
> The above example will work if you remove ccache out of the "compiler" statement and put it into the full command statement like this for example:
> 
> env['CXXCOM'] = 'ccache ' +  env['CXXCOM']   
> 
> That results in tempfile link being correctly given to the "compiler" and 'ccache' still sitting out in front of the final command like it is supposed to. This does have a draw back that you can't set CXX from the command line potentially, or rather it would be difficult to set CXXCOM from the command line.
> 
> So seems there should be a github issue to update either tempfile to work with two word "compilers"  OR documentation discouraging or denying support for two word compiler statements
> 
> 
>> On Wed, Dec 19, 2018 at 10:45 AM Andrew C. Morrow <andrew.c.morrow at gmail.com> wrote:
>> 
>> In this case, I'm sort of just relaying word from someone who was trying to build MongoDB this way. I don't use ccache, I use the SCons cache. But I thought the assumption baked into Action was worth flagging.
>> 
>>> On Wed, Dec 19, 2018 at 10:12 AM Mats Wichmann <mats at wichmann.us> wrote:
>>> On 12/19/18 8:03 AM, Andrew C. Morrow wrote:
>>> > While investigating this issue, I think I discovered a potentially more
>>> > serious issue. Evaluation of a build where CC is set to a multi token value
>>> > like ccache gcc will not rebuild files if the compiler is upgraded, because
>>> > CommandAction.get_implicit_deps only considers the first token of the
>>> > command line as an implicit dependency which must be checked for
>>> > up-to-date-ness. In the case of CC="ccache gcc" build, that would result in
>>> > a rebuild when the ccache binary changed, but not when the gcc binary
>>> > changed.
>>> > 
>>> > I'm not sure if there is anything really that can be done about that, but
>>> > maybe some guard rails around setting tool variables to multi token values
>>> > would be a good idea?
>>> > 
>>> > 
>>> > 
>>> > On Thu, Dec 6, 2018 at 12:14 PM Andrew C. Morrow <andrew.c.morrow at gmail.com>
>>> > wrote:
>>> > 
>>> >>
>>> >> In the MongoDB build, we enable TempFileMunge for non-windows platforms,
>>> >> since sometimes we have truly astronomically long link lines which break
>>> >> even the linux command line length limitations.
>>> >>
>>> >> https://github.com/mongodb/mongo/blob/master/SConstruct#L1478-L1496
>>> >>
>>> >> A user reported that their builds failed when trying to use ccache:
>>> >> https://jira.mongodb.org/browse/SERVER-38389
>>> >>
>>> >> It appears that TempFileMunge just assumes that only the first token of
>>> >> the command is actually the "compiler", and ends up feeding the response
>>> >> file directly to ccache, rather than invoking, say `ccache
>>> >> g++ @/path/to/temp/file`.
>>> >>
>>> >> On the other hand, could this even work in principle? Is ccache smart
>>> >> enough to know to "see into" the response file in order to interact with
>>> >> its cache correctly? This CMake bug suggests not:
>>> >> https://github.com/kripken/emscripten/issues/4750
>>> >>
>>> >> I did steer the user towards using the SCons cache instead of ccache, and
>>> >> provided a workaround of setting MAXLINELENGTH=<huge-number>
>>> >>
>>> >> At the very least, if ccache and response files can't be mixed, perhaps
>>> >> TempFileMunge could include some guard rails here to bail out if it sees
>>> >> the first token to be ccache.
>>> >>
>>> >> Thoughts?
>>> 
>>> 
>>> Isn't the more normal usage model to let ccache masquerade as gcc, in
>>> which case there's only one compiler token value on the line?
>>> 
>>> That is, from the manpage:
>>> 
>>> SYNOPSIS
>>>        ...
>>>        ccache compiler [compiler options]
>>>        compiler [compiler options]                   (via symbolic link)
>>> 
>>> use the second form?  that doesn't necessarily excuse scons not
>>> understanding about the other scenario, but could provide a way forward.
>>> 
>>> 
>>> 
>>> 
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20200511/3ea1649e/attachment-0001.html>