[Scons-users] Code not rebuilt when switching between two VisualStudio versions
Steve Hill (Wireless)
Steve.Hill1 at viavisolutions.com
Fri Apr 9 02:31:45 EDT 2021
I’m not sure how to create a pull-request for this since the code is outside of the SCons codebase. However, the code is below if it helps in any way.
The other patches are around database storage (there was a thread on this a few years back as, at the time, we were using a 32-bit version of Python and the use of a single, monolithic SConsign was causing us to exhaust the 2GB memory space) and a patch to the underlying Python, actually, to retry failing steps under some circumstance as we get very rare Access Denied problems that we’ve never been able to get to the bottom of (due to the inability to reproduce it) so we retry to avoid it breaking our CI machine. I’m not sure that the latter is generically useful and I seem to remember that there was something already in progress on redesigning the SConsign side of things – although I’m happy to share the patch, if it is useful in any way.
Thanks again,
S.
The patching of get_implicit_deps
def _patch_implicit_deps():
"""
For some tools, e.g. the MSVS compiler, we can sometimes have one executable (e.g. the
BullseyeCoverage cl.exe) call the MSVC cl.exe. In this case, SCons sees the Bullseye cl.exe
as an implicit dependency but not the MSVC one, which results in files not being rebuilt when
they should be under some circumstances. Hence, we patch the method that works out the implicit
dependencies so that the extra cl.exe is added to the list and files will be correctly
rebuilt, as a result.
"""
originalGid = SCons.Action.CommandAction.get_implicit_deps
def patched_gid(self, target, source, env, executor=None):
deps = originalGid(self, target, source, env, executor)
additional = []
try:
eid = env["EXTRA_IMPLICIT_DEPS"]
except KeyError:
# No EXTRA_IMPLICIT_DEPS dict so skip this
pass
else:
for dep in deps:
try:
additional += eid[str(dep)]
except KeyError:
# No additional dependency for this tool
pass
return deps + additional
SCons.Action.CommandAction.get_implicit_deps = patched_gid
And the code to determine the extra deps:
def _check_implicit_deps(self, tool):
# Populate the EXTRA_IMPLICIT_DEPS dict with any additional implicit dependencies.
# See workarounds.py/_patch_implicit_deps() for where this is used...
try:
eid = self._env["EXTRA_IMPLICIT_DEPS"]
except KeyError:
eid = {}
self._env["EXTRA_IMPLICIT_DEPS"] = eid
fullPath = self._env.WhereIs(tool)
if fullPath not in eid:
extraDeps = []
if "Microsoft Visual Studio" not in fullPath:
toolDir = os.path.dirname(fullPath)
newPath = [path for path in self._env["ENV"]["PATH"].split(";") if path != toolDir]
baseCompiler = self._env.WhereIs(tool, ";".join(newPath))
assert "Microsoft Visual Studio" in baseCompiler, "Didn't find expected compiler"
extraDeps.append(self._env.fs.File(baseCompiler))
eid[fullPath] = extraDeps
def find_extra_implicit_deps(self):
"""
When something like BullseyeCoverage is installed, the cl.exe invoked is the Bullseye
version so the actual compiler isn't spotted as an implicit dependency. This is an issue
when switching between Visual Studio versions (as the object files don't get rebuilt as
they should, which can cause link failures due to incompatibility in file formats) so
check the core tools and ensure that the core compiler is added to the
EXTRA_IMPLICIT_DEPS dict, which will be added by our monkey patched version of SCons.
"""
self._check_implicit_deps(self._env.subst(self._env["CC"]))
self._check_implicit_deps(self._env.subst(self._env["CXX"]))
self._check_implicit_deps(self._env.subst(self._env["LINK"]))
From: Bill Deegan
Sent: 09 April 2021 01:38
To: SCons users mailing list <scons-users at scons.org>
Cc: Steve Hill (Wireless)
Subject: Re: [Scons-users] Code not rebuilt when switching between two VisualStudio versions
Steve,
Glad we could get you to a solution!
This is a new use case for me.. and I've been around a while. ;)
If you're up for it, create a pull request for your changes?
We'll see if we can get it into a form which makes sense for general use (docs,tests as well)?
https://github.com/SCons/scons/wiki/GitWorkflows<https://urldefense.com/v3/__https:/github.com/SCons/scons/wiki/GitWorkflows__;!!Niha4SQ!qOO0pPNQSMayDdtpAuKsHSVx1eFJi5MPgVlJJpdecJLqF2b-8pb62oS0EPVzfuGpg5v-YkTJ$>
Any other patches you have in place you'd like to share and/or are general?
-Bill
On Wed, Apr 7, 2021 at 11:26 PM Steve Hill (Wireless) via Scons-users <scons-users at scons.org<mailto:scons-users at scons.org>> wrote:
Hi Bill,
Thanks for this guidance. I’ve put something like this in place and it seems to have the desired effect.
I do wonder whether providing some official mechanism for doing this would be worthwhile (like adding the core compiler as a dependency of the coverage compiler, which can already be done but doesn’t – currently – work)? We have a few tools that have a compiler which ends up calling the MSVC compiler, e.g. Dynamic Analysis tools. I’m not against patching SCons (we already have a couple of other patches) but this does seem to be a fairly standard pattern, in the Windows world, at least. The patch that I put in place is generic, so we can use it for all tools that need to add this dependency, but it does feel like a use-case that SCons should be able to support without patching?
Anyway, my problem is resolved so thanks once again!
S.
From: Bill Deegan
Sent: 06 April 2021 22:50
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Cc: Steve Hill (Wireless)
Subject: Re: [Scons-users] Code not rebuilt when switching between two VisualStudio versions
Steve,
O.k.
So the way SCons typically adds the compiler as an implicit dependency works by checking the first item on the command line and then checking in env['ENV']['PATH'] to find the first one.
The function which does that is in SCons/Action.py
CommandAction's method get_implicit_deps().
You might be able to "monkey patch" that function to always add the second path in PATH where it finds cl.exe as well.
import SCons.Action
old_get_implicit_deps = SCons.Action.CommandAction.get_implicit_deps
def my_get_implicit_deps(self, target, source, env, executor=None):
ret_list = old_get_implicit_deps(self, target, source,env, executor)
if PATH_TO_FAKE_CL in ret_list:
ret_list.append(PATH_TO_REAL_CL)
return ret_list
SCons.Action.CommandAction.get_implicit_deps = my_get_implicit_deps
Or something like that.
A bit of a hack, but then this is a hacky situation...
-Bill
On Tue, Apr 6, 2021 at 6:57 AM Steve Hill (Wireless) via Scons-users <scons-users at scons.org<mailto:scons-users at scons.org>> wrote:
Hi Bill,
>> Is there a way by running the coverage version of cl.exe to find out the path of the real CL.exe?
Sorry for the delay in replying. I’ve now had a chance to have a play and, as far as I can tell, the coverage cl.exe just passes the command-line arguments on to the underlying cl.exe. If there isn’t one available, I get:
$ ./cl
Exception: compiler not found cl.exe
And I get the same regardless of the arguments that are added to the command line…
Anyway, I could figure out the real cl.exe (by removing the coverage bin directory from the path and using shutil.which()) but I’m not sure how to then make this a default dependency without explicitly adding it to every object file, which would be quite unpleasant.
Thanks,
S.
From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> On Behalf Of Bill Deegan
Sent: 26 March 2021 17:57
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] Code not rebuilt when switching between two VisualStudio versions
Is there a way by running the coverage version of cl.exe to find out the path of the real CL.exe?
On Fri, Mar 26, 2021 at 7:54 AM Mats Wichmann <mats at wichmann.us<mailto:mats at wichmann.us>> wrote:
On 3/26/21 1:52 AM, Steve Hill (Wireless) via Scons-users wrote:
> Hi Bill,
>
> The command line is unchanged. When you install the coverage tool, it
> results in vcvarsall.bat adding the path to the coverage ‘bin’ directory
> ahead of the one for MSVC:
>
> PATH=*C:\Program Files (x86)\BullseyeCoverage\bin*;C:\Program Files
> (x86)\Microsoft Visual Studio
> 14.0\VC\Tools\MSVC\14.15.26726\bin\HostX64\x64;C:\Program Files
> (x86)\Microsoft Visual Studio 14.0\Common7\IDE\VC\VCPackages;C:\Program
> Files (x86)\...
>
> That means that the cl.exe and link.exe that are invoked are the
> coverage versions, which then do what they need to do before invoking
> the real compiler or linker (presumably using the directories in the
> PATH other than its own.) We have other tools (e.g. dynamic code
> analysis) that work in the same way – it seems quite standard, under
> Windows at least; this just happens to be the first tool where we have
> encountered it.
>
> Since it supports multiple versions of MSVS, it may call the compiler
> for either MSVS 2017 or MSVS 2019 – but SCons has no way of knowing
> that. I figure that I need to somehow tell SCons that this extra
> dependency exists.
>
> I’ve tried:
>
> env.Depends(
One of the decision points as to whether to rebuild is whether the
string which will be issued as the command to build a file (the
builder's "action") has changed. The string is hashed, and that is
compared with the action signature stored away from earlier builds. The
problem you're having here is that the action doesn't contain the actual
compiler name/path, so SCons sees no difference.
There is a way to add things to be considered as part of an action
signature - namely you can include construction variables, whose values
will be slurped up and included. It looks something like this (from the
manpage):
a = Action(build_it, varlist=['XXX'])
The problem is you're not writing your own actions, you want this to
happen for pre-defined actions attached to the SCons-provided builders
you're using (I assume you're using the standard ones like Program,
Object, etc.). Changing an existing action is tricky.
You can read a bit more here:
https://scons.org/doc/production/HTML/scons-man.html#action_objects<https://urldefense.com/v3/__https:/scons.org/doc/production/HTML/scons-man.html*action_objects__;Iw!!Niha4SQ!qR9jdHlKZb4Q12-1IWkJkaq7WqSre3SA9Usoz8hb3Xr7kZ-Nkq-qaqReYIn3r8L4rq65oc3a$>
Maybe someone else knows another route?
(I'm going to look at eventually adding something on this somewhere,
either docs or scons-recipes, you're right that the concept of a tool
masquerading as another tool is relatively common - on Linux/UNIX ccache
is one of those. This means SCons doesn't see the actual one, and might
not detect changes).
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://urldefense.com/v3/__https:/pairlist4.pair.net/mailman/listinfo/scons-users__;!!Niha4SQ!qR9jdHlKZb4Q12-1IWkJkaq7WqSre3SA9Usoz8hb3Xr7kZ-Nkq-qaqReYIn3r8L4ru8AVJjU$>
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://urldefense.com/v3/__https:/pairlist4.pair.net/mailman/listinfo/scons-users__;!!Niha4SQ!tnrUVzUNvgvY_UnajnO9Y21xMpMm0NoreufM-0h94hbZOvFqt78ZUi0IIDL4MDO8xlSCN0Oe$>
_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://urldefense.com/v3/__https:/pairlist4.pair.net/mailman/listinfo/scons-users__;!!Niha4SQ!qOO0pPNQSMayDdtpAuKsHSVx1eFJi5MPgVlJJpdecJLqF2b-8pb62oS0EPVzfuGpgwFGvM7W$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20210409/628d3cf1/attachment-0001.htm>
More information about the Scons-users
mailing list