[Scons-users] Using a Decider to prevent shared lib relinks

[orenaud at coventor.com]

Thanks, your message and the recent message from Andrew C. Morrow ("My own Decider function") put me on the right track. The missing piece for me was to understand that it is possible, and even necessary, to assign the value of csig in this decider function. I now realize that it is more or less mentioned in the doc, in this excerpt:
Note how the signature information for the dependency file has to get initialized via get_csig during each function call (this is mandatory!)
Here is the solution I came up with. From my limited testing, it seems to do what I want:


def compute_symbols_md5(lib_filename):
    dumpbin = WhereIs('dumpbin', env['ENV']['PATH'])
    s = subprocess.check_output([dumpbin, '/exports', lib_filename])
    return SCons.Util.MD5signature(s)

def is_lib(filename, env):
    return filename.startswith(env.subst('$LIBPREFIX')) and filename.endswith(env.subst('$LIBSUFFIX'))

def gen_my_decider(env):
    default_decider = env.decide_target

    def my_decider(dep, tgt, prev_ni, repo_node=None):
        if not is_lib(str(dep), env):
            return default_decider(dep, tgt, prev_ni, repo_node)

        dep_node_info = dep.get_ninfo()

        if not hasattr(dep.attributes, 'csig_is_lib_symbols_md5'):
            dep_node_info.csig = compute_symbols_md5(str(dep))
            dep.attributes.csig_is_lib_symbols_md5 = object()

        if prev_ni is None or not hasattr(prev_ni, 'csig'):
            return True

        return prev_ni.csig != dep_node_info.csig

    return my_decider

my_env.Decider(gen_my_decider(my_env))



I use `csig_is_lib_symbols_md5` to mark the lib nodes that contain a md5 of the dumped symbols in `csig`, as opposed to a md5 of the file content (it works because the lib file is always represented by the same instance of a Node during a build session). I introduced `gen_my_decider` so that I have a place to store the "default" decider to use for non-lib files. I still need to distinguish between static lib file and dynamic lib file on windows.

On 5/25/2020 11:26 PM, Bill Deegan wrote:
Olivier,

You're not the first to head down this path as it would be useful to (configurably) avoid such unnecessary relinks.
I think MongoDB has implemented something like this.

Please keep in mind that dep and tgt are not strings, the are File() nodes, which have access to information stored in .sconsign with regards to the signature of the file from the previous build.
This is (currently) a md5sum of the full contents of the file.
So to adequately do what you're thinking you'd need a way to store the ABI info (or a hash thereof) as the content signature (also known as csig) for the node(s) in question.

So it's going to be a pretty deep dive into the code.

Perhaps someone from MongoDB will chime in here with a pointer to their code/experience.

-Bill

On Mon, May 25, 2020 at 10:11 AM <orenaud at coventor.com<mailto:orenaud at coventor.com>> wrote:

Hi,

I want to prevent unnecessary re-links against my shared libraries. By "unnecessary", I mean the re-links that happen even when the list of exported symbols did not change.

In practice, I would like to use the output of `nm --extern-only` on Linux or `dumpbin /exports` on Windows as the content to use for the signature of the shared library.

I was able to approximate this on a simple example, by explicitly specifying the dependency of a target (myExe) that links against a specific shared library (myLib.lib):

def dumpLibSymbols(target, source, env):
    with open(str(target[0]), 'w') as dumpFile:
        nm = WhereIs('nm', env['ENV']['PATH'])
        subprocess.call([nm, '--extern-only', str(source[0])], stdout=dumpFile)

dumpLibSymbolsAction = Action(dumpLibSymbols,
                              "Dumping Symbols for $SOURCE into $TARGET")
dumpLibSymbolsBuilder = Builder(action=dumpLibSymbolsAction,
                                src_suffix="$LIBSUFFIX",
                                suffix=".lib.exports")
env.Append(BUILDERS= {'DumpLibSymbols': dumpLibSymbolsBuilder})

Ignore(myExe, 'myLib.lib')
Depends(myExe, 'myLib.lib.exports')


Basically, it changes the dependency chain [myExe -> myLib.lib] to [myExe -> myLib.lib.exports -> myLib.lib], without changing the link command itself.

While this example works, I am unable to generalize it: the calls to Ignore and Depends must be explicit for all the pairs exe/lib (or lib/lib). What I want is for it to be automatic for anything that depends on a shared library.

It seems to me that the Decider function (https://scons.org/doc/production/HTML/scons-user/ch06.html#idm962<https://urldefense.proofpoint.com/v2/url?u=https-3A__scons.org_doc_production_HTML_scons-2Duser_ch06.html-23idm962&d=DwMFaQ&c=RWI7EqL8K9lqtga8KxgfzvOYoob76EZWE0yAO85PVMQ&r=4kZkTHayCjeGAQcCy0E295XGr9xJGf-CHs_MqTnCu2I&m=BObPKvTudrvwwNERpOKI_nAnPLa7214U5QnkFJLchJs&s=w6CW5DrsZgcPu8gkgq_bP5jjjWlISlfBgzcaWyEoSLM&e=>) is exactly what I need. Indeed, I want to decide a target is out of date based on the output of nm/dumpbin. The problem is that I don't understand how I can do that in practice. The example in the docs uses an unspecified `specific_part_of_file_has_changed(dep, tgt)` function, but I don't see how this function can do its job given its inputs. To me, this function needs an additional information: the previous content of the dependency file (or a hash of only the part it is interested in). Similarly, my custom Decider would also need to know the previous content of nm/dumpbin in addition to its current content.

Can someone points me to a real world usage of the Decider function? Is it the right tool I need, or did I overlook a better way to achieve my goal?

Thanks,

Olivier Renaud

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://urldefense.proofpoint.com/v2/url?u=https-3A__pairlist4.pair.net_mailman_listinfo_scons-2Dusers&d=DwMFaQ&c=RWI7EqL8K9lqtga8KxgfzvOYoob76EZWE0yAO85PVMQ&r=4kZkTHayCjeGAQcCy0E295XGr9xJGf-CHs_MqTnCu2I&m=BObPKvTudrvwwNERpOKI_nAnPLa7214U5QnkFJLchJs&s=fHlnPOUCfVYT_y1hB7DzmO1UwCCHOf89CzXmmg0i1gk&e=>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20200526/9cf3c643/attachment.html>