[Scons-users] CompilationDatabase support for SCons

[Andrew C. Morrow]

In the interest of completeness, here is the SCons Tool that I first wrote,
which generates the compile_commands.json file by way of actions and
emitters. It basically works, with two caveats:

- Enabling it before running Configure checks seems not to work if any of
your configure checks fail. You can just enable it with
env.Tool('compiledb') after running all configure checks.

- It has the side effect that if you build the compilation database, that
all of the .o's are generated. This was unacceptable in the project I would
like to use it for, so this was not a viable solution for me.

Suggestions for improvements or workarounds for either of the above issues
is greatly appreciated.

Thanks,
Andrew


----- site_scons/site_tools/compiledb.py ----

import json
import SCons
import itertools

# TODO: Is there a better way to do this than this global? Right now this
exists so that the
# emitter we add can record all of the things it emits, so that the scanner
for the top level
# compilation database can access the complete list, and also so that the
writer has easy
# access to write all of the files. But it seems clunky.  How can the
emitter and the scanner
# communicate more gracefully?
COMPILATION_DB_ENTRIES = []

def MakeCompilationDbEntryAction(com):

    def CompilationDbEntryAction(target, source, env):
        with open(str(target[1]), 'wb') as out:
            # TODO: IS using abspath for directory correct here?
            # TODO: How would we deal with a builder that did a chdir?
            entry = {
                "directory": env.Dir('#').abspath,
                "command": env.subst(com, source=source, target=target),
                "file": str(source[0]),
            }
            json.dump(entry, out)

    return SCons.Action.Action(CompilationDbEntryAction,
'$COMPILATIONDBENTRYCOMSTR')

def EmitCompilationDbEntry(target, source, env):
    dbentries = [env.File(str(t) + env['COMPILATIONDBSUFFIX']) for t in
target]
    COMPILATION_DB_ENTRIES.extend(dbentries)
    return target + dbentries, source

def WriteCompilationDb(target, source, env):
    entries = []

    for s in COMPILATION_DB_ENTRIES:
        with open(str(s), 'r') as source_file:
            entries.append(json.load(source_file))

    with open(str(target[0]), 'w') as target_file:
        json.dump(entries, target_file)

def ScanCompilationDb(node, env, path):
    return COMPILATION_DB_ENTRIES

def CompilationDatabase(env, target=None):

    builder = SCons.Builder.Builder(
        action=SCons.Action.Action(WriteCompilationDb,
"$COMPILATIONDBCOMSTR"),
        target_scanner=SCons.Scanner.Scanner(ScanCompilationDb),
    )

    target = (target or env.Dir("#").File(env['COMPILATIONDBSUFFIX'][1:]))
    return builder(env=env, target=target, source=[])

def generate(env, **kwargs):

    static_obj, shared_obj = SCons.Tool.createObjBuilders(env)

    # TODO: Is there a way to obtain the configured suffixes for C and C++
    # from the existing obj builders? Seems unfortunate to re-iterate them.
    CSuffixes = ['.c']
    CXXSuffixes = ['.cc', '.cxx', '.cpp']

    tuples = itertools.chain(
        itertools.product(CSuffixes, [
            (static_obj, SCons.Defaults.CAction,
SCons.Defaults.StaticObjectEmitter, '$CCCOM'),
            (shared_obj, SCons.Defaults.ShCAction,
SCons.Defaults.SharedObjectEmitter, '$SHCCCOM'),
        ]),
        itertools.product(CXXSuffixes, [
            (static_obj, SCons.Defaults.CXXAction,
SCons.Defaults.StaticObjectEmitter, '$CXXCOM'),
            (shared_obj, SCons.Defaults.ShCXXAction,
SCons.Defaults.SharedObjectEmitter, '$SHCXXCOM'),
        ]),
    )

    env['COMPILATIONDBSUFFIX'] = ".compile_commands.json"
    env['COMPILATIONDBCOMSTR'] = "Building compilation database $TARGET"
    env['COMPILATIONDBENTRYCOMSTR'] = "Writing compilation database entry
for $SOURCE"

    for tup in tuples:
        suffix = tup[0]
        builder, base_action, base_emitter, command = tup[1]

        builder.add_action(
            suffix, SCons.Action.ListAction(
                [
                    MakeCompilationDbEntryAction(command),
                    base_action,
                ]
            ))

        builder.add_emitter(
            suffix, SCons.Builder.ListEmitter(
                [
                    EmitCompilationDbEntry,
                    base_emitter,
                ]
            ))

    env.AddMethod(CompilationDatabase, "CompilationDatabase")

def exists(env):
    return True



On Sun, Dec 21, 2014 at 5:55 PM, Andrew C. Morrow <andrew.c.morrow at gmail.com
> wrote:

>
> I spent some time working on this, because having compile_commands.json is
> hugely useful.
>
> I spent a great deal of time trying to use emitters and actions to do
> this, trying to have every C or C++ object not just emit a foo.o for foo.c,
> but also emit a foo.o.compile_commands.json file as well, and then having a
> top level #compile_commands.json depend on all of those
> .o.compile_commands.json files.
>
> This worked well except that it meant that that if you built
> x.o.compile_commands.json, then x.o was built as well. That defeats the
> purpose, since it should be fast to compute the compilation database and
> not require that you actually build each target. I don't entirely
> understand why given a graph like this
>
> node src/foo.c emits src/foo.o
> node src/foo.c emits src/foo.o.compile_commands.json
> node src/foo.c has an action that produces src/foo.o
> node src/foo.c has an action that produces src/foo.o.compile_commands.json
> node #compile_commands.json depends on src/foo.o.compile_commands.json
>
> that requesting #compile_commands.json causes src/foo.o to be built, since
> there isn't a dependency path from one to the other. My guess is that if
> any emitted object from a node is depended on, then all of the actions for
> that node must be run, since SCons cannot determine which action leads to
> which emitted item? Is there a way around this? If so then I can pursue
> this again, perhaps.
>
> After getting stuck on that issue, I settled upon the following extremely
> hacked approach, which I'd like to share. I'm not exactly proud of this,
> but it does seem to produce a working compilation database:
>
> if write_compilation_database:
>
>     # We want generating the DB to be fast, so we don't execute the real
> build actions. We are
>     # going to abuse the facility that handles xCOMSTR to write the db.
>     SetOption('no_exec', True)
>
>     # We want everything to be out of date so that we build it.
>     def AlwaysOutOfDate(x, y, z):
>         return True;
>     env.Decider(AlwaysOutOfDate)
>
>     # We also want to build everything (unless otherwise asked) so that
> the database reflects
>     # all targets.
>     env.Default("#")
>
>     # Do we actually need the lock in a no-exec build Seems like no?
>     compdb_lock = threading.Lock()
>     compdb_items = []
>
>     # A factory for the action we are going to use instead of the normal C
> and C++ action. The
>     # action it self is a no-op, which is fine since it isn't going to get
> executed anyway. We
>     # use the strfunc (which is invoked in a no-exec build) to write the
> entry for the current
>     # target into the list of compilation database entries.
>     def MakeCompDbStringFunction(command):
>
>         def NopAction(target, source, env):
>             pass
>
>         def AddToCompDb(target, source, env):
>             compile_command = env.subst(command, source=source,
> target=target)
>             with compdb_lock:
>                 compdb_items.append({
>                     # TODO: Is the abspath here necessary? Or can we just
> say '.'? I think
>                     # that depends on the semantics of compilation
> database, but abspath
>                     # seems safe for sure.
>                     "directory" : env.Dir('#').abspath,
>                     "command" : compile_command,
>                     "file": str(source[0])
>                 })
>             return "Calculating compilation DB entry for %s" %
> str(source[0])
>
>         return Action(NopAction, strfunction=AddToCompDb)
>
>     static, shared = SCons.Tool.createObjBuilders(env)
>
>     # TODO: Make this a little more declarative. How do we get all the
>     # suffixes that SCons has registered?
>     static.add_action(".c", MakeCompDbStringFunction("$CCCOM"))
>     static.add_action(".cpp", MakeCompDbStringFunction("$CXXCOM"))
>     static.add_action(".cc", MakeCompDbStringFunction("$CXXCOM"))
>     shared.add_action(".c", MakeCompDbStringFunction("$SHCCCOM"))
>     shared.add_action(".cpp", MakeCompDbStringFunction("$SHCXXCOM"))
>     shared.add_action(".cc", MakeCompDbStringFunction("$SHCXXCOM"))
>
>     # Register an atexit handler to serialize all of the compilation
> database entries we
>     # recorded.
>     def writeCompDb():
>         print("Writing compile_commands.json...")
>         with open(env.File("#compile_commands.json").abspath, "w") as
> compdb:
>             with compdb_lock:
>                 compdb.truncate()
>                 json.dump(compdb_items, compdb)
>     import atexit
>     atexit.register(writeCompDb)
>
> This works best if you toggle no_exec after executing any configure
> checks, since otherwise you get the complaint that you cannot execute
> configured checks in a dry run.
>
> I still think that the best way to do this would be if SCons itself was
> aware of how to emit the compilation database, but short of that the above
> mechanism does seem to work.
>
> Thanks,
> Andrew
>
>
> On Sat, Aug 3, 2013 at 10:51 AM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>> I'm interested in writing some clang tools with LibTooling for a codebase
>> that uses SCons as its build system. LibTooling is driven by an artifact
>> known as a CompilationDatabase, which captures state about the build so
>> that the tool can be consistently invoked:
>>
>> http://clang.llvm.org/docs/JSONCompilationDatabase.html
>>
>> CMake has support for emitting a CompilationDatabase as a side effect of
>> a build. I'd like to consider adding similar support to SCons. Before I
>> start, though, a few questions:
>>
>> - Would such support be considered for inclusion in SCons?
>>
>> - If so, is anyone already working on this? I'd prefer not to duplicate
>> the effort.
>>
>> - There was recently a thread here about support for considering all
>> targets out of date. I believe such a thing would be needed to produce a
>> compilation database as well (probably along with running in --dry-run
>> mode). The solution in that thread was a custom Decider, but this doesn't
>> work well if the goal is to support emitting a compilation database without
>> users needing to customize the Environment. Would support for emitting a
>> CompilationDatabase justify adding direct support for the "consider all out
>> of date" feature?
>>
>> - What, generally, is the right place to begin? Should
>> CompilationDatabase support be a tool? A side effect of invoking the Object
>> and SharedObject builders? How should this be done if the goal is to
>> support this in such a way that it continues to operate if users customize
>> their own Object builders?
>>
>> Thanks,
>> Andrew
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141223/631edc7c/attachment-0001.html>