[Scons-users] CompilationDatabase support for SCons

[Andrew C. Morrow]

I spent some time working on this, because having compile_commands.json is
hugely useful.

I spent a great deal of time trying to use emitters and actions to do this,
trying to have every C or C++ object not just emit a foo.o for foo.c, but
also emit a foo.o.compile_commands.json file as well, and then having a top
level #compile_commands.json depend on all of those
.o.compile_commands.json files.

This worked well except that it meant that that if you built
x.o.compile_commands.json, then x.o was built as well. That defeats the
purpose, since it should be fast to compute the compilation database and
not require that you actually build each target. I don't entirely
understand why given a graph like this

node src/foo.c emits src/foo.o
node src/foo.c emits src/foo.o.compile_commands.json
node src/foo.c has an action that produces src/foo.o
node src/foo.c has an action that produces src/foo.o.compile_commands.json
node #compile_commands.json depends on src/foo.o.compile_commands.json

that requesting #compile_commands.json causes src/foo.o to be built, since
there isn't a dependency path from one to the other. My guess is that if
any emitted object from a node is depended on, then all of the actions for
that node must be run, since SCons cannot determine which action leads to
which emitted item? Is there a way around this? If so then I can pursue
this again, perhaps.

After getting stuck on that issue, I settled upon the following extremely
hacked approach, which I'd like to share. I'm not exactly proud of this,
but it does seem to produce a working compilation database:

if write_compilation_database:

    # We want generating the DB to be fast, so we don't execute the real
build actions. We are
    # going to abuse the facility that handles xCOMSTR to write the db.
    SetOption('no_exec', True)

    # We want everything to be out of date so that we build it.
    def AlwaysOutOfDate(x, y, z):
        return True;
    env.Decider(AlwaysOutOfDate)

    # We also want to build everything (unless otherwise asked) so that the
database reflects
    # all targets.
    env.Default("#")

    # Do we actually need the lock in a no-exec build Seems like no?
    compdb_lock = threading.Lock()
    compdb_items = []

    # A factory for the action we are going to use instead of the normal C
and C++ action. The
    # action it self is a no-op, which is fine since it isn't going to get
executed anyway. We
    # use the strfunc (which is invoked in a no-exec build) to write the
entry for the current
    # target into the list of compilation database entries.
    def MakeCompDbStringFunction(command):

        def NopAction(target, source, env):
            pass

        def AddToCompDb(target, source, env):
            compile_command = env.subst(command, source=source,
target=target)
            with compdb_lock:
                compdb_items.append({
                    # TODO: Is the abspath here necessary? Or can we just
say '.'? I think
                    # that depends on the semantics of compilation
database, but abspath
                    # seems safe for sure.
                    "directory" : env.Dir('#').abspath,
                    "command" : compile_command,
                    "file": str(source[0])
                })
            return "Calculating compilation DB entry for %s" %
str(source[0])

        return Action(NopAction, strfunction=AddToCompDb)

    static, shared = SCons.Tool.createObjBuilders(env)

    # TODO: Make this a little more declarative. How do we get all the
    # suffixes that SCons has registered?
    static.add_action(".c", MakeCompDbStringFunction("$CCCOM"))
    static.add_action(".cpp", MakeCompDbStringFunction("$CXXCOM"))
    static.add_action(".cc", MakeCompDbStringFunction("$CXXCOM"))
    shared.add_action(".c", MakeCompDbStringFunction("$SHCCCOM"))
    shared.add_action(".cpp", MakeCompDbStringFunction("$SHCXXCOM"))
    shared.add_action(".cc", MakeCompDbStringFunction("$SHCXXCOM"))

    # Register an atexit handler to serialize all of the compilation
database entries we
    # recorded.
    def writeCompDb():
        print("Writing compile_commands.json...")
        with open(env.File("#compile_commands.json").abspath, "w") as
compdb:
            with compdb_lock:
                compdb.truncate()
                json.dump(compdb_items, compdb)
    import atexit
    atexit.register(writeCompDb)

This works best if you toggle no_exec after executing any configure checks,
since otherwise you get the complaint that you cannot execute configured
checks in a dry run.

I still think that the best way to do this would be if SCons itself was
aware of how to emit the compilation database, but short of that the above
mechanism does seem to work.

Thanks,
Andrew


On Sat, Aug 3, 2013 at 10:51 AM, Andrew C. Morrow <andrew.c.morrow at gmail.com
> wrote:

>
> I'm interested in writing some clang tools with LibTooling for a codebase
> that uses SCons as its build system. LibTooling is driven by an artifact
> known as a CompilationDatabase, which captures state about the build so
> that the tool can be consistently invoked:
>
> http://clang.llvm.org/docs/JSONCompilationDatabase.html
>
> CMake has support for emitting a CompilationDatabase as a side effect of a
> build. I'd like to consider adding similar support to SCons. Before I
> start, though, a few questions:
>
> - Would such support be considered for inclusion in SCons?
>
> - If so, is anyone already working on this? I'd prefer not to duplicate
> the effort.
>
> - There was recently a thread here about support for considering all
> targets out of date. I believe such a thing would be needed to produce a
> compilation database as well (probably along with running in --dry-run
> mode). The solution in that thread was a custom Decider, but this doesn't
> work well if the goal is to support emitting a compilation database without
> users needing to customize the Environment. Would support for emitting a
> CompilationDatabase justify adding direct support for the "consider all out
> of date" feature?
>
> - What, generally, is the right place to begin? Should CompilationDatabase
> support be a tool? A side effect of invoking the Object and SharedObject
> builders? How should this be done if the goal is to support this in such a
> way that it continues to operate if users customize their own Object
> builders?
>
> Thanks,
> Andrew
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141221/90d6f82e/attachment.html>