[Scons-users] CompilationDatabase support for SCons
Andrew C. Morrow
andrew.c.morrow at gmail.com
Sat Jan 10 12:15:15 EST 2015
Hi All -
I must admit I'm somewhat disappointed that there appears to be no interest
in discussing on this list how to get SCons to emit a compilation database.
This is an incredibly useful feature that other build systems like CMake
and Ninja offer, and enables deep integration between the build system,
compiler, and IDE or editor. Many clang tools (clang-modernize, clang-tidy,
etc.) either require or work much better with a compilation database.
I see no reason why SCons could not or should not provide this
functionality as well.
Is there any feedback on my above attempts, or thoughts about how this
should best be approached?
Thanks,
Andrew
On Tue, Dec 23, 2014 at 9:58 AM, Andrew C. Morrow <andrew.c.morrow at gmail.com
> wrote:
>
> In the interest of completeness, here is the SCons Tool that I first
> wrote, which generates the compile_commands.json file by way of actions and
> emitters. It basically works, with two caveats:
>
> - Enabling it before running Configure checks seems not to work if any of
> your configure checks fail. You can just enable it with
> env.Tool('compiledb') after running all configure checks.
>
> - It has the side effect that if you build the compilation database, that
> all of the .o's are generated. This was unacceptable in the project I would
> like to use it for, so this was not a viable solution for me.
>
> Suggestions for improvements or workarounds for either of the above issues
> is greatly appreciated.
>
> Thanks,
> Andrew
>
>
> ----- site_scons/site_tools/compiledb.py ----
>
> import json
> import SCons
> import itertools
>
> # TODO: Is there a better way to do this than this global? Right now this
> exists so that the
> # emitter we add can record all of the things it emits, so that the
> scanner for the top level
> # compilation database can access the complete list, and also so that the
> writer has easy
> # access to write all of the files. But it seems clunky. How can the
> emitter and the scanner
> # communicate more gracefully?
> COMPILATION_DB_ENTRIES = []
>
> def MakeCompilationDbEntryAction(com):
>
> def CompilationDbEntryAction(target, source, env):
> with open(str(target[1]), 'wb') as out:
> # TODO: IS using abspath for directory correct here?
> # TODO: How would we deal with a builder that did a chdir?
> entry = {
> "directory": env.Dir('#').abspath,
> "command": env.subst(com, source=source, target=target),
> "file": str(source[0]),
> }
> json.dump(entry, out)
>
> return SCons.Action.Action(CompilationDbEntryAction,
> '$COMPILATIONDBENTRYCOMSTR')
>
> def EmitCompilationDbEntry(target, source, env):
> dbentries = [env.File(str(t) + env['COMPILATIONDBSUFFIX']) for t in
> target]
> COMPILATION_DB_ENTRIES.extend(dbentries)
> return target + dbentries, source
>
> def WriteCompilationDb(target, source, env):
> entries = []
>
> for s in COMPILATION_DB_ENTRIES:
> with open(str(s), 'r') as source_file:
> entries.append(json.load(source_file))
>
> with open(str(target[0]), 'w') as target_file:
> json.dump(entries, target_file)
>
> def ScanCompilationDb(node, env, path):
> return COMPILATION_DB_ENTRIES
>
> def CompilationDatabase(env, target=None):
>
> builder = SCons.Builder.Builder(
> action=SCons.Action.Action(WriteCompilationDb,
> "$COMPILATIONDBCOMSTR"),
> target_scanner=SCons.Scanner.Scanner(ScanCompilationDb),
> )
>
> target = (target or env.Dir("#").File(env['COMPILATIONDBSUFFIX'][1:]))
> return builder(env=env, target=target, source=[])
>
> def generate(env, **kwargs):
>
> static_obj, shared_obj = SCons.Tool.createObjBuilders(env)
>
> # TODO: Is there a way to obtain the configured suffixes for C and C++
> # from the existing obj builders? Seems unfortunate to re-iterate them.
> CSuffixes = ['.c']
> CXXSuffixes = ['.cc', '.cxx', '.cpp']
>
> tuples = itertools.chain(
> itertools.product(CSuffixes, [
> (static_obj, SCons.Defaults.CAction,
> SCons.Defaults.StaticObjectEmitter, '$CCCOM'),
> (shared_obj, SCons.Defaults.ShCAction,
> SCons.Defaults.SharedObjectEmitter, '$SHCCCOM'),
> ]),
> itertools.product(CXXSuffixes, [
> (static_obj, SCons.Defaults.CXXAction,
> SCons.Defaults.StaticObjectEmitter, '$CXXCOM'),
> (shared_obj, SCons.Defaults.ShCXXAction,
> SCons.Defaults.SharedObjectEmitter, '$SHCXXCOM'),
> ]),
> )
>
> env['COMPILATIONDBSUFFIX'] = ".compile_commands.json"
> env['COMPILATIONDBCOMSTR'] = "Building compilation database $TARGET"
> env['COMPILATIONDBENTRYCOMSTR'] = "Writing compilation database entry
> for $SOURCE"
>
> for tup in tuples:
> suffix = tup[0]
> builder, base_action, base_emitter, command = tup[1]
>
> builder.add_action(
> suffix, SCons.Action.ListAction(
> [
> MakeCompilationDbEntryAction(command),
> base_action,
> ]
> ))
>
> builder.add_emitter(
> suffix, SCons.Builder.ListEmitter(
> [
> EmitCompilationDbEntry,
> base_emitter,
> ]
> ))
>
> env.AddMethod(CompilationDatabase, "CompilationDatabase")
>
> def exists(env):
> return True
>
>
>
> On Sun, Dec 21, 2014 at 5:55 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>> I spent some time working on this, because having compile_commands.json
>> is hugely useful.
>>
>> I spent a great deal of time trying to use emitters and actions to do
>> this, trying to have every C or C++ object not just emit a foo.o for foo.c,
>> but also emit a foo.o.compile_commands.json file as well, and then having a
>> top level #compile_commands.json depend on all of those
>> .o.compile_commands.json files.
>>
>> This worked well except that it meant that that if you built
>> x.o.compile_commands.json, then x.o was built as well. That defeats the
>> purpose, since it should be fast to compute the compilation database and
>> not require that you actually build each target. I don't entirely
>> understand why given a graph like this
>>
>> node src/foo.c emits src/foo.o
>> node src/foo.c emits src/foo.o.compile_commands.json
>> node src/foo.c has an action that produces src/foo.o
>> node src/foo.c has an action that produces src/foo.o.compile_commands.json
>> node #compile_commands.json depends on src/foo.o.compile_commands.json
>>
>> that requesting #compile_commands.json causes src/foo.o to be built,
>> since there isn't a dependency path from one to the other. My guess is that
>> if any emitted object from a node is depended on, then all of the actions
>> for that node must be run, since SCons cannot determine which action leads
>> to which emitted item? Is there a way around this? If so then I can pursue
>> this again, perhaps.
>>
>> After getting stuck on that issue, I settled upon the following extremely
>> hacked approach, which I'd like to share. I'm not exactly proud of this,
>> but it does seem to produce a working compilation database:
>>
>> if write_compilation_database:
>>
>> # We want generating the DB to be fast, so we don't execute the real
>> build actions. We are
>> # going to abuse the facility that handles xCOMSTR to write the db.
>> SetOption('no_exec', True)
>>
>> # We want everything to be out of date so that we build it.
>> def AlwaysOutOfDate(x, y, z):
>> return True;
>> env.Decider(AlwaysOutOfDate)
>>
>> # We also want to build everything (unless otherwise asked) so that
>> the database reflects
>> # all targets.
>> env.Default("#")
>>
>> # Do we actually need the lock in a no-exec build Seems like no?
>> compdb_lock = threading.Lock()
>> compdb_items = []
>>
>> # A factory for the action we are going to use instead of the normal
>> C and C++ action. The
>> # action it self is a no-op, which is fine since it isn't going to
>> get executed anyway. We
>> # use the strfunc (which is invoked in a no-exec build) to write the
>> entry for the current
>> # target into the list of compilation database entries.
>> def MakeCompDbStringFunction(command):
>>
>> def NopAction(target, source, env):
>> pass
>>
>> def AddToCompDb(target, source, env):
>> compile_command = env.subst(command, source=source,
>> target=target)
>> with compdb_lock:
>> compdb_items.append({
>> # TODO: Is the abspath here necessary? Or can we just
>> say '.'? I think
>> # that depends on the semantics of compilation
>> database, but abspath
>> # seems safe for sure.
>> "directory" : env.Dir('#').abspath,
>> "command" : compile_command,
>> "file": str(source[0])
>> })
>> return "Calculating compilation DB entry for %s" %
>> str(source[0])
>>
>> return Action(NopAction, strfunction=AddToCompDb)
>>
>> static, shared = SCons.Tool.createObjBuilders(env)
>>
>> # TODO: Make this a little more declarative. How do we get all the
>> # suffixes that SCons has registered?
>> static.add_action(".c", MakeCompDbStringFunction("$CCCOM"))
>> static.add_action(".cpp", MakeCompDbStringFunction("$CXXCOM"))
>> static.add_action(".cc", MakeCompDbStringFunction("$CXXCOM"))
>> shared.add_action(".c", MakeCompDbStringFunction("$SHCCCOM"))
>> shared.add_action(".cpp", MakeCompDbStringFunction("$SHCXXCOM"))
>> shared.add_action(".cc", MakeCompDbStringFunction("$SHCXXCOM"))
>>
>> # Register an atexit handler to serialize all of the compilation
>> database entries we
>> # recorded.
>> def writeCompDb():
>> print("Writing compile_commands.json...")
>> with open(env.File("#compile_commands.json").abspath, "w") as
>> compdb:
>> with compdb_lock:
>> compdb.truncate()
>> json.dump(compdb_items, compdb)
>> import atexit
>> atexit.register(writeCompDb)
>>
>> This works best if you toggle no_exec after executing any configure
>> checks, since otherwise you get the complaint that you cannot execute
>> configured checks in a dry run.
>>
>> I still think that the best way to do this would be if SCons itself was
>> aware of how to emit the compilation database, but short of that the above
>> mechanism does seem to work.
>>
>> Thanks,
>> Andrew
>>
>>
>> On Sat, Aug 3, 2013 at 10:51 AM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> I'm interested in writing some clang tools with LibTooling for a
>>> codebase that uses SCons as its build system. LibTooling is driven by an
>>> artifact known as a CompilationDatabase, which captures state about the
>>> build so that the tool can be consistently invoked:
>>>
>>> http://clang.llvm.org/docs/JSONCompilationDatabase.html
>>>
>>> CMake has support for emitting a CompilationDatabase as a side effect of
>>> a build. I'd like to consider adding similar support to SCons. Before I
>>> start, though, a few questions:
>>>
>>> - Would such support be considered for inclusion in SCons?
>>>
>>> - If so, is anyone already working on this? I'd prefer not to duplicate
>>> the effort.
>>>
>>> - There was recently a thread here about support for considering all
>>> targets out of date. I believe such a thing would be needed to produce a
>>> compilation database as well (probably along with running in --dry-run
>>> mode). The solution in that thread was a custom Decider, but this doesn't
>>> work well if the goal is to support emitting a compilation database without
>>> users needing to customize the Environment. Would support for emitting a
>>> CompilationDatabase justify adding direct support for the "consider all out
>>> of date" feature?
>>>
>>> - What, generally, is the right place to begin? Should
>>> CompilationDatabase support be a tool? A side effect of invoking the Object
>>> and SharedObject builders? How should this be done if the goal is to
>>> support this in such a way that it continues to operate if users customize
>>> their own Object builders?
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150110/18d0b4a6/attachment-0001.html>
More information about the Scons-users
mailing list