[Scons-users] CompilationDatabase support for SCons
Gary Oberbrunner
garyo at oberbrunner.com
Sat Jan 10 17:43:49 EST 2015
Hi Andrew; I think it's a cool idea, and some might find it useful. I admit
I don't understand exactly what this database would be or would be useful
for. Would it make builds faster, safer, more reproducible? Does it help
the compiler create better code (like profile-driven optimization)? I read
your original reference but it didn't make a lot of sense to me -- SCons
can easily rebuild one object when needed, without such a database. Maybe
if you explain more about what the database is useful for and who would use
it, you'd get more traction.
On Sat, Jan 10, 2015 at 12:15 PM, Andrew C. Morrow <
andrew.c.morrow at gmail.com> wrote:
>
> Hi All -
>
> I must admit I'm somewhat disappointed that there appears to be no
> interest in discussing on this list how to get SCons to emit a compilation
> database. This is an incredibly useful feature that other build systems
> like CMake and Ninja offer, and enables deep integration between the build
> system, compiler, and IDE or editor. Many clang tools (clang-modernize,
> clang-tidy, etc.) either require or work much better with a compilation
> database.
>
> I see no reason why SCons could not or should not provide this
> functionality as well.
>
> Is there any feedback on my above attempts, or thoughts about how this
> should best be approached?
>
> Thanks,
> Andrew
>
>
>
> On Tue, Dec 23, 2014 at 9:58 AM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>> In the interest of completeness, here is the SCons Tool that I first
>> wrote, which generates the compile_commands.json file by way of actions and
>> emitters. It basically works, with two caveats:
>>
>> - Enabling it before running Configure checks seems not to work if any of
>> your configure checks fail. You can just enable it with
>> env.Tool('compiledb') after running all configure checks.
>>
>> - It has the side effect that if you build the compilation database, that
>> all of the .o's are generated. This was unacceptable in the project I would
>> like to use it for, so this was not a viable solution for me.
>>
>> Suggestions for improvements or workarounds for either of the above
>> issues is greatly appreciated.
>>
>> Thanks,
>> Andrew
>>
>>
>> ----- site_scons/site_tools/compiledb.py ----
>>
>> import json
>> import SCons
>> import itertools
>>
>> # TODO: Is there a better way to do this than this global? Right now this
>> exists so that the
>> # emitter we add can record all of the things it emits, so that the
>> scanner for the top level
>> # compilation database can access the complete list, and also so that the
>> writer has easy
>> # access to write all of the files. But it seems clunky. How can the
>> emitter and the scanner
>> # communicate more gracefully?
>> COMPILATION_DB_ENTRIES = []
>>
>> def MakeCompilationDbEntryAction(com):
>>
>> def CompilationDbEntryAction(target, source, env):
>> with open(str(target[1]), 'wb') as out:
>> # TODO: IS using abspath for directory correct here?
>> # TODO: How would we deal with a builder that did a chdir?
>> entry = {
>> "directory": env.Dir('#').abspath,
>> "command": env.subst(com, source=source, target=target),
>> "file": str(source[0]),
>> }
>> json.dump(entry, out)
>>
>> return SCons.Action.Action(CompilationDbEntryAction,
>> '$COMPILATIONDBENTRYCOMSTR')
>>
>> def EmitCompilationDbEntry(target, source, env):
>> dbentries = [env.File(str(t) + env['COMPILATIONDBSUFFIX']) for t in
>> target]
>> COMPILATION_DB_ENTRIES.extend(dbentries)
>> return target + dbentries, source
>>
>> def WriteCompilationDb(target, source, env):
>> entries = []
>>
>> for s in COMPILATION_DB_ENTRIES:
>> with open(str(s), 'r') as source_file:
>> entries.append(json.load(source_file))
>>
>> with open(str(target[0]), 'w') as target_file:
>> json.dump(entries, target_file)
>>
>> def ScanCompilationDb(node, env, path):
>> return COMPILATION_DB_ENTRIES
>>
>> def CompilationDatabase(env, target=None):
>>
>> builder = SCons.Builder.Builder(
>> action=SCons.Action.Action(WriteCompilationDb,
>> "$COMPILATIONDBCOMSTR"),
>> target_scanner=SCons.Scanner.Scanner(ScanCompilationDb),
>> )
>>
>> target = (target or env.Dir("#").File(env['COMPILATIONDBSUFFIX'][1:]))
>> return builder(env=env, target=target, source=[])
>>
>> def generate(env, **kwargs):
>>
>> static_obj, shared_obj = SCons.Tool.createObjBuilders(env)
>>
>> # TODO: Is there a way to obtain the configured suffixes for C and C++
>> # from the existing obj builders? Seems unfortunate to re-iterate
>> them.
>> CSuffixes = ['.c']
>> CXXSuffixes = ['.cc', '.cxx', '.cpp']
>>
>> tuples = itertools.chain(
>> itertools.product(CSuffixes, [
>> (static_obj, SCons.Defaults.CAction,
>> SCons.Defaults.StaticObjectEmitter, '$CCCOM'),
>> (shared_obj, SCons.Defaults.ShCAction,
>> SCons.Defaults.SharedObjectEmitter, '$SHCCCOM'),
>> ]),
>> itertools.product(CXXSuffixes, [
>> (static_obj, SCons.Defaults.CXXAction,
>> SCons.Defaults.StaticObjectEmitter, '$CXXCOM'),
>> (shared_obj, SCons.Defaults.ShCXXAction,
>> SCons.Defaults.SharedObjectEmitter, '$SHCXXCOM'),
>> ]),
>> )
>>
>> env['COMPILATIONDBSUFFIX'] = ".compile_commands.json"
>> env['COMPILATIONDBCOMSTR'] = "Building compilation database $TARGET"
>> env['COMPILATIONDBENTRYCOMSTR'] = "Writing compilation database entry
>> for $SOURCE"
>>
>> for tup in tuples:
>> suffix = tup[0]
>> builder, base_action, base_emitter, command = tup[1]
>>
>> builder.add_action(
>> suffix, SCons.Action.ListAction(
>> [
>> MakeCompilationDbEntryAction(command),
>> base_action,
>> ]
>> ))
>>
>> builder.add_emitter(
>> suffix, SCons.Builder.ListEmitter(
>> [
>> EmitCompilationDbEntry,
>> base_emitter,
>> ]
>> ))
>>
>> env.AddMethod(CompilationDatabase, "CompilationDatabase")
>>
>> def exists(env):
>> return True
>>
>>
>>
>> On Sun, Dec 21, 2014 at 5:55 PM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> I spent some time working on this, because having compile_commands.json
>>> is hugely useful.
>>>
>>> I spent a great deal of time trying to use emitters and actions to do
>>> this, trying to have every C or C++ object not just emit a foo.o for foo.c,
>>> but also emit a foo.o.compile_commands.json file as well, and then having a
>>> top level #compile_commands.json depend on all of those
>>> .o.compile_commands.json files.
>>>
>>> This worked well except that it meant that that if you built
>>> x.o.compile_commands.json, then x.o was built as well. That defeats the
>>> purpose, since it should be fast to compute the compilation database and
>>> not require that you actually build each target. I don't entirely
>>> understand why given a graph like this
>>>
>>> node src/foo.c emits src/foo.o
>>> node src/foo.c emits src/foo.o.compile_commands.json
>>> node src/foo.c has an action that produces src/foo.o
>>> node src/foo.c has an action that produces
>>> src/foo.o.compile_commands.json
>>> node #compile_commands.json depends on src/foo.o.compile_commands.json
>>>
>>> that requesting #compile_commands.json causes src/foo.o to be built,
>>> since there isn't a dependency path from one to the other. My guess is that
>>> if any emitted object from a node is depended on, then all of the actions
>>> for that node must be run, since SCons cannot determine which action leads
>>> to which emitted item? Is there a way around this? If so then I can pursue
>>> this again, perhaps.
>>>
>>> After getting stuck on that issue, I settled upon the following
>>> extremely hacked approach, which I'd like to share. I'm not exactly proud
>>> of this, but it does seem to produce a working compilation database:
>>>
>>> if write_compilation_database:
>>>
>>> # We want generating the DB to be fast, so we don't execute the real
>>> build actions. We are
>>> # going to abuse the facility that handles xCOMSTR to write the db.
>>> SetOption('no_exec', True)
>>>
>>> # We want everything to be out of date so that we build it.
>>> def AlwaysOutOfDate(x, y, z):
>>> return True;
>>> env.Decider(AlwaysOutOfDate)
>>>
>>> # We also want to build everything (unless otherwise asked) so that
>>> the database reflects
>>> # all targets.
>>> env.Default("#")
>>>
>>> # Do we actually need the lock in a no-exec build Seems like no?
>>> compdb_lock = threading.Lock()
>>> compdb_items = []
>>>
>>> # A factory for the action we are going to use instead of the normal
>>> C and C++ action. The
>>> # action it self is a no-op, which is fine since it isn't going to
>>> get executed anyway. We
>>> # use the strfunc (which is invoked in a no-exec build) to write the
>>> entry for the current
>>> # target into the list of compilation database entries.
>>> def MakeCompDbStringFunction(command):
>>>
>>> def NopAction(target, source, env):
>>> pass
>>>
>>> def AddToCompDb(target, source, env):
>>> compile_command = env.subst(command, source=source,
>>> target=target)
>>> with compdb_lock:
>>> compdb_items.append({
>>> # TODO: Is the abspath here necessary? Or can we
>>> just say '.'? I think
>>> # that depends on the semantics of compilation
>>> database, but abspath
>>> # seems safe for sure.
>>> "directory" : env.Dir('#').abspath,
>>> "command" : compile_command,
>>> "file": str(source[0])
>>> })
>>> return "Calculating compilation DB entry for %s" %
>>> str(source[0])
>>>
>>> return Action(NopAction, strfunction=AddToCompDb)
>>>
>>> static, shared = SCons.Tool.createObjBuilders(env)
>>>
>>> # TODO: Make this a little more declarative. How do we get all the
>>> # suffixes that SCons has registered?
>>> static.add_action(".c", MakeCompDbStringFunction("$CCCOM"))
>>> static.add_action(".cpp", MakeCompDbStringFunction("$CXXCOM"))
>>> static.add_action(".cc", MakeCompDbStringFunction("$CXXCOM"))
>>> shared.add_action(".c", MakeCompDbStringFunction("$SHCCCOM"))
>>> shared.add_action(".cpp", MakeCompDbStringFunction("$SHCXXCOM"))
>>> shared.add_action(".cc", MakeCompDbStringFunction("$SHCXXCOM"))
>>>
>>> # Register an atexit handler to serialize all of the compilation
>>> database entries we
>>> # recorded.
>>> def writeCompDb():
>>> print("Writing compile_commands.json...")
>>> with open(env.File("#compile_commands.json").abspath, "w") as
>>> compdb:
>>> with compdb_lock:
>>> compdb.truncate()
>>> json.dump(compdb_items, compdb)
>>> import atexit
>>> atexit.register(writeCompDb)
>>>
>>> This works best if you toggle no_exec after executing any configure
>>> checks, since otherwise you get the complaint that you cannot execute
>>> configured checks in a dry run.
>>>
>>> I still think that the best way to do this would be if SCons itself was
>>> aware of how to emit the compilation database, but short of that the above
>>> mechanism does seem to work.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> On Sat, Aug 3, 2013 at 10:51 AM, Andrew C. Morrow <
>>> andrew.c.morrow at gmail.com> wrote:
>>>
>>>>
>>>> I'm interested in writing some clang tools with LibTooling for a
>>>> codebase that uses SCons as its build system. LibTooling is driven by an
>>>> artifact known as a CompilationDatabase, which captures state about the
>>>> build so that the tool can be consistently invoked:
>>>>
>>>> http://clang.llvm.org/docs/JSONCompilationDatabase.html
>>>>
>>>> CMake has support for emitting a CompilationDatabase as a side effect
>>>> of a build. I'd like to consider adding similar support to SCons. Before I
>>>> start, though, a few questions:
>>>>
>>>> - Would such support be considered for inclusion in SCons?
>>>>
>>>> - If so, is anyone already working on this? I'd prefer not to duplicate
>>>> the effort.
>>>>
>>>> - There was recently a thread here about support for considering all
>>>> targets out of date. I believe such a thing would be needed to produce a
>>>> compilation database as well (probably along with running in --dry-run
>>>> mode). The solution in that thread was a custom Decider, but this doesn't
>>>> work well if the goal is to support emitting a compilation database without
>>>> users needing to customize the Environment. Would support for emitting a
>>>> CompilationDatabase justify adding direct support for the "consider all out
>>>> of date" feature?
>>>>
>>>> - What, generally, is the right place to begin? Should
>>>> CompilationDatabase support be a tool? A side effect of invoking the Object
>>>> and SharedObject builders? How should this be done if the goal is to
>>>> support this in such a way that it continues to operate if users customize
>>>> their own Object builders?
>>>>
>>>> Thanks,
>>>> Andrew
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
--
Gary
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150110/3b97eed3/attachment-0001.html>
More information about the Scons-users
mailing list