[Scons-users] CompilationDatabase support for SCons
Andrew C. Morrow
andrew.c.morrow at gmail.com
Sun Jan 11 11:27:23 EST 2015
Hi Gary -
Thanks for getting back to me.
The utility of a compilation database
<http://clang.llvm.org/docs/JSONCompilationDatabase.html> is that it lets
you capture with fidelity the compile actions that the build system would
take (given the current options, configure checks, etc), and then load that
information into other tools. A good overview of why this is needed can be
found here:
http://eli.thegreenplace.net/2014/05/21/compilation-databases-for-clang-based-tools
For some additional practical examples, here are some examples of things
that can be done once you have a compilation database (usually named
compile_commands.json):
- Run clang-modernize
<http://clang.llvm.org/extra/clang-modernize.html> (convert
C++03 to C++11) and clang-tidy
<http://clang.llvm.org/extra/clang-tidy.html> (AST
aware linter)
- Configure emacs or VI to have a "compile current buffer" facility, by
extracting the last compile command for the current buffer (
https://github.com/randomphrase/ede-compdb)
- Drive your own AST level source introspection or transformation tools
implemented with clang's LibTooling
<http://clang.llvm.org/docs/LibTooling.html>
- Simplify configuration of emacs flycheck or VI you-complete-me tools for
real-time in-editor syntax checking
I'm sure there are many more I'm not aware of.
I acknowledge that this is all fairly clang specific, which is why my hope
had been that it would be possible to generate a compile_commands.json file
in the build system (by writing a SCons Tool), rather than by the build
system (needing to implement it as an intrinsic SCons feature, as I believe
CMake and Ninja do).
Also, I think it is very important that it be possible to produce the
compilation database without needing to actually compile the entire source
tree. For some projects a complete rebuild may be prohibitively expensive,
but this should not preclude using the above tools. My second attempt above
does work, but has the unwanted side effect that all objects are built, so
does not meet this goal.
If you have any thoughts on how to proceed I think this would be a very
useful feature for SCons and I would be happy to help make it a reality.
Thanks,
Andrew
On Sat, Jan 10, 2015 at 5:43 PM, Gary Oberbrunner <garyo at oberbrunner.com>
wrote:
> Hi Andrew; I think it's a cool idea, and some might find it useful. I
> admit I don't understand exactly what this database would be or would be
> useful for. Would it make builds faster, safer, more reproducible? Does
> it help the compiler create better code (like profile-driven
> optimization)? I read your original reference but it didn't make a lot of
> sense to me -- SCons can easily rebuild one object when needed, without
> such a database. Maybe if you explain more about what the database is
> useful for and who would use it, you'd get more traction.
>
> On Sat, Jan 10, 2015 at 12:15 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>> Hi All -
>>
>> I must admit I'm somewhat disappointed that there appears to be no
>> interest in discussing on this list how to get SCons to emit a compilation
>> database. This is an incredibly useful feature that other build systems
>> like CMake and Ninja offer, and enables deep integration between the build
>> system, compiler, and IDE or editor. Many clang tools (clang-modernize,
>> clang-tidy, etc.) either require or work much better with a compilation
>> database.
>>
>> I see no reason why SCons could not or should not provide this
>> functionality as well.
>>
>> Is there any feedback on my above attempts, or thoughts about how this
>> should best be approached?
>>
>> Thanks,
>> Andrew
>>
>>
>>
>> On Tue, Dec 23, 2014 at 9:58 AM, Andrew C. Morrow <
>> andrew.c.morrow at gmail.com> wrote:
>>
>>>
>>> In the interest of completeness, here is the SCons Tool that I first
>>> wrote, which generates the compile_commands.json file by way of actions and
>>> emitters. It basically works, with two caveats:
>>>
>>> - Enabling it before running Configure checks seems not to work if any
>>> of your configure checks fail. You can just enable it with
>>> env.Tool('compiledb') after running all configure checks.
>>>
>>> - It has the side effect that if you build the compilation database,
>>> that all of the .o's are generated. This was unacceptable in the project I
>>> would like to use it for, so this was not a viable solution for me.
>>>
>>> Suggestions for improvements or workarounds for either of the above
>>> issues is greatly appreciated.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> ----- site_scons/site_tools/compiledb.py ----
>>>
>>> import json
>>> import SCons
>>> import itertools
>>>
>>> # TODO: Is there a better way to do this than this global? Right now
>>> this exists so that the
>>> # emitter we add can record all of the things it emits, so that the
>>> scanner for the top level
>>> # compilation database can access the complete list, and also so that
>>> the writer has easy
>>> # access to write all of the files. But it seems clunky. How can the
>>> emitter and the scanner
>>> # communicate more gracefully?
>>> COMPILATION_DB_ENTRIES = []
>>>
>>> def MakeCompilationDbEntryAction(com):
>>>
>>> def CompilationDbEntryAction(target, source, env):
>>> with open(str(target[1]), 'wb') as out:
>>> # TODO: IS using abspath for directory correct here?
>>> # TODO: How would we deal with a builder that did a chdir?
>>> entry = {
>>> "directory": env.Dir('#').abspath,
>>> "command": env.subst(com, source=source, target=target),
>>> "file": str(source[0]),
>>> }
>>> json.dump(entry, out)
>>>
>>> return SCons.Action.Action(CompilationDbEntryAction,
>>> '$COMPILATIONDBENTRYCOMSTR')
>>>
>>> def EmitCompilationDbEntry(target, source, env):
>>> dbentries = [env.File(str(t) + env['COMPILATIONDBSUFFIX']) for t in
>>> target]
>>> COMPILATION_DB_ENTRIES.extend(dbentries)
>>> return target + dbentries, source
>>>
>>> def WriteCompilationDb(target, source, env):
>>> entries = []
>>>
>>> for s in COMPILATION_DB_ENTRIES:
>>> with open(str(s), 'r') as source_file:
>>> entries.append(json.load(source_file))
>>>
>>> with open(str(target[0]), 'w') as target_file:
>>> json.dump(entries, target_file)
>>>
>>> def ScanCompilationDb(node, env, path):
>>> return COMPILATION_DB_ENTRIES
>>>
>>> def CompilationDatabase(env, target=None):
>>>
>>> builder = SCons.Builder.Builder(
>>> action=SCons.Action.Action(WriteCompilationDb,
>>> "$COMPILATIONDBCOMSTR"),
>>> target_scanner=SCons.Scanner.Scanner(ScanCompilationDb),
>>> )
>>>
>>> target = (target or
>>> env.Dir("#").File(env['COMPILATIONDBSUFFIX'][1:]))
>>> return builder(env=env, target=target, source=[])
>>>
>>> def generate(env, **kwargs):
>>>
>>> static_obj, shared_obj = SCons.Tool.createObjBuilders(env)
>>>
>>> # TODO: Is there a way to obtain the configured suffixes for C and
>>> C++
>>> # from the existing obj builders? Seems unfortunate to re-iterate
>>> them.
>>> CSuffixes = ['.c']
>>> CXXSuffixes = ['.cc', '.cxx', '.cpp']
>>>
>>> tuples = itertools.chain(
>>> itertools.product(CSuffixes, [
>>> (static_obj, SCons.Defaults.CAction,
>>> SCons.Defaults.StaticObjectEmitter, '$CCCOM'),
>>> (shared_obj, SCons.Defaults.ShCAction,
>>> SCons.Defaults.SharedObjectEmitter, '$SHCCCOM'),
>>> ]),
>>> itertools.product(CXXSuffixes, [
>>> (static_obj, SCons.Defaults.CXXAction,
>>> SCons.Defaults.StaticObjectEmitter, '$CXXCOM'),
>>> (shared_obj, SCons.Defaults.ShCXXAction,
>>> SCons.Defaults.SharedObjectEmitter, '$SHCXXCOM'),
>>> ]),
>>> )
>>>
>>> env['COMPILATIONDBSUFFIX'] = ".compile_commands.json"
>>> env['COMPILATIONDBCOMSTR'] = "Building compilation database $TARGET"
>>> env['COMPILATIONDBENTRYCOMSTR'] = "Writing compilation database
>>> entry for $SOURCE"
>>>
>>> for tup in tuples:
>>> suffix = tup[0]
>>> builder, base_action, base_emitter, command = tup[1]
>>>
>>> builder.add_action(
>>> suffix, SCons.Action.ListAction(
>>> [
>>> MakeCompilationDbEntryAction(command),
>>> base_action,
>>> ]
>>> ))
>>>
>>> builder.add_emitter(
>>> suffix, SCons.Builder.ListEmitter(
>>> [
>>> EmitCompilationDbEntry,
>>> base_emitter,
>>> ]
>>> ))
>>>
>>> env.AddMethod(CompilationDatabase, "CompilationDatabase")
>>>
>>> def exists(env):
>>> return True
>>>
>>>
>>>
>>> On Sun, Dec 21, 2014 at 5:55 PM, Andrew C. Morrow <
>>> andrew.c.morrow at gmail.com> wrote:
>>>
>>>>
>>>> I spent some time working on this, because having compile_commands.json
>>>> is hugely useful.
>>>>
>>>> I spent a great deal of time trying to use emitters and actions to do
>>>> this, trying to have every C or C++ object not just emit a foo.o for foo.c,
>>>> but also emit a foo.o.compile_commands.json file as well, and then having a
>>>> top level #compile_commands.json depend on all of those
>>>> .o.compile_commands.json files.
>>>>
>>>> This worked well except that it meant that that if you built
>>>> x.o.compile_commands.json, then x.o was built as well. That defeats the
>>>> purpose, since it should be fast to compute the compilation database and
>>>> not require that you actually build each target. I don't entirely
>>>> understand why given a graph like this
>>>>
>>>> node src/foo.c emits src/foo.o
>>>> node src/foo.c emits src/foo.o.compile_commands.json
>>>> node src/foo.c has an action that produces src/foo.o
>>>> node src/foo.c has an action that produces
>>>> src/foo.o.compile_commands.json
>>>> node #compile_commands.json depends on src/foo.o.compile_commands.json
>>>>
>>>> that requesting #compile_commands.json causes src/foo.o to be built,
>>>> since there isn't a dependency path from one to the other. My guess is that
>>>> if any emitted object from a node is depended on, then all of the actions
>>>> for that node must be run, since SCons cannot determine which action leads
>>>> to which emitted item? Is there a way around this? If so then I can pursue
>>>> this again, perhaps.
>>>>
>>>> After getting stuck on that issue, I settled upon the following
>>>> extremely hacked approach, which I'd like to share. I'm not exactly proud
>>>> of this, but it does seem to produce a working compilation database:
>>>>
>>>> if write_compilation_database:
>>>>
>>>> # We want generating the DB to be fast, so we don't execute the
>>>> real build actions. We are
>>>> # going to abuse the facility that handles xCOMSTR to write the db.
>>>> SetOption('no_exec', True)
>>>>
>>>> # We want everything to be out of date so that we build it.
>>>> def AlwaysOutOfDate(x, y, z):
>>>> return True;
>>>> env.Decider(AlwaysOutOfDate)
>>>>
>>>> # We also want to build everything (unless otherwise asked) so that
>>>> the database reflects
>>>> # all targets.
>>>> env.Default("#")
>>>>
>>>> # Do we actually need the lock in a no-exec build Seems like no?
>>>> compdb_lock = threading.Lock()
>>>> compdb_items = []
>>>>
>>>> # A factory for the action we are going to use instead of the
>>>> normal C and C++ action. The
>>>> # action it self is a no-op, which is fine since it isn't going to
>>>> get executed anyway. We
>>>> # use the strfunc (which is invoked in a no-exec build) to write
>>>> the entry for the current
>>>> # target into the list of compilation database entries.
>>>> def MakeCompDbStringFunction(command):
>>>>
>>>> def NopAction(target, source, env):
>>>> pass
>>>>
>>>> def AddToCompDb(target, source, env):
>>>> compile_command = env.subst(command, source=source,
>>>> target=target)
>>>> with compdb_lock:
>>>> compdb_items.append({
>>>> # TODO: Is the abspath here necessary? Or can we
>>>> just say '.'? I think
>>>> # that depends on the semantics of compilation
>>>> database, but abspath
>>>> # seems safe for sure.
>>>> "directory" : env.Dir('#').abspath,
>>>> "command" : compile_command,
>>>> "file": str(source[0])
>>>> })
>>>> return "Calculating compilation DB entry for %s" %
>>>> str(source[0])
>>>>
>>>> return Action(NopAction, strfunction=AddToCompDb)
>>>>
>>>> static, shared = SCons.Tool.createObjBuilders(env)
>>>>
>>>> # TODO: Make this a little more declarative. How do we get all the
>>>> # suffixes that SCons has registered?
>>>> static.add_action(".c", MakeCompDbStringFunction("$CCCOM"))
>>>> static.add_action(".cpp", MakeCompDbStringFunction("$CXXCOM"))
>>>> static.add_action(".cc", MakeCompDbStringFunction("$CXXCOM"))
>>>> shared.add_action(".c", MakeCompDbStringFunction("$SHCCCOM"))
>>>> shared.add_action(".cpp", MakeCompDbStringFunction("$SHCXXCOM"))
>>>> shared.add_action(".cc", MakeCompDbStringFunction("$SHCXXCOM"))
>>>>
>>>> # Register an atexit handler to serialize all of the compilation
>>>> database entries we
>>>> # recorded.
>>>> def writeCompDb():
>>>> print("Writing compile_commands.json...")
>>>> with open(env.File("#compile_commands.json").abspath, "w") as
>>>> compdb:
>>>> with compdb_lock:
>>>> compdb.truncate()
>>>> json.dump(compdb_items, compdb)
>>>> import atexit
>>>> atexit.register(writeCompDb)
>>>>
>>>> This works best if you toggle no_exec after executing any configure
>>>> checks, since otherwise you get the complaint that you cannot execute
>>>> configured checks in a dry run.
>>>>
>>>> I still think that the best way to do this would be if SCons itself was
>>>> aware of how to emit the compilation database, but short of that the above
>>>> mechanism does seem to work.
>>>>
>>>> Thanks,
>>>> Andrew
>>>>
>>>>
>>>> On Sat, Aug 3, 2013 at 10:51 AM, Andrew C. Morrow <
>>>> andrew.c.morrow at gmail.com> wrote:
>>>>
>>>>>
>>>>> I'm interested in writing some clang tools with LibTooling for a
>>>>> codebase that uses SCons as its build system. LibTooling is driven by an
>>>>> artifact known as a CompilationDatabase, which captures state about the
>>>>> build so that the tool can be consistently invoked:
>>>>>
>>>>> http://clang.llvm.org/docs/JSONCompilationDatabase.html
>>>>>
>>>>> CMake has support for emitting a CompilationDatabase as a side effect
>>>>> of a build. I'd like to consider adding similar support to SCons. Before I
>>>>> start, though, a few questions:
>>>>>
>>>>> - Would such support be considered for inclusion in SCons?
>>>>>
>>>>> - If so, is anyone already working on this? I'd prefer not to
>>>>> duplicate the effort.
>>>>>
>>>>> - There was recently a thread here about support for considering all
>>>>> targets out of date. I believe such a thing would be needed to produce a
>>>>> compilation database as well (probably along with running in --dry-run
>>>>> mode). The solution in that thread was a custom Decider, but this doesn't
>>>>> work well if the goal is to support emitting a compilation database without
>>>>> users needing to customize the Environment. Would support for emitting a
>>>>> CompilationDatabase justify adding direct support for the "consider all out
>>>>> of date" feature?
>>>>>
>>>>> - What, generally, is the right place to begin? Should
>>>>> CompilationDatabase support be a tool? A side effect of invoking the Object
>>>>> and SharedObject builders? How should this be done if the goal is to
>>>>> support this in such a way that it continues to operate if users customize
>>>>> their own Object builders?
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
>
> --
> Gary
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150111/202f453c/attachment-0001.html>
More information about the Scons-users
mailing list