[Scons-users] CompilationDatabase support for SCons

[Andrew C. Morrow]

Hi Bill -

I'm reluctant to engage on the "faster rebuilds" front in this thread,
because while that is something I'm hugely interested in, it isn't the
primary motivation for generating a compilation database. But, I have
experimented with --interactive a bit, and didn't find it to be a
significant improvement.

You might be interested in this thread from earlier this year:
https://groups.google.com/forum/#!topic/mongodb-dev/0bN7eRxUNEs

Thanks,
Andrew


On Sun, Jan 11, 2015 at 6:57 PM, Bill Deegan <bill at baddogconsulting.com>
wrote:

> Andrew,
>
> Have you had the developers use SCons's interactive mode for quick
> re-builds?
>
> -Bill
>
> On Sun, Jan 11, 2015 at 3:27 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>>
>> On Sun, Jan 11, 2015 at 12:33 PM, Gary Oberbrunner <garyo at oberbrunner.com
>> > wrote:
>>
>>>
>>>
>>> On Sun, Jan 11, 2015 at 11:27 AM, Andrew C. Morrow <
>>> andrew.c.morrow at gmail.com> wrote:
>>>
>>>>
>>>> Hi Gary -
>>>>
>>>> Thanks for getting back to me.
>>>>
>>>> The utility of a compilation database
>>>> <http://clang.llvm.org/docs/JSONCompilationDatabase.html> is that it
>>>> lets you capture with fidelity the compile actions that the build system
>>>> would take (given the current options, configure checks, etc), and then
>>>> load that information into other tools. A good overview of why this is
>>>> needed can be found here:
>>>> http://eli.thegreenplace.net/2014/05/21/compilation-databases-for-clang-based-tools
>>>>
>>>> OK, from that I gather that it's a clang-specific database of which
>>> flags were used to compile which source files -- presumably from the last
>>> build.
>>>
>>
>> Correct. That it is clang specific is why I would prefer to do this by
>> building a SCons Tool, rather than needing to build support for generating
>> it into SCons itself. FWIW, CMake and Ninja seem to have baked support
>> directly into the build system though.
>>
>>
>>>
>>>
>>>> For some additional practical examples, here are some examples of
>>>> things that can be done once you have a compilation database (usually named
>>>> compile_commands.json):
>>>>
>>>> - Run clang-modernize
>>>> <http://clang.llvm.org/extra/clang-modernize.html> (convert C++03 to
>>>> C++11) and clang-tidy <http://clang.llvm.org/extra/clang-tidy.html> (AST
>>>> aware linter)
>>>> - Configure emacs or VI to have a "compile current buffer" facility, by
>>>> extracting the last compile command for the current buffer (
>>>> https://github.com/randomphrase/ede-compdb)
>>>>
>>>
>>> This in particular seems like a bad idea.  If you tell SCons to
>>> recompile the current buffer (its .o file) it will check for any modified
>>> headers, different compile flags, generated sources that need generating,
>>> and so on, before issuing the compile command -- and that only if needed.
>>> Re-running the same last command seems at best error-prone, and at worst it
>>> could confuse SCons because it doesn't know you updated the .o outside of
>>> it (so it doesn't get into the repository if you're using one, and so on).
>>>
>>
>> Well, I'll say that from having worked on several codebases using SCons,
>> developers frequently copy and paste compiler invocation lines into the
>> shell and run them, and then run them again and again via Ctrl-R. The most
>> frequent reason I've heard for doing this is that it takes too long for the
>> build system to start up.
>>
>> In any event, this is just one use case out of dozens, so even if it
>> isn't a compelling example, I don't think it alone invalidates the idea of
>> producing a compile_commands.json file.
>>
>>
>>>
>>> - Drive your own AST level source introspection or transformation tools
>>>> implemented with clang's LibTooling
>>>> <http://clang.llvm.org/docs/LibTooling.html>
>>>> - Simplify configuration of emacs flycheck or VI you-complete-me tools
>>>> for real-time in-editor syntax checking
>>>>
>>>> I'm sure there are many more I'm not aware of.
>>>>
>>>> I acknowledge that this is all fairly clang specific, which is why my
>>>> hope had been that it would be possible to generate a compile_commands.json
>>>> file in the build system (by writing a SCons Tool), rather than by the
>>>> build system (needing to implement it as an intrinsic SCons feature, as I
>>>> believe CMake and Ninja do).
>>>>
>>>> Also, I think it is very important that it be possible to produce the
>>>> compilation database without needing to actually compile the entire source
>>>> tree. For some projects a complete rebuild may be prohibitively expensive,
>>>> but this should not preclude using the above tools. My second attempt above
>>>> does work, but has the unwanted side effect that all objects are built, so
>>>> does not meet this goal.
>>>>
>>>
>>> I'm pretty sure this can't work in the general case.  It's easy with
>>> SCons to create generated headers and sources, for instance, and even
>>> introspect sources (which may be generated) to see what needs to be
>>> compiled, and how.  (My own day-job build does exactly this, by building a
>>> precompiler (written in C++), precompiling a proprietary high-level
>>> language into C++ code which is then compiled.) You can't just do an
>>> old-school "make -n" and have it be likely to resemble the real list of
>>> commands, except in very simple, straightforward cases.
>>>
>>
>> Whether or not it works in the general case is a separate issue. If it
>> doesn't work for you, don't use it. The clang tooling ecosystem is widely
>> used in practice, and those tools are driven by compilation databases
>> produced by build systems like CMake and Ninja. Obviously, there are build
>> environments where it is not a viable technique. But surely that isn't an
>> argument against making it work with SCons in the cases where it can and
>> enables access to a set of very powerful new development tools?
>>
>>
>>>
>>> In a simple C++-only single-pass build, I can see this working to some
>>> extent.  I suspect the simplest method is to wrap your compiler with a
>>> shell script that emits $* to the database before (or after?) running the
>>> comand, and use that shell script as $CC/$CXX etc.  Modifying the existing
>>> tools as you've started to do could also work, as could overriding the
>>> CC*OMSTR variables perhaps.  However, these will only produce the partial
>>> results of what actually got compiled -- if you change flags or compiler
>>> versions or paths, you could end up with an out-of-sync database.
>>>
>>
>> Right, and this is why I want to do it as something that SCons does
>> understand. The shell script approach is not viable for day to day
>> development, and as you point out is unlikely to give complete results and
>> has synchronization and correctness issues.
>>
>> Having SCons produce the file as a consequence of walking the graph
>> ensures that the genereated compile_commands.json file does reflect reality
>> and won't be out of date.
>>
>> Anyway, thanks for your thoughts on the matter.
>>
>> Andrew
>>
>>
>>
>>>
>>> --
>>> Gary
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150111/dd41cbeb/attachment.html>