[Scons-users] CompilationDatabase support for SCons

[Andrew C. Morrow]

Hi William -

Are you suggesting that each source file be named twice, once within its
Program or Library target, and second time in a list that is fed to the
CompilationDatabase builder? That seems very fragile and unlikely to remain
in sync in a large and evolving codebase.

If not, perhaps I'm not understanding what you are suggesting.

It also seems a little odd to me because the C Object builder already has
all of the appropriate context: source and target names, flags, scanners,
etc., whereas building a completely novel builder will be recreating a lot
of those mechanics. I've struggled with this problem before, where the
apparently correct thing to do is to modify the low level builders (e.g.
adding support for -gsplit-dwarf, where each C compiler invocation results
in two output files), but this seems to be discouraged.

Personally, as someone adding new features to the build system of an
existing project, I want to be able to define what Program, Library, etc.
actually mean in the context of how the user invoked the build, and
conditionally adorn them with supplementary behavior, so that my developers
can just write Program and Library and not need to remember to use
MySpecialMagic[Program|Library] instead. Because they won't remember.

I also don't want them, every time they declare a Program or a Library, to
also have to declare a ProgramDb or a LibraryDb re-iterating the sources.
Because, again, they won't do it, at least not 100% of the time, which
undermines the whole aim of this particular project.

Anyway, thanks for your thoughts on the matter,
Andrew


On Sun, Jan 11, 2015 at 10:03 PM, William Blevins <wblevins001 at gmail.com>
wrote:

> Andrew,
>
> Now that I have read up on Clang modernize, I realize what you are trying
> to do.  I recommend making a builder that generates the JSON compilation
> database files separate from the current c-object builder rather than
> trying to shoehorn in the extra behavior.  This is the standard approach to
> problems like the one you are trying to solve.
>

> You should be able to use the supplied CScanner with your custom build
> action, and the emitter part should be straightforward since I imagine 1
> c-source to 1 JSON compilation database file.
>
> This does imply that compiling and building the compilation database are
> independent activities, so you would need to call your builder on all
> source files which implies modifying your SConscripts.  Might be hard on a
> large code base, but that's what sed/awk is for...
>

> http://www.scons.org/wiki/ToolsForFools
>
> V/R,
> William
>
> On Sun, Jan 11, 2015 at 9:35 PM, Andrew C. Morrow <
> andrew.c.morrow at gmail.com> wrote:
>
>>
>>
>> On Sun, Jan 11, 2015 at 6:45 PM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>
>>> Andrew,
>>>
>>> Maybe I don't fully understand what you are trying to accomplish, but I
>>> find that it's easy to ask the wrong questions.  I still don't think you
>>> have explained what you are trying to accomplish which isn't the same as
>>> what you have tried and decided didn't work.
>>>
>>
>> OK, let me try again. I seem to be doing a poor job explaining what I'm
>> after.
>>
>> I would like to write a new SCons Tool which can emit a
>> compile_commands.json file meeting the specification
>> <http://clang.llvm.org/docs/JSONCompilationDatabase.html>, in which, for
>> each C or C++ source file known to SCons, there is a JSON object containing
>> the name of the source file, the working directory at the time it was
>> compiled, and the full compiler invocation given the current SCons
>> environment. I would like this compile_commands.json file to be able to be
>> named as a SCons target, so that I can say 'scons compiledb' or similar,
>> and have it generated for me. Additionally, I would like the generation of
>> this file to not require the actual compilation of any source files, so
>> that it can be generated quickly.
>>
>> I would like to do this so that I can use clang tools like clang-tidy,
>> clang-modernize, or other LibTooling driven clang tools which require a
>> compilation database, to automate tasks over the codebase.
>>
>> I'm suggesting that SCons should be able to do this because other build
>> system have integrated support for producing such a compilation database:
>>
>> Ninja: http://martine.github.io/ninja/manual.html#_extra_tools
>> CMake: http://www.cmake.org/pipermail/cmake/2013-December/056517.html
>>
>>
>>
>>>
>>> If you just want "What SCons commands SCons would run if SCons users
>>> could run SCons?" Then it might be straight-forward enough to pipe "SCons
>>> --dry-run" into a file or another script.  You don't have to wait for code
>>> to compile and you get the most up-to-date commands.
>>>
>>
>> There are a few problems with this approach:
>>
>> - The format of the compile_commands.json file is not identical to
>> directly executable shell, so this would require parsing the SCons output.
>> That seems fragile, compared to having SCons emit directly, which it seems
>> to me is certainly within its capabilities.
>>
>> - Which target should you select for your dry-run? There is no way to say
>> "all .o" files. Otherwise, the compilation database will not contain a
>> record for all targets that SCons is aware of, only those of the target
>> selected for the dry run.
>>
>> - I can't combine it with a regular build of some target: I need a second
>> dry-run build. If SCons could do this work directly, I could just always
>> leave whatever target generates the compilation database on my SCons
>> invocation, and name other targets as well.
>>
>>
>>>
>>> Well, I'll say that from having worked on several codebases using SCons,
>>>> developers frequently copy and paste compiler invocation lines into the
>>>> shell and run them, and then run them again and again via Ctrl-R. The most
>>>> frequent reason I've heard for doing this is that it takes too long for the
>>>> build system to start up.
>>>
>>>
>>> IMPE: most user's with this compliant don't understand how to call SCons
>>> on specific targets, and the others would rather risk losing hours
>>> debugging an incomplete rebuild than wait the 1 second to read the
>>> SConscripts.
>>>
>>
>> See my other reply, but it is more than 1 second, unfortunately.
>>
>>
>>
>>>
>>> Maybe this was helpful,
>>> William
>>>
>>
>> I do appreciate all the comments. Perhaps, lets assume that my reasons
>> for wanting this are not completely misguided. I guess the information that
>> would be most useful to me would be to know how, based on the Tool from the
>> email on December 23rd, I can solve the issue where building the
>> compilation database causes the .o's to be built as well.
>>
>>
>> Thanks again,
>> Andrew
>>
>>
>>>
>>>
>>> On Sun, Jan 11, 2015 at 6:27 PM, Andrew C. Morrow <
>>> andrew.c.morrow at gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Sun, Jan 11, 2015 at 12:33 PM, Gary Oberbrunner <
>>>> garyo at oberbrunner.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sun, Jan 11, 2015 at 11:27 AM, Andrew C. Morrow <
>>>>> andrew.c.morrow at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> Hi Gary -
>>>>>>
>>>>>> Thanks for getting back to me.
>>>>>>
>>>>>> The utility of a compilation database
>>>>>> <http://clang.llvm.org/docs/JSONCompilationDatabase.html> is that it
>>>>>> lets you capture with fidelity the compile actions that the build system
>>>>>> would take (given the current options, configure checks, etc), and then
>>>>>> load that information into other tools. A good overview of why this is
>>>>>> needed can be found here:
>>>>>> http://eli.thegreenplace.net/2014/05/21/compilation-databases-for-clang-based-tools
>>>>>>
>>>>>> OK, from that I gather that it's a clang-specific database of which
>>>>> flags were used to compile which source files -- presumably from the last
>>>>> build.
>>>>>
>>>>
>>>> Correct. That it is clang specific is why I would prefer to do this by
>>>> building a SCons Tool, rather than needing to build support for generating
>>>> it into SCons itself. FWIW, CMake and Ninja seem to have baked support
>>>> directly into the build system though.
>>>>
>>>>
>>>>>
>>>>>
>>>>>> For some additional practical examples, here are some examples of
>>>>>> things that can be done once you have a compilation database (usually named
>>>>>> compile_commands.json):
>>>>>>
>>>>>> - Run clang-modernize
>>>>>> <http://clang.llvm.org/extra/clang-modernize.html> (convert C++03 to
>>>>>> C++11) and clang-tidy <http://clang.llvm.org/extra/clang-tidy.html> (AST
>>>>>> aware linter)
>>>>>> - Configure emacs or VI to have a "compile current buffer" facility,
>>>>>> by extracting the last compile command for the current buffer (
>>>>>> https://github.com/randomphrase/ede-compdb)
>>>>>>
>>>>>
>>>>> This in particular seems like a bad idea.  If you tell SCons to
>>>>> recompile the current buffer (its .o file) it will check for any modified
>>>>> headers, different compile flags, generated sources that need generating,
>>>>> and so on, before issuing the compile command -- and that only if needed.
>>>>> Re-running the same last command seems at best error-prone, and at worst it
>>>>> could confuse SCons because it doesn't know you updated the .o outside of
>>>>> it (so it doesn't get into the repository if you're using one, and so on).
>>>>>
>>>>
>>>> Well, I'll say that from having worked on several codebases using
>>>> SCons, developers frequently copy and paste compiler invocation lines into
>>>> the shell and run them, and then run them again and again via Ctrl-R. The
>>>> most frequent reason I've heard for doing this is that it takes too long
>>>> for the build system to start up.
>>>>
>>>> In any event, this is just one use case out of dozens, so even if it
>>>> isn't a compelling example, I don't think it alone invalidates the idea of
>>>> producing a compile_commands.json file.
>>>>
>>>>
>>>>>
>>>>> - Drive your own AST level source introspection or transformation
>>>>>> tools implemented with clang's LibTooling
>>>>>> <http://clang.llvm.org/docs/LibTooling.html>
>>>>>> - Simplify configuration of emacs flycheck or VI you-complete-me
>>>>>> tools for real-time in-editor syntax checking
>>>>>>
>>>>>> I'm sure there are many more I'm not aware of.
>>>>>>
>>>>>> I acknowledge that this is all fairly clang specific, which is why my
>>>>>> hope had been that it would be possible to generate a compile_commands.json
>>>>>> file in the build system (by writing a SCons Tool), rather than by the
>>>>>> build system (needing to implement it as an intrinsic SCons feature, as I
>>>>>> believe CMake and Ninja do).
>>>>>>
>>>>>> Also, I think it is very important that it be possible to produce the
>>>>>> compilation database without needing to actually compile the entire source
>>>>>> tree. For some projects a complete rebuild may be prohibitively expensive,
>>>>>> but this should not preclude using the above tools. My second attempt above
>>>>>> does work, but has the unwanted side effect that all objects are built, so
>>>>>> does not meet this goal.
>>>>>>
>>>>>
>>>>> I'm pretty sure this can't work in the general case.  It's easy with
>>>>> SCons to create generated headers and sources, for instance, and even
>>>>> introspect sources (which may be generated) to see what needs to be
>>>>> compiled, and how.  (My own day-job build does exactly this, by building a
>>>>> precompiler (written in C++), precompiling a proprietary high-level
>>>>> language into C++ code which is then compiled.) You can't just do an
>>>>> old-school "make -n" and have it be likely to resemble the real list of
>>>>> commands, except in very simple, straightforward cases.
>>>>>
>>>>
>>>> Whether or not it works in the general case is a separate issue. If it
>>>> doesn't work for you, don't use it. The clang tooling ecosystem is widely
>>>> used in practice, and those tools are driven by compilation databases
>>>> produced by build systems like CMake and Ninja. Obviously, there are build
>>>> environments where it is not a viable technique. But surely that isn't an
>>>> argument against making it work with SCons in the cases where it can and
>>>> enables access to a set of very powerful new development tools?
>>>>
>>>>
>>>>>
>>>>> In a simple C++-only single-pass build, I can see this working to some
>>>>> extent.  I suspect the simplest method is to wrap your compiler with a
>>>>> shell script that emits $* to the database before (or after?) running the
>>>>> comand, and use that shell script as $CC/$CXX etc.  Modifying the existing
>>>>> tools as you've started to do could also work, as could overriding the
>>>>> CC*OMSTR variables perhaps.  However, these will only produce the partial
>>>>> results of what actually got compiled -- if you change flags or compiler
>>>>> versions or paths, you could end up with an out-of-sync database.
>>>>>
>>>>
>>>> Right, and this is why I want to do it as something that SCons does
>>>> understand. The shell script approach is not viable for day to day
>>>> development, and as you point out is unlikely to give complete results and
>>>> has synchronization and correctness issues.
>>>>
>>>> Having SCons produce the file as a consequence of walking the graph
>>>> ensures that the genereated compile_commands.json file does reflect reality
>>>> and won't be out of date.
>>>>
>>>> Anyway, thanks for your thoughts on the matter.
>>>>
>>>> Andrew
>>>>
>>>>
>>>>
>>>>>
>>>>> --
>>>>> Gary
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20150119/40f80c72/attachment-0001.html>