[Scons-users] Generated headers in variant build

[Ian Scons]

Hello,

I am trying to make a change to a big legacy SCons-based build and have run
into some issues.

I am not by any means a SCons expert. Our build infrastructure has been
around for a long time and generally it "just works". It is easy to extend
when new source is added and so on, so this is the first time in a long
time that I have had to dig into SCons in any detail.

The new thing that I am trying to do involves some code generation.
Specifically, I am generating source and headers from gRPC "proto" files.
So far I have only attempted this on Windows.

The main aspects of our system in general are:

* The source is rooted at a directory named "Source" containing the
SConstruct.
* It is a variant build into "Builds/<platform>" tree ("Builds" is a
sibling directory of "Source"). Specified via VariantDir in SConstruct.
Duplication is off.
* Hierarchical. The source is organised into "packages", each with own
SConscript, typically building to a shared library/DLL.

The package which involves generated builds fits in like this:

Source/
   ...
   GrpcPackage/
      protos/
         client.proto
      client.hpp
      src/
         client.cpp
         ...

Here, "client.cpp" has a #include of "protos/client.grpc.pb.h", which is a
generated header.

This is what is expected when we build:

Builds/<platform>/
   GrpcPackage/
      protos/
         client.grpc.pb.cc  # generated
         client.grpc.pb.h   #generated
         client.grpc.pb.obj  #built from cc above
         ...
      src/
          client.obj  # note: needs client.grpc.pb.h to build

The target library includes client.obj from the non-generated source and
client.grpc.pb.obj from the generated source.

I have defined a builder to generate the code and this appears to be
working in as far as it runs and generates the expected files when this
package is built.
The issues that I am facing are:

(i) Assuming that client.grpc.pb.h is generated, I found I had to do some
work to allow the compilation of client.cpp to find it. Generally the
compile command includes -I. and -I<packagename> (-IGrpcPackage in this
case). The only way I could get it to work was by including a further
-I<path> where <path> is the full path to Builds/<platform>/GrpcPackage.
This feels a bit clunky and I wondered both whether there was a more
"seamless" way of incorporating generated source and whether the fact that
I have to do this indicates that I am getting something wrong more
generally.

(ii) Even if I do (i), sometimes the build fails because client.grpc.pb.h
cannot be found. It can be seen that the file does exist on the filesystem
at the location specified in (i) so it is clearly a timing/dependency
issue. I am 90% sure I have only seen this issue when doing a parallel (-j
<N>) build. I wonder whether I am defining the dependencies to SCons well
enough for it to be able to identify that client.cpp has a dependency on
client.grpc.pb.obj and sequence the builds correctly. It could be luck that
it seems to work with a non-parallel build. Details of what I have in the
SConstruct/SConscript for this are below.


In order to implement the code generation, I have defined a new builder in
the SConstruct:

    def protoc_emitter(tgt, src, env):
        gen_tgt = []
        for t in tgt:
            ... # append names of generated *.h, *.cc based on t.path
        return gen_tgt, src

    env["BUILDERS"]["GrpcProtoc"] = Builder(
        action=[...], # this is fine, and runs protoc to generate the files
as expected
        emitter=protoc_emitter,
    )

And I use it in the SConscript like this:

    def getProtoObjs(protoFiles):
        protoFiles = env.GrpcProtoc(protoFiles)
        protoSrcFiles = [f for f in protoFiles if f.suffix == ".cc"]
        return env.SharedObject(
            moduleName="GrpcPackage",
            source=protoSrcFiles,
            ...
        )

    protoObjs = getProtoObjs(["protos/client.proto"])
    sources = ["src/client.cpp"]

    env.SharedLibrary(
        "GrpcPackage",
        protoObjs + sources,
        ...
    )

I can't quite get my head around whether this provides enough information
to SCons to track the generated headers. I believe it does because of the
emitter definition in the builder and in any case I have no idea what I
could do differently.

I would be grateful for any suggestions.

Thanks,
I.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20231126/6fa89ee5/attachment.htm>