[Scons-users] Generated headers in variant build

Ian Scons sconsian at gmail.com
Tue Nov 28 04:55:27 EST 2023 

Thanks Bill.

> Can you try moving to the latest released version of SCons?

I will be able to try this at least for my local builds, for now.

> VariantDir(<build root dir>, "#Source", duplicate=False)

Could I confirm that what you are suggesting is to move the SConstruct up a
level so that both Source and Builds are children of the directory that
contains SConstruct?

    SConstruct
    Source/
        ...
    Builds/
        ...

I think this could be done in principle but I don't know what the
consequences would be in terms of what would need changing in the
SConstruct. I'll look into it.

> When you call your builder what path are you using for the source and the
target?

My apologies, but I'm not really sure how to answer this. When the builder
is "called" in the relevant SConscript, it is just given a list of source
(proto) files, relative to the current SConscript directory.
Essentially, it comes down to this:

    env.GrpcProtoc(["protos/client.proto"])

When the builder is invoked during the build, the action references
${TARGET.dir} and ${SOURCE.dir} as follows:

    action=[
        f'"{PROTOC}"'
        + " --cpp_out=${TARGET.dir}  -I"
        + PROTO_INCLUDE
        + " $SOURCE -I${SOURCE.dir}",
        f'"{PROTOC}"'
        + " --grpc_out=${TARGET.dir} --plugin=protoc-gen-grpc="
        + f'"{PROTOC_GRPC}"'
        + " $SOURCE  -I"
        + PROTO_INCLUDE
        + " -I${SOURCE.dir}"
    ],

I'm not sure if that answers your question though.

Anyway, between what you and Mats have suggested I now have a list of
things I need to try/look into.

Thanks for your help.

I.




On Mon, 27 Nov 2023 at 17:53, Bill Deegan <bill at baddogconsulting.com> wrote:

> Can you try moving to the latest released version of SCons?
> Any issues you might have won't be fixed on an older version but only in
> the next release.
>
> So your VariandDir() usage is slightly non-ideal.
> We don't advise having the source dir for a variant dir including the
> build dir.
> If all your sources are under Source then this statement would be better.
>
> VariantDir(<build root dir>, "#Source", duplicate=False)
>
> When you call your builder what path are you using for the source and the
> target?
> Are they both specified to be in the variant dir?
>
> -Bill
>
> On Mon, Nov 27, 2023 at 2:42 AM Ian Scons <sconsian at gmail.com> wrote:
>
>> Hi Bill,
>>
>> > Do you know how your variant dirs are specified? Are you using
>> VariantDir(), or specifying variant_dir when calling SConscript()?
>>
>> We are using VariantDir. VariantDir(<build root dir>, "#",
>> duplicate=False)
>>
>> I did wonder whether the duplicate flag was relevant, because then all
>> the source, including generated, is effectively in one place in the build
>> tree. But I did briefly try with True and the build failed in other
>> packages before it even got to mine. I don't want to do anything that would
>> destabilise the build for anyone else.
>>
>> I wonder whether I should try switching to using variant_dir when calling
>> the SConscripts, and try selectively turning on duplication for my package.
>> I don't know if that makes sense.
>>
>> > Which version of SCons and Python are you using?
>>
>> SCons 4.3.0
>> CPython 3.10.9
>>
>> Thanks,
>> I.
>>
>>
>> On Sun, 26 Nov 2023 at 18:27, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>> Ian,
>>>
>>> Do you know how your variant dirs are specified? Are you using
>>> VariantDir(), or specifying variant_dir when calling SConscript()?
>>> Which version of SCons and Python are you using?
>>>
>>> -Bill
>>>
>>> On Sun, Nov 26, 2023 at 7:15 AM Mats Wichmann <mats at wichmann.us> wrote:
>>>
>>>> On 11/26/23 06:12, Ian Scons wrote:
>>>>
>>>> That's a lot of detail - thanks, though it will take a while to pick
>>>> through it.
>>>>
>>>> > Here, "client.cpp" has a #include of "protos/client.grpc.pb.h", which
>>>> is
>>>> > a generated header.
>>>>
>>>> ...
>>>>
>>>> > In order to implement the code generation, I have defined a new
>>>> builder
>>>> > in the SConstruct:
>>>> >
>>>> >      def protoc_emitter(tgt, src, env):
>>>> >          gen_tgt = []
>>>> >          for t in tgt:
>>>> >              ... # append names of generated *.h, *.cc based on t.path
>>>> >          return gen_tgt, src
>>>> >
>>>> >      env["BUILDERS"]["GrpcProtoc"] = Builder(
>>>> >          action=[...], # this is fine, and runs protoc to generate
>>>> the
>>>> > files as expected
>>>> >          emitter=protoc_emitter,
>>>> >      )
>>>>
>>>> The first reaction is that the plumbing may work more easily if the
>>>> builder is defined to "know" the source and target file suffixes it is
>>>> responsible for.  That would mean adding the suffix and src_suffix
>>>> kwargs to the builder call. I might use suffix=".grpc.pb.h" and
>>>> "src_suffix=.proto"  (although it gets trickier if you generate two
>>>> outputs from protoc, as it seems you're doing - a .h and a .cc).
>>>>
>>>>  > Even if I do (i), sometimes the build fails because client.grpc.pb.h
>>>> cannot be found. It can be seen that the file does exist on the
>>>> filesystem
>>>>
>>>> One issue in dealing with generated files is that SCons relocates the
>>>> things it needs to based on the directory of the SConscript, but when
>>>> indirect execution is involved, like an emitter, it doesn't always
>>>> figure out it needs to - and you end up with the human saying "but the
>>>> file is there" and SCons saying "I don't see anything *where I'm
>>>> looking*".  In your emitter, you might try generating the added targets
>>>> as File nodes, rather than as strings, and see if that helps - nodes
>>>> have path information in them so they don't get lost track of.
>>>>
>>>>
>>>> By the way, there's a protoc builder in the scons-contrib repository,
>>>> you might look at what it does differently/same as yours (my memory is
>>>> it was fairly convoluted)
>>>>
>>>> https://github.com/scons/scons-contrib
>>>>
>>>> https://github.com/SCons/scons-contrib/tree/master/sconscontrib/SCons/Tool/Protoc
>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20231128/5f2abebb/attachment.htm>

More information about the Scons-users mailing list