[Scons-users] AddPostAction memoization problem

Wed May 29 04:18:43 EDT 2024

Bill,

Our real-life build defines a Tool: EnsureMaxSize(target, max_size).
We use it to enforce a size limit policy on a build output. The
Builder for the target fails if the produced file exceeds the limit.
The Tool calls AddPostAction on the target with an ActionFunction that
renames the target (target + ".toobig") and raises a StopError if the
value returned by target.get_size() exceeds the max_size argument.

MAX_APP_SIZE = 8 * 1024 * 1024

env = Environment()
env.Tool("ensure_max_size")
app = env.Program(target="myapp", source="main.c")
env.EnsureMaxSize(app, MAX_APP_SIZE)

We don't use a Builder for a Tool like this, because we don't want to
introduce a new target / Node / file. For example, we could write:
env.EnsureMaxSize(target=app+".sizeok", source=app), but we don't want
any further nodes in the dependency graph that depend on "app" to be
built using the too-large file, and we don't want to add app.sizeok to
the dependency graph in place of app.

By not introducing a new Node, we may be making some unsafe
assumptions about the Actions within the Builder.
1. Our use of PostAction assumes the primary Actions for building a
target have finished.
2. The execution order of PostActions can be significant if you were
to combine Tools with operations like env.Compress(app) and
env.EnsureMaxSize(app).

Generally though, our Builders are simple, and the code we use to
build a thing is close to the tools we invoke to post-process the
thing. We're unlikely to write a Tool like env.Compress that would
modify the content of a target in a PostAction. For us, a tool like
that would be written as a Builder and the compressed file would be
part of the dependency graph.

As to the correct behavior, every Action in a Builder's Action list
has the opportunity to modify a target and or a side effect. I think
we should call a function similar to invalidate_node_memos after each
Action executes. The set of invalidated nodes should be the union of
the Action's target list and each target's side_effect list. I'm not
sure how much performance would be lost by clearing the cache for
targets and side effects.

I don't know that there's a practical way for an Action to express the
set of targets or side effects that it has invalidated.

I also don't think it's correct to modify a source Node for a Builder
inside an Action. A source should remain unmodified as the Builder
executes, as its content was evaluated to determine if the Builder
needed to execute in the first place.

Mike

On Tue, May 28, 2024 at 11:11 PM Bill Deegan <bill at baddogconsulting.com> wrote:
>
> Mike,
>
> Indeed it was not what I would have expected until I looked at the code.
> That said, what's the solution, clear the memoization after each action? Other?
>
> As an aside, what does your actual post action do?
>
> -Bill
>
> On Tue, May 28, 2024 at 7:02 PM Mike Haboustak <haboustak at gmail.com> wrote:
>>
>> Bill,
>>
>> I think it's a fair clarification to say the Builder hasn't completed until the entire set of PreAction, Action, and PostAction lists have been executed. However, I'm still of the opinion that it's a problem when Actions that call a memoized function on a target or a side effect of the Builder get the wrong result.
>>
>> We could note in the documentation for memoized Node functions to indicate they can't be called during an Action, but that note would need to propagate to any function that has a memoized function in its call graph.
>>
>>
>> Mike
>>
>>
>>
>> On Mon, May 27, 2024 at 7:03 PM Bill Deegan <bill at baddogconsulting.com> wrote:
>>>
>>> Mike,
>>>
>>> AddPostAction() adds to the list of Actions for a builder, so when your action is called the builder in question has not completed.
>>>
>>> AddPreAction() adds to a pre list
>>> AddPostAction() adds to a post list
>>> Then when the target is built, the executor will run the actions in this order pre-list + normal-action-list + post list.
>>> It looks like the memoization is cleared when the builder is called.
>>> But certainly not called between each of the Action()'s in the list of actions for a builder.
>>>
>>> -Bill
>>>
>>>
>>>
>>> On Thu, May 23, 2024 at 6:59 PM Mike Haboustak <haboustak at gmail.com> wrote:
>>>>
>>>> I encountered a problem recently with AddPostAction and the Node.get_size() function. The memoization features of the Node.exists() function does not appear to be reset after the Builder completes and before the PostAction runs. As a result, some of the cached values are not correct.
>>>>
>>>> Here's a simplified SConstruct that reproduces the issue when run with SCons 4.7.0 with GCC on Linux. The "PostAction" size of the test.o object file is based on the state of the file prior to the builder running.
>>>>
>>>> ---- SConstruct
>>>> env = Environment()
>>>> def print_size(target, source, env):
>>>>     print("PostAction:", target[0].getsize())
>>>>
>>>> test = env.Object("test.c")
>>>> env.AddPostAction(test, Action(print_size))
>>>> ----
>>>>
>>>> Test Output
>>>> ----
>>>> $ touch test.c
>>>> $ scons
>>>> scons: Reading SConscript files ...
>>>> scons: done reading SConscript files.
>>>> scons: Building targets ...
>>>> gcc -o test.o -c test.c
>>>> print_size(["test.o"], ["test.c"])
>>>> PostAction: None
>>>> scons: done building targets.
>>>>
>>>> $ echo -e "\n" >>test.c
>>>> $ scons
>>>> scons: Reading SConscript files ...
>>>> scons: done reading SConscript files.
>>>> scons: Building targets ...
>>>> gcc -o test.o -c test.c
>>>> print_size(["test.o"], ["test.c"])
>>>> PostAction: 800
>>>> scons: done building targets.
>>>> ----
>>>>
>>>> Thanks,
>>>> Mike
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users