[Scons-users] Implicit cache bug

[Brian Cody]

I've managed to get myself into a very odd situation. The question on many
minds will be "WHY!?" after reading this, but please be assured that if I
were to reintroduce some business logic, there is some sense to the madness
beneath.

Problem: When the implicit cache is enabled, implicitly cached scan file
results are used even if SCons knows that the scanned file is out of date.
This results in extra targets being specified.

Code:
----------
import SCons
import os

SCons.Defaults.DefaultEnvironment(tools = [])
env = Environment()
output_folder = 'outputs'

def _emit_target(target, source, env):
    return (os.path.join(output_folder, 'result.out'), source)
def _build_target(target, source, env):
    env.Execute(SCons.Script.Touch(target))
def _scan_run_test_list(node, env, path):
    print('Scanning')
    items = node.get_contents().splitlines()
    return items
def _scan_check(node, env):
    print('_scan_check reports ' + str(node.exists() and
node.is_up_to_date()))
    return node.exists() and node.is_up_to_date()
def create_build_rules(env):
    for i in range(1, 10):
        target = os.path.join(output_folder, str(i))
        env.Command(target, [], SCons.Script.Touch(target))

source_of_target_list = 'source_of_target_list.txt'
target_list = os.path.join(output_folder, 'target_list.txt')

env.Append(BUILDERS={'get_things': SCons.Builder.Builder(emitter =
_emit_target,
                                                         action =
_build_target,

source_scanner=SCons.Scanner.Scanner(_scan_run_test_list, recursive=False,
scan_check=_scan_check))})
targets = env.InstallAs(target_list, source_of_target_list) # "Builds" the
target list
results = env.get_things(targets)
create_build_rules(env)

Alias('abc', results)
----------
source_of_target_list.txt
----------
1
2
3
4
----------
Expected behavior when running scons abc:
SCons will check to see if  outputs/results.out is up to date with respect
to outputs/target_list.txt
--> SCons will check to see if outputs/target_list.txt is up to date with
source_of_target_list.txt
--> If not, target_list.txt will be rebuilt
If not, SCons will scan outputs/target_list.txt to get a list of sources
and build the required files.

This works perfectly when the implicit cache is disabled. When it is
enabled, the following sequence will produce the bug:
1. Build scons abc, with the source_of_target_list.txt as noted above.
2. Delete the 1, 2, 3, and 4 files from the outputs directory
3. Remove a line (say, "3") from the source_of_target_list.txt file.
4. Kick off the build again.

Result:
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: rebuilding `outputs\target_list.txt' because
`source_of_target_list.txt' changed
Install file: "source_of_target_list.txt" as "outputs\target_list.txt"
scons: building `outputs\1' because it doesn't exist
Touch("outputs\1")
scons: building `outputs\2' because it doesn't exist
Touch("outputs\2")
scons: building `outputs\3' because it doesn't exist
Touch("outputs\3")
scons: building `outputs\4' because it doesn't exist
Touch("outputs\4")
_scan_check reports True
Scanning
scons: rebuilding `outputs\result.out' because:
           `outputs\3' is no longer a dependency
           `outputs\target_list.txt' changed
_build_target(["outputs\result.out"], ["outputs\target_list.txt"])
Touch(["outputs\result.out"])
+-abc
  +-outputs\result.out
    +-outputs\target_list.txt
    | +-source_of_target_list.txt
    +-outputs\1
    +-outputs\2
    +-outputs\4
scons: done building targets.

SCons built a file that's not even in the dependency tree!! Notice it built
the '3' file AFTER the line stating that the target list was replaced, and
yet BEFORE it scanned the new version of the list.

Exactly the same steps again, but with the implicit cache disabled:
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
_scan_check reports False
scons: rebuilding `outputs\target_list.txt' because
`source_of_target_list.txt' changed
Install file: "source_of_target_list.txt" as "outputs\target_list.txt"
_scan_check reports True
Scanning
scons: building `outputs\1' because it doesn't exist
Touch("outputs\1")
scons: building `outputs\2' because it doesn't exist
Touch("outputs\2")
scons: building `outputs\4' because it doesn't exist
Touch("outputs\4")
scons: rebuilding `outputs\result.out' because:
           `outputs\3' is no longer a dependency
           `outputs\target_list.txt' changed
_build_target(["outputs\result.out"], ["outputs\target_list.txt"])
Touch(["outputs\result.out"])
+-abc
  +-outputs\result.out
    +-outputs\target_list.txt
    | +-source_of_target_list.txt
    +-outputs\1
    +-outputs\2
    +-outputs\4
scons: done building targets.

It behaves exactly as expected! The file to scan is out of date. It's
skipped from scanning, then re-made, and then scanned and used.


This behavior has forced me to disable the implicit cache, which does have
an impact on our build times.



A little bit on the business logic of why this is being done: We have a
situation where it's computationally simple to tell whether or not a target
list has changed, however it's computationally difficult to determine what
the target list is. We use a file to dump the target list into, only
updating it when the list needs to be changed. When the list doesn't need
to be changed, we can just re-use the contents of the file, saving us time
is the great majority of our builds (99%+).

Thanks for looking, all.
-Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161228/2b9e74c4/attachment.html>