[Scons-users] Custom nodes with a fixed ID

[Dirk Bächle]

Hi Florian,

attached you can find a small text, describing how the signature 
subsystem works...roughly. I wrote it while debugging SCons and it might 
give you a better impression about which call trees are involved.

Hope this helps you further somehow.

Best regards,

Dirk

-------------- next part --------------

== signature subsystem ==

Run 1:

For FileNodes we get a first csig for our ninfo struct in

  next_task
    make_ready
      make_ready_current
        visited

first we call 

  ninfo = self.get_ninfo()

which adds an empty struct with version_id s only.
Then, in

  csig = self.get_max_drift_csig()

max_drift_csig calls get_stored_info, which initializes the 
self.dir.sconsign() entry for this node. This means (on the first invocation) that we land in

  SConsign.DB.__init__

where the raw entries for all (!) folders are read from the dblite database.
Then we try to read the entry for the current directory (of the node in question), and if we don't succeed, a new SConsignEntry with a fresh binfo/ninfo pair is added:

    import SCons.SConsign
    sconsign_entry = SCons.SConsign.SConsignEntry()
    sconsign_entry.binfo = self.new_binfo()
    sconsign_entry.ninfo = self.new_ninfo()

The entry is then stored in _memo cache:

    self._memo['get_stored_info'] = sconsign_entry

This is a very very (!!!) important step, because the rest of the signature
system relies on the fact that all sig infos are collected together alog the way...and not recomputed every time we call get_stored_info(). This instance of
sconsign_entry always needs to get cached in one way or the other for each node.

Back in visited() again, the ninfo.csig is not set because we don't have any old siginfos. Then the current timestamp and file size is recorded with

        ninfo.timestamp = self.get_timestamp()
        ninfo.size      = self.get_size()

Finally, we store the new info via self.store_info() which calls

  Node.FS.store_info()
    SConsign.Base.store_info()
      
and merges the binfo of the sconsign_entry so far (see above), with the current
binfo for this very node. This is to accommodate chained builds (see also chained_builds.py for an example):

      def store_info(self, filename, node):
        entry = node.get_stored_info()
        entry.binfo.merge(node.get_binfo())
        self.to_be_merged[filename] = node
        self.dirty = True

Additionally, the dirty flag is set, to signal that new data has to get written into the Database.??


This whole action chain for visited() is called again after a Node was
built (or not, as a no-op action) in Taskmaster.executed_with_callbacks().
Here again, only the timestamp and size are reset to their current values...


All this is true for source files and implicit dependencies, like SConstruct, main.cpp and /usr/bin/g++ in our simple example.

For targets, like main.o, the get_stored_info() method is called from:

  Taskmaster.make_ready_current()
    Node.FS.is_up_to_date()
      Node.Node.changed()

Then, in changed() we compare the list of old binfos (then) with the current
list of children for this node (children). In a first run their length differs,
so then gets extended to the same length as children by empty entries.
Now, for each (children, then) pair we call:

  
  Taskmaster.make_ready_current()
    Node.FS.is_up_to_date()
      Node.Node.changed()
        Node.FS.decide_source()
          Environment.default_decide_source()
            Environment.Base._changed_content()
              Node.FS.changed_content()
                Node.FS.get_csig()

where the content signature gets computed and stored in ninfo.csig.
The same hold for targets like in

  Taskmaster.make_ready_current()
    Node.FS.is_up_to_date()
      Node.Node.changed()
        Node.FS.decide_target()
          Environment.default_decide_target()
            Environment.Base._changed_content()
              Node.FS.changed_content()
                Node.FS.get_csig()

TODO: Check where Node.Node.changed gets overwritten with either decide_source
or decide_target!


After a node is actually built, we call:

  Taskmaster.executed_with_callbacks()
    Node.Node.built()
      Node.NodeInfoBase.update()
        FS.File.get_csig()

where now a new content signature gets computed via get_max_drift_csig(),
but it's None the first time anyways,
and then, back in get_csig() stored in ninfo.csig.

Now, when executed_with_callbacks() calls visited() after the built(), the ninfo
already properly carries the ninfo.csig. Okey-dokey.
Since get_max_drift_csig doen't return any useful value, the ninfo.csig doesn't get overwritten...important!

In 

  Taskmaster.executed_with_callbacks()
    Node.Node.built()
      Node.NodeInfoBase.update()

we now have the final command

    def update(self, node):
        try:
            field_list = self.field_list
        except AttributeError:
            return
        for f in field_list:
            try:
                delattr(self, f)
            except AttributeError:
                pass
            try:
                func = getattr(node, 'get_' + f)
            except AttributeError:
                pass
            else:
                setattr(self, f, func())

(setattr) which sets the result of get_csig() (see above) to the data field
"csig" for the current NodeInfoBase entry.

Now, on the next call to visited() from executed_with_callbacks(), the ninfo field of the current node should already contain the csig value.

Once, all nodes were built we get to

  Main._main()
    Main._build_targets()
      Job.run()
        Main.jobs_postfunc()
          SConsign.write()

where we loop over the SConsign entries for all dirs (global var sigfiles)
and call SConsign.DB.write() for each. In this write() we have the important
merge via SConsign.Base.merge()
where the content signature from the nodes are taken into account.

Call stack here is:

  ... see above
    SConsign.write()
      SConsig.DB.write()
        SConsign.Base.merge()
          Node.NodeInfoBase.merge()