[Scons-users] Reducing toposort time for Java builds

[Greg Ward]

On 04 July 2012, I said:

> So I've been thinking, why not just represent each Java source set as

> a Node? That is, instead of

> 

>   com/example/lib1/*.java     # 35 files

>   com/example/lib2/*.java     # 17 files

>   com/example/app1/*.java     # 483 files

> 

> being 35 + 17 + 483 nodes, it should be 3 nodes: one for lib1, one for

> lib2, and one for app1. (Assuming each of those directories correspond

> to one source set, of course.)

> 

> Has anyone tried anything like this? My one previous attempt led me to

> the conclusion that subclassing Node directly is a hell-spawned

> nightmare, whereas subclassing File is fairly straightforward. But I

> don't want to subclass File, I need to subclass Node, because a

> collection of related files is not a file. It's a different kind of

> Node.

> 

> Would be curious to hear other opinions before I embark on this

> possibly mad journey. ;-)


I have embarked on this mad journey. As suspected, directly
subclassing Node is a fairly hellish nightmare. So far I've figured
out that I have to:

  * subclass Node (I'm calling it CollectiveNode)

  * implement CollectiveNode.changed_since_last_build()
    (so far I'm just returning true if any files have been
    added to or removed from the set: obviously incomplete,
    but it's a start)

  * subclass NodeInfo and make sure CollectiveNode uses that

  * subclass BuildInfo and make sure CollectiveNode uses that

So far so good, although I haven't actually done the BuildInfo
subclass yet, because I don't understand how it connects to .sconsign.
I mean, I can see from the docstrings that there is some connection
between BuildInfo and .sconsign, but how it works is a mystery.

Obviously File makes it work somehow, but there is a lot of code in
File, and it's totally unclear to me which of that code is necessary
to make this build's signatures wind up in .sconsign.

Any tips on where the magic lies? What do I have to do in my Node
subclass to make sure signatures from its NodeInfo object get saved in
.sconsign?

Thanks --

       Greg