[Scons-users] CacheDir entries

Tue Oct 28 06:44:01 EDT 2014

Apologies if the quoting is all peculiar, my email client at work is - meh.

As far as I understand it, the hash for stuff in the cache consists the md5sum of the md5sum of all things involved in the build, which will be
any header files spotted by header scan
any explicit dependencies
the command line

Depending on your exact setup this could include the md5sum of the compiler (it certainly does for us, which occasionally causes fun as the IBM compiler is actually a wrapper script and depending on the precise options given at install time it can have different contents and hence a different md5sum on different machines, even though it behaves the same).

It doesn't include the md5sum of the object being produced, because you won't know that at the time.

So you should be able to succesfully share the cache directory even if you're using different compiler versions on different machines.

However, I have to say I'd find installing different versions of gcc with the same path on different machines is going to give you all sorts of issues. We make sure that different versions of a compilers are installed under different directories (gcc-4.8.1, gcc-4.9.0) because of the sort of mess that can get you into.

Being a pessimist, I have a different cache for each architecture. It's not strictly necessary but you can end up with quite a lot of files in the cache, which has other issues (and in theory we could keep the cache local to the machine)

----- Original Message -----
From: scons-users at scons.org
To: scons-users at scons.org
At: Oct 25 2014 20:06:20

I don't think it actually includes the hash of the contents of the compiler binary. As an experiment, I built my project with /opt/local/bin/clang being a symlink to /opt/local/bin/clang-3.5. I then updated the symlink to point to /opt/local/bin/clang-3.4. After switching the symlink, rebuilding was a no-op.

I also tried editing a system header (/usr/include/stdio.h), by adding a new #define to it. SCons did not attempt to rebuild targets after this edit. I verified that the file was in fact included by adding an intentional error to stdio.h and manually rebuilding a target, which failed with a compiler error as expected.

So, unless I'm mistaken in expecting that if SCons would not rebuilt targets on the basis of a change to a system header that it would also not use the contents of that system header when producing a cache entry, I think the CacheDir mechanism cannot differentiate between identical build invocations in the same source tree across systems with varying system headers or compiler versions.

What Decider type are you using?  In this case, I imagine that any of the above changes SHOULD trigger a rebuild unless SCons doesn't track information on dependencies outside the SConstruct tree (other than existence).  That brings up a set of concerns regarding build consistency.  I don't use CacheDir functionality generally, but I know that under a basic build structure SCons rebuilds when these external dependencies change.

V/R,
William 

On Sat, Oct 25, 2014 at 2:49 PM, Andrew C. Morrow <andrew.c.morrow at gmail.com> wrote:

On Fri, Oct 24, 2014 at 7:13 PM, William Blevins <wblevins001 at gmail.com> wrote:

Andrew,

I'm not sure I am qualified to answer this question, but since no one else has attempted, I will try to help the best I can :)

Thanks!

What pieces of information are combined to identify a CacheDir entry?

I imagine that all the information written into .sconsign.dblite for a target is used.

OK. That is somewhat helpful as it means one less thing to understand. However, it also means that now I have the same question but about the sconsign file. Is a complete description of the inputs used to derive an entry in .sconsign.dblite described somewhere?

I ask because I'm interested in setting up a CacheDir on a network share, but it is not clear to me what I need to do to make this work with heterogeneous client systems.

I wonder if CacheDir is thread safe.  Can more than 1 SCons instance interact with CacheDir files at a time?  That might be the biggest question. 

Based on the comment here:

https://bitbucket.org/scons/scons/src/cff67baa7ab35fc262bfc08298fe365e537422fc/src/engine/SCons/CacheDir.py?at=default#cl-82

it seems that it is intended to be possible to share CacheDirs between concurrent builds. Thinking about it abstractly it seems like it should work: if you look for the file in the cache and don't find it, you rebuild it. If you look for it and find it, you use it. The only race I can see would be if you were to consume an entry that was in the process of being written by another process and read an incomplete object file, but it appears from a quick look at CacheDir.py that new cache entries are first written to a temporary file and only then renamed to the cached name. Assuming the underlying FS provides atomic file renames, I think this is expected to work.

Some further evidence that this is intended to work correctly comes from this comment in the man page:

"The  --random  option  is useful to prevent multiple builds from trying to update the cache simultaneously."

As a practical example, consider two otherwise identical linux workstations, where the only difference is that one has GCC 4.8 and the other has GCC 4.9, but both installed as /usr/bin/gcc. Would these workstations share cache entries, assuming identical source files and tool invocations? Why, or why not? Similar question for two machines with varying contents of /usr/include/stdio.h or other system C and C++ headers reached by translation units? For /usr/bin/ld?

A SCons target essentially depends on every artifact that goes into a build command which includes the compiler and/or linker.  Looking at the dependency tree for an object compiled with gcc will show the gcc executable as a dependency, so the cached objects will (most likely) differ (based on the Decider) since the gcc binaries will not be equivalent.  Of course this relates to ALL target traits including compiler, compiler arguments, library paths, include paths, etc.

I don't think it actually includes the hash of the contents of the compiler binary. As an experiment, I built my project with /opt/local/bin/clang being a symlink to /opt/local/bin/clang-3.5. I then updated the symlink to point to /opt/local/bin/clang-3.4. After switching the symlink, rebuilding was a no-op.

I also tried editing a system header (/usr/include/stdio.h), by adding a new #define to it. SCons did not attempt to rebuild targets after this edit. I verified that the file was in fact included by adding an intentional error to stdio.h and manually rebuilding a target, which failed with a compiler error as expected.

So, unless I'm mistaken in expecting that if SCons would not rebuilt targets on the basis of a change to a system header that it would also not use the contents of that system header when producing a cache entry, I think the CacheDir mechanism cannot differentiate between identical build invocations in the same source tree across systems with varying system headers or compiler versions.

The system I'm considering would have a few populations of mostly homogeneous systems, so if necessary I could capture some information from the external environment (contents of env['CC'] --version, contents of /etc/lsb_release, etc.) to identify which population a given machine belongs to. If I had such a machine fingerprint, what would be the best way to inject it into the SCons view of the world such that cache entries with different fingerprints were differentiated?

In this case, I don't this that the platform will have an effect as long as the tools and dependencies are the same.  A different linux OS may have different system headers, etc, which would cause cache misses potentially.  Also, I know that the md5sum implementation for linux changed between RHEL5 and RHEL6, so that may also cause issues if python uses a 3rd-party library for this?  I imagine python uses its own implementation though...

Based on the two experiments above, my current view is that variations across systems headers and compiler versions will not be detected, so I think some sort of system fingerprint is still required to avoid false sharing of cache entries. Given that, do you have any thoughts on the best way to get such a system fingerprint (assuming I can construct a correct one) to affect every cache entry computation? Essentially, I want to use the system fingerprint as a salt for the cache entry hash.

Sorry I couldn't be more helpful,
William

I appreciate your taking the time to get the conversation going. Hopefully some other people will chime in with more details.

Thanks,
Andrew

On Fri, Oct 24, 2014 at 12:47 PM, Andrew C. Morrow <andrew.c.morrow at gmail.com> wrote:

Hi All -

What pieces of information are combined to identify a CacheDir entry?

I ask because I'm interested in setting up a CacheDir on a network share, but it is not clear to me what I need to do to make this work with heterogeneous client systems.

As a practical example, consider two otherwise identical linux workstations, where the only difference is that one has GCC 4.8 and the other has GCC 4.9, but both installed as /usr/bin/gcc. Would these workstations share cache entries, assuming identical source files and tool invocations? Why, or why not? Similar question for two machines with varying contents of /usr/include/stdio.h or other system C and C++ headers reached by translation units? For /usr/bin/ld?

The system I'm considering would have a few populations of mostly homogeneous systems, so if necessary I could capture some information from the external environment (contents of env['CC'] --version, contents of /etc/lsb_release, etc.) to identify which population a given machine belongs to. If I had such a machine fingerprint, what would be the best way to inject it into the SCons view of the world such that cache entries with different fingerprints were differentiated?

One obvious solution would be to have different subdirectories of the network mount for each client fingerprint, but that seems clunky. Are any contents of the Environment hashed into the cache id? If not, can I request that an Environment entry be incorporated, so that I could have env['MACHINE_FINGERPRINT'] = get_machine_fingerprint() and make that affect the cache id? Are there other techniques for injecting this sort of auxilliary information into the cache id for a node?

Thoughts on the above questions, comments from others who have attempted this, or pointers to the relevant subsets of the SCons source would be greatly appreciated.

Thanks, and apologies for the wall of questions,
Andrew

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________ Scons-users mailing list Scons-users at scons.org https://pairlist4.pair.net/mailman/listinfo/scons-users 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141028/efcca438/attachment-0001.html>