[Scons-users] Issue with repositories

Tom Tanner (BLOOMBERG/ LONDON) ttanner2 at bloomberg.net
Wed Sep 25 07:16:21 EDT 2013

I think the correct fix for thit issue is to change Get_Database in SConsign.py to return the calculated mode in the fallthrough try block at the end.

What happens is that Get_DatabaseDir determines that the directory is a repository, but it can't then open the database file at line 73 (because it doesn't exist, because there is only one). However, the file is in a repository.

This leads onto a whole load of other things that I don't understand...

Why is this in the .sconsign file:
=== /usr/local/bin:
g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332

as it's already in the implicit dependency for anything that needs it.

Does give a performance improvement?

I have a sort of 'nicer' (for some interpretation of nicer) that replaces line 74
except (IOError, OSError): pass
with
except (IOError, OSError):
if mode == "r":
GetDatabase(top)
return DataBase[top], mode

however, that removes all the top level information about files in the repository from the .sconsign file, and I don't know if that is a performance issue.

----- Original Message -----
From: ttanner2 at bloomberg.net
To: scons-users at scons.org
At: Sep 24 2013 16:52:40

OK, I think I have discovered a problem.

Firstly, the deduced includes are stored in .sconsign.dblite under the full path of the place they were found, thus:
=== /home13/ttanner/test:
a.hh: d41d8cd98f00b204e9800998ecf8427e 1342541574 0

but the source code files and the object files are stored relative to the repository base:
=== fred:
b.cc: d41d8cd98f00b204e9800998ecf8427e 1379947770 0
b.o: 7398ff69997948bc3dd4bc0fd2d12e7a 1380007349 452
fred/b.cc: d41d8cd98f00b204e9800998ecf8427e 1379947770 0
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
bd195602be1db3bf1c055a2cd88bcc22 [$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES]

Now, when the SConsign *reads* from the sconsign file, it has this in class DB, Sconsign.py round about line 200:

# Read using the path relative to the top of the Repository
# (self.dir.tpath) from which we're fetching the signature
# information.
path = normcase(dir.tpath)

this means that when it fetches header files which appear to have ended up in the .sconsign.dblite with a full path, it turns the full path it is given into a repository base path, and fetches the source and derived files, not the header files (slightly peculiar), but NOT the header file (!)

Which means instead of reading the header file information from the repo, it reads the .o files and source code, and puts them in the wrong place.

note when you write stuff. it has this comment:
# Write using the path relative to the top of the SConstruct
# directory (self.dir.path), not relative to the top of
# the Repository; we only write to our own .sconsign file,
# not to .sconsign files in Repositories.

(some of these comments are a little confusing as there's only one sconsign file!).

So it appears to be of the opinion that the include files (in particular) have been deduced relative to the repository. But clearly they haven't, they've got absolute paths in them, and the repository hasn't been stripped.

I'm at a bit of a loss as to where things are going wrong here, or at least where they should be corrected. Include files aren't the only thing that get this issue (or at least not necessarily).

A 'quick hack' is to replace dir.path with str(dir) when reading the sconsfile. This is currently working for me, but I can envisage all sorts of exciting issues.

HELP!!


----- Original Message -----
From: ttanner2 at bloomberg.net
To: scons-users at scons.org
At: Sep 24 2013 09:06:56

*sigh* I forgot to add the command line:

> cd out

)> scons -f ../SConstruct -Y ..


----- Original Message -----
From: ttanner2 at bloomberg.net
To: scons-users at scons.org
At: Sep 24 2013 08:32:47

I've discovered something odd happens to my sconsign file with repositories:
Given this directory structure in my 'test' directory
a.hh
a.cc includes a.hh
fred/b.cc
SConstruct
out/ (for building in)

SConstruct contains:
----
env = DefaultEnvironment()

o2 = env.StaticObject('fred/b.cc')
a = env.Object('a.cc')
env.Program('a.out', (a, o2))
-----
build once, and run sconsign:
--------
=== .:
a.cc: 49995f9409d51c51534c02483d262eb9 1380007290 46
a.o: bf0d95f1615e5adc89d702276b0aa2a9 1380007349 492
a.cc: 49995f9409d51c51534c02483d262eb9 1380007290 46
/home13/ttanner/test/a.hh: d41d8cd98f00b204e9800998ecf8427e 1342541574 0
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
7ddbddaea8e8f90e89d45e9fa7d470d3 [$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES]
a.out: 567092ea1ea42d1d1a17d6563f5c2404 1380007349 5744
a.o: bf0d95f1615e5adc89d702276b0aa2a9 1380007349 492
fred/b.o: 7398ff69997948bc3dd4bc0fd2d12e7a 1380007349 452
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
6492ffd1a58e6c69946cb8b6d4001bb7 [$LINK -o $TARGET $LINKFLAGS $__RPATH $SOURCES $_LIBDIRFLAGS $_LIBFLAGS]
=== /home13/ttanner/test:
a.hh: d41d8cd98f00b204e9800998ecf8427e 1342541574 0
=== /usr/local/bin:
g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
=== fred:
b.cc: d41d8cd98f00b204e9800998ecf8427e 1379947770 0
b.o: 7398ff69997948bc3dd4bc0fd2d12e7a 1380007349 452
fred/b.cc: d41d8cd98f00b204e9800998ecf8427e 1379947770 0
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
bd195602be1db3bf1c055a2cd88bcc22 [$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES]
--------
Now all I do is change a.cc by adding a comment, and the sconsign file changes to this:
----------
=== .:
a.cc: 478319ac3c75658c260975e0a6943067 1380007583 55
a.o: bf0d95f1615e5adc89d702276b0aa2a9 1380007591 492
a.cc: 478319ac3c75658c260975e0a6943067 1380007583 55
/home13/ttanner/test/a.hh: d41d8cd98f00b204e9800998ecf8427e 1342541574 0
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
7ddbddaea8e8f90e89d45e9fa7d470d3 [$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES]
a.out: 567092ea1ea42d1d1a17d6563f5c2404 1380007349 5744
a.o: bf0d95f1615e5adc89d702276b0aa2a9 1380007591 492
fred/b.o: 7398ff69997948bc3dd4bc0fd2d12e7a 1380007349 452
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
6492ffd1a58e6c69946cb8b6d4001bb7 [$LINK -o $TARGET $LINKFLAGS $__RPATH $SOURCES $_LIBDIRFLAGS $_LIBFLAGS]
=== /home13/ttanner/test:
a.cc: 49995f9409d51c51534c02483d262eb9 1380007290 46
a.hh: d41d8cd98f00b204e9800998ecf8427e 1342541574 0
a.o: bf0d95f1615e5adc89d702276b0aa2a9 1380007349 492
a.cc: 49995f9409d51c51534c02483d262eb9 1380007290 46
/home13/ttanner/test/a.hh: d41d8cd98f00b204e9800998ecf8427e 1342541574 0
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
7ddbddaea8e8f90e89d45e9fa7d470d3 [$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES]
a.out: 567092ea1ea42d1d1a17d6563f5c2404 1380007349 5744
a.o: bf0d95f1615e5adc89d702276b0aa2a9 1380007349 492
fred/b.o: 7398ff69997948bc3dd4bc0fd2d12e7a 1380007349 452
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
6492ffd1a58e6c69946cb8b6d4001bb7 [$LINK -o $TARGET $LINKFLAGS $__RPATH $SOURCES $_LIBDIRFLAGS $_LIBFLAGS]
=== /usr/local/bin:
g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
=== fred:
b.cc: d41d8cd98f00b204e9800998ecf8427e 1379947770 0
b.o: 7398ff69997948bc3dd4bc0fd2d12e7a 1380007349 452
fred/b.cc: d41d8cd98f00b204e9800998ecf8427e 1379947770 0
/usr/local/bin/g++: 7cb5dc0848a5cc6435829c5579423bd0 1067169538 379332
bd195602be1db3bf1c055a2cd88bcc22 [$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES]
---------------

Note that a.o is now showing in /home13/ttanner/test and .

This causes at least one of my problems with checking for out-of-date files because there's 2 versions of a.o and one of them gets the signature updated but it appears it's the *other* one that gets passed to the Decider function so when I do the target.get_stored_info().ninfo.csig, I get a different target to what the build is using, making it look like the md5 is incorrect.


Not to mention it doubles the size of the .sconsign file which is quite considerable for our real build.
_______________________________________________
Scons-users mailing list
Scons-users at scons.org
http://four.pairlist.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
Scons-users at scons.org
http://four.pairlist.net/mailman/listinfo/scons-users


_______________________________________________
Scons-users mailing list
Scons-users at scons.org
http://four.pairlist.net/mailman/listinfo/scons-users

More information about the Scons-users mailing list