[Scons-users] Having issue with scanner in Command()

Jason Kenny dragon512 at live.com
Mon May 14 14:05:18 EDT 2018 

will be happy to try it out
-Jason
________________________________
From: Scons-users <scons-users-bounces at scons.org> on behalf of Bill Deegan <bill at baddogconsulting.com>
Sent: Monday, May 14, 2018 12:05 PM
To: SCons users mailing list
Subject: Re: [Scons-users] Having issue with scanner in Command()

Jason,

I believe I have a patch for the rebuilds.
I found a simliar issue working on another problem.

If sconsign crashes reading the .sconsign then you likely have the same issue.
Give it a try and let me know.
I'm away from computer for a bit, but I'll send you the patch to try.
Basically the sconsign ends up with mismatched md5's for file names but only when ignore is being used because the names aren't filtered, but the md5's are.

-Bill


On Mon, May 14, 2018 at 12:44 PM, Jason Kenny <dragon512 at live.com<mailto:dragon512 at live.com>> wrote:
Hi Bill
I was out traveling last week, however, I made some progress on this.
If I disable the scanner ( changed the code to always return [] ) The build worked fine. So the issue is the scanner and how it adds depends to a target. I am unclear on why Ignore() is not working as expected. Scons wants to keep rebuilding because buildinf.h changed. If I don't have the Ignore() I get an error from the taskmaster on the second run with a circular depends. I would expect the only Ignore call would prevent rebuilds
As a workaround, I change the code from using the Parts Pattern object. which is basically a recursive Glob() at this point, to just copy the openssl source directory to the variant directory. This work as the c scanner is global on any C file being defined. By making this a directory the copy logic does not define any C file nodes. Since the C files are not defined as nodes they are not seen by the scanner to be scanned and added as a depends to the target of the Command() that generated the Makefile.
Is there an official way to turn off the scanner for a given command?? Or is this more of a bug with the Ignore() logic as Scons does not seem to "fully" ignore the depends added by the scanner.
either way, I have a workaround. However, it seems this is could be an unexpected catch 22 for people that have to call a "third party" build system to get a full build to work.
-Jason

________________________________
From: Jason Kenny
Sent: Friday, May 4, 2018 7:29 PM
To: SCons users mailing list
Subject: RE: [Scons-users] Having issue with scanner in Command()


In Parts the VCS_DIR is where the code is checked out too. The BUILD_DIR is basically the SCons Variant directory for the given “Part”/component. I was copying data to this place as OpenSSL does not have an out of source build option, and as such would “pollute” the source area.



I was looking at it more in details. If I build

“Scons _build/build_debug_posix-x86_64/openssl/Makefile”



Everything is fine SCons says everything is up to date the next round. If I build “_makeinstall/openssl/bin/c_rehash” which is defined in the second Command() the next pass has this problem. It seems for some reason once the generated building.h file is generated something is adding it as a depends. I am not sure at the moment if this is seen as implicit or explicated by the system. I am assuming it is implicate and being added by a scanner. However I have not found what added it as of yet. I don’t think it is anything in Parts. I did notice that if I remove the copyed->[] after everything is built remove the scons db file everything is fine. I think it has to do with my coping object as the source and they are getting scanned by the c-scanner but I am unclear how this would be adding it as a depends to the target. It has been a while since I have been in this code. I think SCons does not make a tree like:



Foo.exe

    + foo.obj

           + foo.c

                  +foo.h



But makes a



Foo.exe

    + foo.obj

           + foo.c

           + foo.h



My theory is the since SCons does the last case. Scons might be adding “generated” file of the openssl make build via the scanner at the wrong level. Since Scons would  add the .h file not as a depends of the .c but to the target that depends on the .c file.



I think I need to do a test run with the C-scanner disabled to verify this



Jason



From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> On Behalf Of Bill Deegan
Sent: Friday, May 4, 2018 3:23 PM
To: SCons users mailing list <scons-users at scons.org<mailto:scons-users at scons.org>>
Subject: Re: [Scons-users] Having issue with scanner in Command()



What's VCS_DIR and BUILD_DIR ?



On Fri, May 4, 2018 at 1:18 PM, Jason Kenny <dragon512 at live.com<mailto:dragon512 at live.com>> wrote:

to simplify the log file the tree looks like this:

+-build::alias::openssl

+-build::alias::openssl::SDKBIN

| +-_sdks/debug_posix-x86_64_default/openssl_1.1.0_51bed083228027f76730230d116bace9/bin/c_rehash

| | +-_build/build_debug_posix-x86_64/openssl/_set_RPATH_/c_rehash

| | +-_makeinstall/openssl/bin/c_rehash

| | +-_build/build_debug_posix-x86_64/openssl/Makefile

| | +-_build/build_debug_posix-x86_64/openssl/crypto/ripemd/build.info<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbuild.info&data=02%7C01%7C%7Cc105254391724c91923b08d5b1fcdf80%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636610621989426130&sdata=1LliS9gWIMdwRhA93U2fKW7HYbJLqp19YyohEjAcZGk%3D&reserved=0>

| | | +-_vcs/openssl/crypto/ripemd/build.info<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbuild.info&data=02%7C01%7C%7Cc105254391724c91923b08d5b1fcdf80%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636610621989426130&sdata=1LliS9gWIMdwRhA93U2fKW7HYbJLqp19YyohEjAcZGk%3D&reserved=0>

....

| | +-[_build/build_debug_posix-x86_64/openssl/crypto/LPdir_win.c]

| | +-[_build/build_debug_posix-x86_64/openssl/crypto/arm_arch.h]

| | +-_build/build_debug_posix-x86_64/openssl/crypto/buildinf.h

| | | +-[_build/build_debug_posix-x86_64/openssl/Makefile]

| | +-[_build/build_debug_posix-x86_64/openssl/crypto/LPdir_nyi.c]


I am not sure why  _makeinstall/openssl/bin/c_rehash -> _build/build_debug_posix-x86_64/openssl/Makefile ->_build/build_debug_posix-x86_64/openssl/crypto/buildinf.h -> _build/build_debug_posix-x86_64/openssl/Makefile

I would think c_rehash should depend on the Makefile. I would not think Scons would "know" anything about buildinf.h

- Jason

________________________________

From: Jason Kenny <dragon512 at live.com<mailto:dragon512 at live.com>>
Sent: Friday, May 4, 2018 11:26 AM

To: SCons users mailing list
Subject: Re: [Scons-users] Having issue with scanner in Command()



copyed was me coping data to the variant directory to allow cross build to work better

log file is here https://bitbucket.org/snippets/dragon512/KeGax5<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbitbucket.org%2Fsnippets%2Fdragon512%2FKeGax5&data=02%7C01%7C%7Cc105254391724c91923b08d5b1fcdf80%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636610621989426130&sdata=%2Fn83xUdN3J5rmYczw5ss%2F9zcjkQnka5BC1nqEw8eLtk%3D&reserved=0>


Below is the Parts file I have been hacking on ( I hope to clean it up more...) It is based on work I have yet to publish in Parts in my branch.


-----
Import('*')
PartVersion("1.1.0")
files = Pattern(src_dir="${VCS_DIR}", excludes=[".git/*"])

copyed = env.CCopy(
    source=files,
    target="$BUILD_DIR",
    copy_logic=CCopy.copy # could hard links be an issue????
)
out = env.Command(
    ["Makefile"],
    copyed,
    "cd ${TARGET.dir} ; ./Configure -DSSL_ALLOW_ADH enable-ec_nistp_64_gcc_128\
     --prefix=${ABSPATH('#_makeinstall')}/${PART_NAME}\
     --openssldir=${ABSPATH('#_makeinstall')}/${PART_NAME} linux-x86_64 shared\
     "
)

# env.Ignore("_build/build_debug_posix-x86_64/openssl/crypto/buildinf.h","${VCS_DIR}/crypto/buildinf.h")
env.Ignore(out, ["crypto/buildinf.h"])



out = env.Command(
    [
        env.File("crypto/buildinf.h"),
        "#_makeinstall/${PART_NAME}/bin/c_rehash",
        "#_makeinstall/${PART_NAME}/bin/openssl",
        "#_makeinstall/${PART_NAME}/include/openssl/aes.h",
        "#_makeinstall/${PART_NAME}/include/openssl/camellia.h",
        "#_makeinstall/${PART_NAME}/include/openssl/des.h",
        "#_makeinstall/${PART_NAME}/include/openssl/e_os2.h",
        "#_makeinstall/${PART_NAME}/include/openssl/md5.h",
        "#_makeinstall/${PART_NAME}/include/openssl/pem2.h",
        "#_makeinstall/${PART_NAME}/include/openssl/rsa.h",
        "#_makeinstall/${PART_NAME}/include/openssl/stack.h",
        "#_makeinstall/${PART_NAME}/include/openssl/x509_vfy.h",
        "#_makeinstall/${PART_NAME}/include/openssl/asn1.h",
        "#_makeinstall/${PART_NAME}/include/openssl/cast.h",
        "#_makeinstall/${PART_NAME}/include/openssl/dh.h",
        "#_makeinstall/${PART_NAME}/include/openssl/err.h",
        "#_makeinstall/${PART_NAME}/include/openssl/mdc2.h",
        "#_makeinstall/${PART_NAME}/include/openssl/pem.h",
        "#_makeinstall/${PART_NAME}/include/openssl/safestack.h",
        "#_makeinstall/${PART_NAME}/include/openssl/symhacks.h",
        "#_makeinstall/${PART_NAME}/include/openssl/asn1_mac.h",
        "#_makeinstall/${PART_NAME}/include/openssl/cmac.h",
        "#_makeinstall/${PART_NAME}/include/openssl/dsa.h",
        "#_makeinstall/${PART_NAME}/include/openssl/evp.h",
        "#_makeinstall/${PART_NAME}/include/openssl/modes.h",
        "#_makeinstall/${PART_NAME}/include/openssl/pkcs12.h",
        "#_makeinstall/${PART_NAME}/include/openssl/seed.h",
        "#_makeinstall/${PART_NAME}/include/openssl/tls1.h",
        "#_makeinstall/${PART_NAME}/include/openssl/asn1t.h",
        "#_makeinstall/${PART_NAME}/include/openssl/cms.h",
        "#_makeinstall/${PART_NAME}/include/openssl/dtls1.h",
        "#_makeinstall/${PART_NAME}/include/openssl/hmac.h",
        "#_makeinstall/${PART_NAME}/include/openssl/objects.h",
        "#_makeinstall/${PART_NAME}/include/openssl/pkcs7.h",
        "#_makeinstall/${PART_NAME}/include/openssl/sha.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ts.h",
        "#_makeinstall/${PART_NAME}/include/openssl/async.h",
        "#_makeinstall/${PART_NAME}/include/openssl/comp.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ebcdic.h",
        "#_makeinstall/${PART_NAME}/include/openssl/idea.h",
        "#_makeinstall/${PART_NAME}/include/openssl/obj_mac.h",
        "#_makeinstall/${PART_NAME}/include/openssl/rand.h",
        "#_makeinstall/${PART_NAME}/include/openssl/srp.h",
        "#_makeinstall/${PART_NAME}/include/openssl/txt_db.h",
        "#_makeinstall/${PART_NAME}/include/openssl/bio.h",
        "#_makeinstall/${PART_NAME}/include/openssl/conf_api.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ecdh.h",
        "#_makeinstall/${PART_NAME}/include/openssl/kdf.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ocsp.h",
        "#_makeinstall/${PART_NAME}/include/openssl/rc2.h",
        "#_makeinstall/${PART_NAME}/include/openssl/srtp.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ui.h",
        "#_makeinstall/${PART_NAME}/include/openssl/blowfish.h",
        "#_makeinstall/${PART_NAME}/include/openssl/conf.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ecdsa.h",
        "#_makeinstall/${PART_NAME}/include/openssl/lhash.h",
        "#_makeinstall/${PART_NAME}/include/openssl/opensslconf.h",
        "#_makeinstall/${PART_NAME}/include/openssl/rc4.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ssl2.h",
        "#_makeinstall/${PART_NAME}/include/openssl/whrlpool.h",
        "#_makeinstall/${PART_NAME}/include/openssl/bn.h",
        "#_makeinstall/${PART_NAME}/include/openssl/crypto.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ec.h",
        "#_makeinstall/${PART_NAME}/include/openssl/md2.h",
        "#_makeinstall/${PART_NAME}/include/openssl/opensslv.h",
        "#_makeinstall/${PART_NAME}/include/openssl/rc5.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ssl3.h",
        "#_makeinstall/${PART_NAME}/include/openssl/x509.h",
        "#_makeinstall/${PART_NAME}/include/openssl/buffer.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ct.h",
        "#_makeinstall/${PART_NAME}/include/openssl/engine.h",
        "#_makeinstall/${PART_NAME}/include/openssl/md4.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ossl_typ.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ripemd.h",
        "#_makeinstall/${PART_NAME}/include/openssl/ssl.h",
        "#_makeinstall/${PART_NAME}/include/openssl/x509v3.h",
        "#_makeinstall/${PART_NAME}/lib/engines-1.1/afalg.so",
        "#_makeinstall/${PART_NAME}/lib/libssl.a",
        "#_makeinstall/${PART_NAME}/lib/engines-1.1/capi.so",
        "#_makeinstall/${PART_NAME}/lib/engines-1.1/padlock.so",
        "#_makeinstall/${PART_NAME}/lib/libssl.so",
        "#_makeinstall/${PART_NAME}/lib/libssl.so.1.1",
        "#_makeinstall/${PART_NAME}/lib/libcrypto.a",
        "#_makeinstall/${PART_NAME}/lib/pkgconfig/libcrypto.pc",
        "#_makeinstall/${PART_NAME}/lib/pkgconfig/libssl.pc",
        "#_makeinstall/${PART_NAME}/lib/libcrypto.so",
        "#_makeinstall/${PART_NAME}/lib/libcrypto.so.1.1",
        "#_makeinstall/${PART_NAME}/lib/pkgconfig/openssl.pc",
    ],
    ["Makefile"],
    [
        "cd ${{SOURCE.dir}} ; make all CC=${{CC}} CXX=${{CXX}} $(-j{jobs}$)".format(jobs=env.GetOption('num_jobs')),
        "cd ${SOURCE.dir} ; make install"
    ]
)

env.InstallBin(
        env.Glob("#_makeinstall/${PART_NAME}/bin/*")
)
env.InstallLib(env.Glob("#_makeinstall/${PART_NAME}/lib/*.a"), auto_add_libs=False)
env.InstallLib(
        env.Glob("#_makeinstall/${PART_NAME}/lib/*.so*")

)
env.InstallLib(env.Glob("#_makeinstall/${PART_NAME}/libengines-1.1/*.so"), auto_add_libs=False, sub_dir="libengines-1.1")
env.InstallInclude(  # bug we need to fix -JK
    env.SdkInclude(
        env.Glob("#_makeinstall/${PART_NAME}/include/openssl/*"),
        sub_dir="openssl",
        add_to_path=False
    ),
    sub_dir="openssl"
)
env.InstallLib(env.Glob("#_makeinstall/${PART_NAME}/lib/pkgconfig/*.pc"), auto_add_libs=False, sub_dir="pkgconfig", add_to_path=False)



________________________________

From: Scons-users <scons-users-bounces at scons.org<mailto:scons-users-bounces at scons.org>> on behalf of Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>>
Sent: Friday, May 4, 2018 10:40 AM
To: SCons users mailing list
Subject: Re: [Scons-users] Having issue with scanner in Command()



Jason,



I"m not aware of a Makefile scanner.



What's in variable 'copyed'?



Can you paste output of --tree=prune?

-Bill



On Fri, May 4, 2018 at 10:31 AM, Jason Kenny <dragon512 at live.com<mailto:dragon512 at live.com>> wrote:

Hi,

I am setting up a build with SCons that has to build openssl. I am using the Command() logic to call the native build system. I am having an issue with this code wanting to rebuild because of a file "crypto/buildinf.h" is rebuilt as part of the process with a new date value/ This causes SCons to say stuff needs to be rebuilt. The issue here is an odd circular reference that is forming up and I am unclear on what is adding it at this time. I am hoping before I dig to deep into this that some one might know, or have some idea on what is happening. Details

I have a two commands:

out = env.Command(

["Makefile"],

copyed,

"cd ${TARGET.dir} ; ./Configure -DSSL_ALLOW_ADH enable-ec_nistp_64_gcc_128\

--prefix=${ABSPATH('#_makeinstall')}/${PART_NAME}\

--openssldir=${ABSPATH('#_makeinstall')}/${PART_NAME} linux-x86_64 shared\

"

)

Then I have a command

env.Ignore(out, ["crypto/buildinf.h"])



env.Command(
[

env.File("crypto/buildinf.h"),

"#_makeinstall/${PART_NAME}/bin/c_rehash",

...

],

["Makefile"],

[

"cd ${{SOURCE.dir}} ; make all CC=${{CC}} CXX=${{CXX}} $(-j{jobs}$)".format(jobs=env.GetOption('num_jobs')),

"cd ${SOURCE.dir} ; make install"

] )



What is happening is that after the first run Scons will tell me it needs to rebuild because:

scons: rebuilding `_build/build_debug_posix-x86_64/openssl/Makefile' because `_build/build_debug_posix-x86_64/openssl/crypto/buildinf.h' changed

I have an Ignore line that should remove this... What is odd for me is that the next build command is run because:

scons: rebuilding `_build/build_debug_posix-x86_64/openssl/crypto/buildinf.h' because `_build/build_debug_posix-x86_64/openssl/Makefile' changed

I seem to have a cycle of some kind. I am unclear on how to break it.

I am unclear on what scanner is adding this depends in Command. Do we have a generic Makefile scanner??
Any thoughts??

Jason

_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7Ccae88553915f47d8c91a08d5b1d5712f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636610452634906164&sdata=LJ6F2%2BiZCnp1WZOIAfNDmAs4%2F3h3JJNnFtz0GnVCX4g%3D&reserved=0>



_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7Cc105254391724c91923b08d5b1fcdf80%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636610621989426130&sdata=6C3axeb9ZEUCd0ra2hdKQ8%2BP5Nr1WGdNl0wPQPgBAwI%3D&reserved=0>



_______________________________________________
Scons-users mailing list
Scons-users at scons.org<mailto:Scons-users at scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7C3d6fa86b3b704e0d3b0408d5b9bcdf81%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636619143209334764&sdata=9XQji%2F6OPpBOB%2Fvk5%2F0KXcpUbuBudRz2trdZxTUM9Ac%3D&reserved=0>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20180514/7ab7f481/attachment-0001.html>

More information about the Scons-users mailing list