[Scons-users] env[X'] and signatures

Thu Dec 29 18:11:15 EST 2016

Albert,

Some other issue with having big blobs of python logic inside your SCons
logic:
1) You're blocking parallel builds as the GIL comes into play
2) Testing and documenting the logic can be harder.

How much info are you holding inside your scons which would need to be
provided to such a script? (10's 100's 1000's of arguments worth?)

-Bill

On Thu, Dec 29, 2016 at 3:07 PM, Albert Arquer <albert.arquer at gmail.com>
wrote:

> Right now its pretty much all python functions instead of separate
> scripts. I see ehat you mean with the scripts though... However there can
> sometimes be many many many information passed throgh the builder, which
> passing it throgh command line makes it a bit messy...
>
>
> El dv, 30 des 2016 a les 0:02 Bill Deegan <bill at baddogconsulting.com> va
> escriure:
>
>> Albert,
>>
>> Is there a command line for the tagging target?
>> Or this a python builder inside your SCons implementation?
>>
>> Generally I suggest not having complex python builders in your
>> implementation, but rather have scripts called by SCons.
>> This would simplify the case where your just changing a version as you'd
>> make the version be part of the command line to the script.
>> Then it would only rerun when the version changed..
>>
>> -Bill
>>
>> On Thu, Dec 29, 2016 at 2:51 PM, Albert Arquer <albert.arquer at gmail.com>
>> wrote:
>>
>> Well it's pretty complex as I am sure you can imagine, you have all sorts
>> of files. Verilos systemverilog, vhdl, constraints, headers, vendor files
>> like xci, qip, etc.
>>
>> However lets say I have an fpga build flow split in 4 phases.
>>   1. Synthesis
>>   2. Place & Route
>>   3. Tagging. Gets the bitstream from place and route and modifies ONLY
>> the contents of a block ram, adding version, timestamp, build information
>> strings, etc. This is to go from an unofficial development build (untested)
>> to an officially tagged and tested release.
>>   4. Configuration image generarion. Get one (or multiple) bitstreams
>> along with some hex files for example, some parameters, and generate an SPI
>> configuration file to load onto the FPGA.
>>
>> Now, each of these targets depends on the previous and some extra stuff
>> (like hex files and parameters), the synthesis target depends on the hdl
>> sources intself. Both synthesis and place and route can add some extra
>> sources, constraints for instance.
>>
>> Lets say I have already built the whole process, so I have the output
>> files and they are up-to-date. Now lets say I just want to go to the
>> tagging target and modify some tagging information, like the version. I go
>> and open the sconscript which defines that target and modify that
>> parameter. However that file also defines all other targets, so if my
>> targets depend on that sconscript they are all out-dated now!
>>
>> However the only target that has trully changed is the tagging target
>> (and therefore the next and last stage too), what I miss about scons is a
>> way of each builder detecting if it's environment has changed since the
>> last build.
>>
>> What I have done to fix this is have scons hash the whole environment for
>> each target and add that hash to the builders signature. This allows me to
>> remove the sconscripts from the target's dependencies, and still have the
>> target rebuilt when a prameter is changed. At least In our case changes in
>> a target's environment are much much more unlikely than the scenario I just
>> described, so this aproach means a huge improvement for us.
>>
>> As I have mentioned in the previous message I would like to be able to
>> use dill instead off pickle to serialise.
>>
>> Albert.
>>
>> El dj, 29 des 2016 a les 23:32 Bill Deegan <bill at baddogconsulting.com>
>> va escriure:
>>
>> Albert,
>>
>> I'm very familiar with the EDA space.
>> What's the input file?
>> What environmnent variables changing would trigger a rebuild for you?
>>
>> -Bill
>>
>> On Thu, Dec 29, 2016 at 9:58 AM, Albert Arquer <albert.arquer at gmail.com>
>> wrote:
>>
>> I agree it is not the ideal solution. In fact we use scons for hdl
>> synthesis/implementation which can take many hours.
>>
>> Before we used to make the target depend on the sconscript which declared
>> the target. However this caused many innecessary rebuilds, as a single
>> sconscript often declares multiple targets, and even if a change in the
>> sconscript which only afected a single target was modified, scons would
>> rebuild all other targets declared in the sconscript and any others which
>> depended on them.
>>
>> By hashing the environment we can isolate sconscript modifications to the
>> targets it actually affects. This is not ideal ofcourse, it would be much
>> better for each builder to define somehow which items from the environment
>> it will be using. However this problem usually causes build to be repeated,
>> not missed, and still counting those cases the innecessary builts are
>> drastically reduced (at least in our case).
>>
>>  There is also the issue on the hashing itself, since not everything is
>> hashable in python, and even if an object is hashable that hash might not
>> change if the object's class definition has changed for example. I opened
>> this thread in stackoverflow a while back trying to find a serialization
>> module suited for this purpose, turns out dill would be a good choice but
>> it has compatibility issues with scons.
>>
>> http://stackoverflow.com/questions/40640797/compute-
>> class-and-instance-hash/40775525#40775525
>>
>>
>> El dj, 29 des 2016 a les 18:51 Bill Deegan <bill at baddogconsulting.com>
>> va escriure:
>>
>> As a workaround, perhaps add an action string which does nothing, but
>> uses the variables you depend on?
>> Just a thought.
>>
>>
>> Are the builders external python scripts which get run from the shell by
>> SCons? or are the builders python logic run inside scons?
>>
>> On Thu, Dec 29, 2016 at 8:54 AM, Tom Tanner <trtanner at btinternet.com>
>> wrote:
>>
>> On 27/12/16 18:22, Bill Deegan wrote:
>>
>>
>>
>>
>> If every builder depends on all values in env['ENV'], then any change
>> will rebuild everything.
>>
>>
>> This is almost never warranted.
>>
>>
>>
>>
>>
>>
>>
>> But then why does scons clear out the supplied environment? It is
>> precisely because everything behaves in a way that is unclear and not
>> reproducable if you don't know exactly what the environment is.
>>
>>
>>
>>
>> I think, ideally, a builder (Action really), would specify what parts of
>> the Environment() it's sensitive too.
>>
>>
>>
>>
>>
>>
>>
>> This would be better. If every builder specified which parts of
>> env['ENV'] it is interested in, and they (and nothing else) got passed
>> through and added to the build signature
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Remember, while many users are only building source code which is fairly
>> quick, SCons is used in other "build" systems where steps can take
>> considerable time and resources so extraneous rebuilds are quite costly for
>> them.
>>
>>
>>
>>
>>
>> -Bill
>>
>>
>>
>>
>>
>>
>>
>> I agree but specifying the correct dependencies are how you avoid costly
>> rebuilds. Not rebuilding something that has changed can be just as costly
>> because instead of finding out about it at the build stage, you find out
>> about it when it is on a production machine.
>>
>>
>>
>> _______________________________________________
>>
>>
>> Scons-users mailing list
>>
>>
>> Scons-users at scons.org
>>
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> Scons-users mailing list
>>
>> Scons-users at scons.org
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>>
>>
>> Scons-users mailing list
>>
>>
>> Scons-users at scons.org
>>
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> Scons-users mailing list
>>
>> Scons-users at scons.org
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>>
>>
>> Scons-users mailing list
>>
>>
>> Scons-users at scons.org
>>
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> Scons-users mailing list
>>
>> Scons-users at scons.org
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161229/f5ce3d0d/attachment-0001.html>