[Scons-users] env[X'] and signatures

Thu Dec 29 18:02:30 EST 2016

Albert,

Is there a command line for the tagging target?
Or this a python builder inside your SCons implementation?

Generally I suggest not having complex python builders in your
implementation, but rather have scripts called by SCons.
This would simplify the case where your just changing a version as you'd
make the version be part of the command line to the script.
Then it would only rerun when the version changed..

-Bill

On Thu, Dec 29, 2016 at 2:51 PM, Albert Arquer <albert.arquer at gmail.com>
wrote:

> Well it's pretty complex as I am sure you can imagine, you have all sorts
> of files. Verilos systemverilog, vhdl, constraints, headers, vendor files
> like xci, qip, etc.
>
> However lets say I have an fpga build flow split in 4 phases.
>   1. Synthesis
>   2. Place & Route
>   3. Tagging. Gets the bitstream from place and route and modifies ONLY
> the contents of a block ram, adding version, timestamp, build information
> strings, etc. This is to go from an unofficial development build (untested)
> to an officially tagged and tested release.
>   4. Configuration image generarion. Get one (or multiple) bitstreams
> along with some hex files for example, some parameters, and generate an SPI
> configuration file to load onto the FPGA.
>
> Now, each of these targets depends on the previous and some extra stuff
> (like hex files and parameters), the synthesis target depends on the hdl
> sources intself. Both synthesis and place and route can add some extra
> sources, constraints for instance.
>
> Lets say I have already built the whole process, so I have the output
> files and they are up-to-date. Now lets say I just want to go to the
> tagging target and modify some tagging information, like the version. I go
> and open the sconscript which defines that target and modify that
> parameter. However that file also defines all other targets, so if my
> targets depend on that sconscript they are all out-dated now!
>
> However the only target that has trully changed is the tagging target (and
> therefore the next and last stage too), what I miss about scons is a way of
> each builder detecting if it's environment has changed since the last build.
>
> What I have done to fix this is have scons hash the whole environment for
> each target and add that hash to the builders signature. This allows me to
> remove the sconscripts from the target's dependencies, and still have the
> target rebuilt when a prameter is changed. At least In our case changes in
> a target's environment are much much more unlikely than the scenario I just
> described, so this aproach means a huge improvement for us.
>
> As I have mentioned in the previous message I would like to be able to use
> dill instead off pickle to serialise.
>
> Albert.
>
> El dj, 29 des 2016 a les 23:32 Bill Deegan <bill at baddogconsulting.com> va
> escriure:
>
>> Albert,
>>
>> I'm very familiar with the EDA space.
>> What's the input file?
>> What environmnent variables changing would trigger a rebuild for you?
>>
>> -Bill
>>
>> On Thu, Dec 29, 2016 at 9:58 AM, Albert Arquer <albert.arquer at gmail.com>
>> wrote:
>>
>> I agree it is not the ideal solution. In fact we use scons for hdl
>> synthesis/implementation which can take many hours.
>>
>> Before we used to make the target depend on the sconscript which declared
>> the target. However this caused many innecessary rebuilds, as a single
>> sconscript often declares multiple targets, and even if a change in the
>> sconscript which only afected a single target was modified, scons would
>> rebuild all other targets declared in the sconscript and any others which
>> depended on them.
>>
>> By hashing the environment we can isolate sconscript modifications to the
>> targets it actually affects. This is not ideal ofcourse, it would be much
>> better for each builder to define somehow which items from the environment
>> it will be using. However this problem usually causes build to be repeated,
>> not missed, and still counting those cases the innecessary builts are
>> drastically reduced (at least in our case).
>>
>>  There is also the issue on the hashing itself, since not everything is
>> hashable in python, and even if an object is hashable that hash might not
>> change if the object's class definition has changed for example. I opened
>> this thread in stackoverflow a while back trying to find a serialization
>> module suited for this purpose, turns out dill would be a good choice but
>> it has compatibility issues with scons.
>>
>> http://stackoverflow.com/questions/40640797/compute-
>> class-and-instance-hash/40775525#40775525
>>
>>
>> El dj, 29 des 2016 a les 18:51 Bill Deegan <bill at baddogconsulting.com>
>> va escriure:
>>
>> As a workaround, perhaps add an action string which does nothing, but
>> uses the variables you depend on?
>> Just a thought.
>>
>>
>> Are the builders external python scripts which get run from the shell by
>> SCons? or are the builders python logic run inside scons?
>>
>> On Thu, Dec 29, 2016 at 8:54 AM, Tom Tanner <trtanner at btinternet.com>
>> wrote:
>>
>> On 27/12/16 18:22, Bill Deegan wrote:
>>
>>
>>
>>
>> If every builder depends on all values in env['ENV'], then any change
>> will rebuild everything.
>>
>>
>> This is almost never warranted.
>>
>>
>>
>>
>>
>>
>>
>> But then why does scons clear out the supplied environment? It is
>> precisely because everything behaves in a way that is unclear and not
>> reproducable if you don't know exactly what the environment is.
>>
>>
>>
>>
>> I think, ideally, a builder (Action really), would specify what parts of
>> the Environment() it's sensitive too.
>>
>>
>>
>>
>>
>>
>>
>> This would be better. If every builder specified which parts of
>> env['ENV'] it is interested in, and they (and nothing else) got passed
>> through and added to the build signature
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Remember, while many users are only building source code which is fairly
>> quick, SCons is used in other "build" systems where steps can take
>> considerable time and resources so extraneous rebuilds are quite costly for
>> them.
>>
>>
>>
>>
>>
>> -Bill
>>
>>
>>
>>
>>
>>
>>
>> I agree but specifying the correct dependencies are how you avoid costly
>> rebuilds. Not rebuilding something that has changed can be just as costly
>> because instead of finding out about it at the build stage, you find out
>> about it when it is on a production machine.
>>
>>
>>
>> _______________________________________________
>>
>>
>> Scons-users mailing list
>>
>>
>> Scons-users at scons.org
>>
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> Scons-users mailing list
>>
>> Scons-users at scons.org
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>> _______________________________________________
>>
>>
>> Scons-users mailing list
>>
>>
>> Scons-users at scons.org
>>
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> Scons-users mailing list
>>
>> Scons-users at scons.org
>>
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161229/e8db6ef4/attachment-0001.html>