[Scons-users] env[X'] and signatures

Thu Dec 29 17:51:41 EST 2016

Well it's pretty complex as I am sure you can imagine, you have all sorts
of files. Verilos systemverilog, vhdl, constraints, headers, vendor files
like xci, qip, etc.

However lets say I have an fpga build flow split in 4 phases.
  1. Synthesis
  2. Place & Route
  3. Tagging. Gets the bitstream from place and route and modifies ONLY the
contents of a block ram, adding version, timestamp, build information
strings, etc. This is to go from an unofficial development build (untested)
to an officially tagged and tested release.
  4. Configuration image generarion. Get one (or multiple) bitstreams along
with some hex files for example, some parameters, and generate an SPI
configuration file to load onto the FPGA.

Now, each of these targets depends on the previous and some extra stuff
(like hex files and parameters), the synthesis target depends on the hdl
sources intself. Both synthesis and place and route can add some extra
sources, constraints for instance.

Lets say I have already built the whole process, so I have the output files
and they are up-to-date. Now lets say I just want to go to the tagging
target and modify some tagging information, like the version. I go and open
the sconscript which defines that target and modify that parameter. However
that file also defines all other targets, so if my targets depend on that
sconscript they are all out-dated now!

However the only target that has trully changed is the tagging target (and
therefore the next and last stage too), what I miss about scons is a way of
each builder detecting if it's environment has changed since the last build.

What I have done to fix this is have scons hash the whole environment for
each target and add that hash to the builders signature. This allows me to
remove the sconscripts from the target's dependencies, and still have the
target rebuilt when a prameter is changed. At least In our case changes in
a target's environment are much much more unlikely than the scenario I just
described, so this aproach means a huge improvement for us.

As I have mentioned in the previous message I would like to be able to use
dill instead off pickle to serialise.

Albert.

El dj, 29 des 2016 a les 23:32 Bill Deegan <bill at baddogconsulting.com> va
escriure:

> Albert,
>
> I'm very familiar with the EDA space.
> What's the input file?
> What environmnent variables changing would trigger a rebuild for you?
>
> -Bill
>
> On Thu, Dec 29, 2016 at 9:58 AM, Albert Arquer <albert.arquer at gmail.com>
> wrote:
>
> I agree it is not the ideal solution. In fact we use scons for hdl
> synthesis/implementation which can take many hours.
>
> Before we used to make the target depend on the sconscript which declared
> the target. However this caused many innecessary rebuilds, as a single
> sconscript often declares multiple targets, and even if a change in the
> sconscript which only afected a single target was modified, scons would
> rebuild all other targets declared in the sconscript and any others which
> depended on them.
>
> By hashing the environment we can isolate sconscript modifications to the
> targets it actually affects. This is not ideal ofcourse, it would be much
> better for each builder to define somehow which items from the environment
> it will be using. However this problem usually causes build to be repeated,
> not missed, and still counting those cases the innecessary builts are
> drastically reduced (at least in our case).
>
>  There is also the issue on the hashing itself, since not everything is
> hashable in python, and even if an object is hashable that hash might not
> change if the object's class definition has changed for example. I opened
> this thread in stackoverflow a while back trying to find a serialization
> module suited for this purpose, turns out dill would be a good choice but
> it has compatibility issues with scons.
>
>
> http://stackoverflow.com/questions/40640797/compute-class-and-instance-hash/40775525#40775525
>
>
> El dj, 29 des 2016 a les 18:51 Bill Deegan <bill at baddogconsulting.com> va
> escriure:
>
> As a workaround, perhaps add an action string which does nothing, but uses
> the variables you depend on?
> Just a thought.
>
>
> Are the builders external python scripts which get run from the shell by
> SCons? or are the builders python logic run inside scons?
>
> On Thu, Dec 29, 2016 at 8:54 AM, Tom Tanner <trtanner at btinternet.com>
> wrote:
>
> On 27/12/16 18:22, Bill Deegan wrote:
>
>
>
>
> If every builder depends on all values in env['ENV'], then any change will
> rebuild everything.
>
>
> This is almost never warranted.
>
>
>
>
>
>
>
> But then why does scons clear out the supplied environment? It is
> precisely because everything behaves in a way that is unclear and not
> reproducable if you don't know exactly what the environment is.
>
>
>
>
> I think, ideally, a builder (Action really), would specify what parts of
> the Environment() it's sensitive too.
>
>
>
>
>
>
>
> This would be better. If every builder specified which parts of env['ENV']
> it is interested in, and they (and nothing else) got passed through and
> added to the build signature
>
>
>
>
>
>
>
>
>
>
> Remember, while many users are only building source code which is fairly
> quick, SCons is used in other "build" systems where steps can take
> considerable time and resources so extraneous rebuilds are quite costly for
> them.
>
>
>
>
>
> -Bill
>
>
>
>
>
>
>
> I agree but specifying the correct dependencies are how you avoid costly
> rebuilds. Not rebuilding something that has changed can be just as costly
> because instead of finding out about it at the build stage, you find out
> about it when it is on a production machine.
>
>
>
> _______________________________________________
>
>
> Scons-users mailing list
>
>
> Scons-users at scons.org
>
>
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
>
> _______________________________________________
>
> Scons-users mailing list
>
> Scons-users at scons.org
>
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
> _______________________________________________
>
>
> Scons-users mailing list
>
>
> Scons-users at scons.org
>
>
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
>
>
> _______________________________________________
>
> Scons-users mailing list
>
> Scons-users at scons.org
>
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20161229/f6fa3ac7/attachment-0001.html>