[Scons-users] Custom Scanner question

Fri Oct 3 00:40:29 EDT 2014

>
> I should provide a little more background:
> - Yes, you run a single command ("Rhapsody") to generate a series of C++
> header/source files.
> - Therefore there is a many-to-many dependency: many input files -> a
> single command -> many output files
> - Each generated C++ source then needs to be compiled into an object file
> - All object files then need to be linked into a single shared library

I have written a python script that is able to parse the top-level Rhapsody
> file and produce:
> - a list of all input files referenced by the top-level file (i.e. what
> the Scanner would do)
> - a list of all output files* that should be generated when the "Rhapsody"
> command is executed
> * = my script isn't perfect. It sometimes prints a list of output files
> that is a superset of what Rhapsody will actually generate. It's difficult
> to perfectly understand the logic of Rhapsody and why it does or does not
> generate certain classes that are contained within the project.  Because
> it's not perfect, I don't know if it's useful.
> Given that, should I write a custom Scanner, a custom Emitter, or both?
> Or give up trying to predict the output and use Gary's approach to add
> targets dynamically (i.e. after-the-fact)?
> One concern I have is start-up cost for SCons.  If the Rhapsody project
> and its sub-files have not been changed, I would like SCons to be able to
> skip parsing the Rhapsody files.  My impression is that Scanners are
> already optimized this way: if the MD5 checksum of the file hasn't changed,
> the Scanner is not re-run and the cached implicit dependencies are used.
> Another possible tool: Rhapsody can be configured to generate a "Makefile"
> which is really any type of file I want. Rhapsody will expand certain
> internal variables (i.e. list of output files) and write them to the
> Makefile.  But this of course only happens -after- executing "Rhapsody".

I agree with Gary on this.  Anytime you have a 1:N or N:M relationship,
especially when you cannot accurately predict the outputs, its better to do
a dynamic system. Scanner/emitter combos cannot accurately predict the
outputs, since that is dependent on your code generator and possibly the
versions of the generator.  I don't think this is a battle (long or short
term) that you want to fight.  If nothing else, you get into performance
problems due to the nature of the process in O( N * M ).  The SCons Java
toolchain is an example of this.  It really needs to be revamped to be
dynamic.

SCons doesn't scan unless files have been updated.  There are several times
of up-to-date checking which include md5sum and timestamp.

V/R,
William

On Wed, Oct 1, 2014 at 12:43 PM, Jason Fritz <jasonfritzpublic at gmail.com>
wrote:

> Emails crossed... attempting to combine Dirk and Gary's responses into one
> thread...
>
> On Tue, Sep 30, 2014 at 11:24 AM, Dirk Bächle <tshortik at gmx.de> wrote:
>
>> Hi Jason,
>>
>> Scanners usually detect "implicit dependencies", meaning you have
>> specified a build step "source->target" where source depends on other files
>> (like includes in C/C++, for example). So whenever a source file of type
>> "x" (in your case "*.rpy") is encountered, you want to automatically scan
>> the file for "includes" and add them as dependencies to the target (!) as
>> well.
>>
>> If you have just one single Action, where you call a command-line tool
>> like "rpy-compile foo.rpy", it may be enough to define a custom Emitter
>> instead. Within the Emitter you'd parse the *.rpy project file too, and
>> directly add the referenced "*.cpp/*.h" files as sources and a possible
>> "*.exe" as target.
>>
>> You can read a bit more about Scanners in this recent post to the mailing
>> list:
>>
>> https://pairlist4.pair.net/pipermail/scons-users/2014-
>> September/002977.html
>>
>> For Emitters you might want to take a look at
>>
>>   http://www.scons.org/wiki/ToolsForFools
>>
>> , and if you get stuck, just come back with more questions. ;)
>>
>> Best regards,
>>
>> Dirk
>>
>
> On Tue, Sep 30, 2014 at 12:21 PM, Gary Oberbrunner <garyo at oberbrunner.com>
> wrote:
>
>>
>> A Scanner won't do enough for you here.  Your Rhapsody is a source
>> code generator, in particular the hard kind where you can't predict
>> which sources will be generated without generating them.  I have a
>> writeup on the wiki for one way to deal with this kind of thing:
>>  http://www.scons.org/wiki/DynamicSourceGenerator
>>
>> There's some more info at
>> http://www.scons.org/wiki/NonDeterministicDependencies as well.  These
>> techniques depend on reading a (generated) file with a list of all the
>> generated sources, but you could probably get that by parsing the
>> Rhapsody output or something.
>>
>> --
>> Gary
>>
>>
> Hi Dirk and Gary,
>
> First of all, thanks a lot for your help!
>
> I should provide a little more background:
> - Yes, you run a single command ("Rhapsody") to generate a series of C++
> header/source files.
> - Therefore there is a many-to-many dependency: many input files -> a
> single command -> many output files
> - Each generated C++ source then needs to be compiled into an object file
> - All object files then need to be linked into a single shared library
>
> I have written a python script that is able to parse the top-level
> Rhapsody file and produce:
> - a list of all input files referenced by the top-level file (i.e. what
> the Scanner would do)
> - a list of all output files* that should be generated when the "Rhapsody"
> command is executed
>
> * = my script isn't perfect. It sometimes prints a list of output files
> that is a superset of what Rhapsody will actually generate. It's difficult
> to perfectly understand the logic of Rhapsody and why it does or does not
> generate certain classes that are contained within the project.  Because
> it's not perfect, I don't know if it's useful.
>
> Given that, should I write a custom Scanner, a custom Emitter, or both?
> Or give up trying to predict the output and use Gary's approach to add
> targets dynamically (i.e. after-the-fact)?
>
> One concern I have is start-up cost for SCons.  If the Rhapsody project
> and its sub-files have not been changed, I would like SCons to be able to
> skip parsing the Rhapsody files.  My impression is that Scanners are
> already optimized this way: if the MD5 checksum of the file hasn't changed,
> the Scanner is not re-run and the cached implicit dependencies are used.
>
> Another possible tool: Rhapsody can be configured to generate a "Makefile"
> which is really any type of file I want. Rhapsody will expand certain
> internal variables (i.e. list of output files) and write them to the
> Makefile.  But this of course only happens -after- executing "Rhapsody".
>
> Thanks!!!
> Jason
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20141003/d5bcdf69/attachment-0001.html>