[Scons-users] Is there a pure-python tar builder?

[Bill Deegan]

Working on subst caching now.. we'll see how it pans out.
It's been tried before, but the efforts weren't completed.

-Bill

On Mon, Jun 26, 2017 at 5:57 PM, Daniel Holth <dholth at gmail.com> wrote:

> When I looked at SCons performance I thought the faster os.walk (avoiding
> separate stat calls) and caching variable substitution would be the
> clearest wins.
>
> On Mon, Jun 26, 2017, 20:19 Bill Deegan <bill at baddogconsulting.com> wrote:
>
>> At some point we'll take a look at using multiprocessing instead of
>> threads for doing the builds called for by the DAG walk.. but sadly not
>> today.
>>
>>
>> On Mon, Jun 26, 2017 at 4:42 PM, Daniel Holth <dholth at gmail.com> wrote:
>>
>>> Python -m tar 😀
>>>
>>> On Mon, Jun 26, 2017, 18:53 Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>>> Remember though, if a flow depends on creating many independent
>>>> tarballs, implementing it in python means that it will no longer run in
>>>> parallel.
>>>>
>>>> I guess create as tar_python, and then we can poll the users if this
>>>> would impact them and should we merge/replace with existing tar tool.
>>>>
>>>> Thanks!
>>>>
>>>> On Mon, Jun 26, 2017 at 12:32 PM, Daniel Holth <dholth at gmail.com>
>>>> wrote:
>>>>
>>>>> I would be satisfied with replacing the official tar builder with a
>>>>> pure Python one. The reward would be consistent cross platform behavior. It
>>>>> would be faster whenever the overhead of spawning a subprocess was greater
>>>>> than the speed difference between Python and native tar (probably true for
>>>>> smaller tar).
>>>>>
>>>>> It would also be possible to build a directory structure inside the
>>>>> .tar.gz without copying the files first as packaging does now, but that
>>>>> would probably require deeper changes to the packaging tool.
>>>>>
>>>>> On Mon, Jun 26, 2017 at 2:51 PM Bill Deegan <bill at baddogconsulting.com>
>>>>> wrote:
>>>>>
>>>>>> Daniel,
>>>>>>
>>>>>> Would the function be a duplicate of the existing tar tool, just pure
>>>>>> python?
>>>>>> Or different function?
>>>>>>
>>>>>> You might take a look at the zip tool which checks for the existence
>>>>>> of the zip package and uses it if available, otherwise calls external zip.
>>>>>> (Likely that can get dropped now as SCons used to support python back
>>>>>> to 1.5.7 (I think)).
>>>>>>
>>>>>> It might make sense to modify the existing tar tool to do similar?
>>>>>>
>>>>>> -Bill
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 26, 2017 at 11:35 AM, Daniel Holth <dholth at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I love reduced parallelism. Anything wrong with writing my own tar
>>>>>>> builder env = Environment(tools=['python_tar']) to overwrite the
>>>>>>> stock env['BUILDERS']['Tar'] with my own pure Python version? The Python
>>>>>>> tar builder will work on any platform and produce exactly the PAX format
>>>>>>> tar files I want with configurable permissions & file timestamps. Then
>>>>>>> hopefully it will work with the packaging tool which already does 95% of
>>>>>>> what I need. Either that or add my new type to SCons.Tool.packaging.__all__
>>>>>>>
>>>>>>> - Daniel
>>>>>>>
>>>>>>> On Mon, Jun 26, 2017 at 11:50 AM Bill Deegan <
>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>
>>>>>>>> Daniel,
>>>>>>>>
>>>>>>>> See:
>>>>>>>> https://bitbucket.org/scons/scons/wiki/DistTarBuilder
>>>>>>>> and
>>>>>>>> https://bitbucket.org/scons/scons/wiki/ArchiveBuilder
>>>>>>>>
>>>>>>>> One note, using python code to implement the builder rather than
>>>>>>>> shelling out to run a command will make the builder subject to the GIL and
>>>>>>>> thus reduce parallelism.
>>>>>>>>
>>>>>>>> -Bill
>>>>>>>>
>>>>>>>> On Mon, Jun 26, 2017 at 8:21 AM, Daniel Holth <dholth at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello, I would like to generate .tar.gz archives cross platform
>>>>>>>>> and on Windows by using the Python tar module instead of the tar command,
>>>>>>>>> similar to how the zip builder uses the zipfile module. Has anyone
>>>>>>>>> developed such a builder?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Daniel
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> Scons-users at scons.org
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170626/dc01a557/attachment.html>