[Scons-users] Is there a pure-python tar builder?

[Daniel Holth]

When I looked at SCons performance I thought the faster os.walk (avoiding
separate stat calls) and caching variable substitution would be the
clearest wins.

On Mon, Jun 26, 2017, 20:19 Bill Deegan <bill at baddogconsulting.com> wrote:

> At some point we'll take a look at using multiprocessing instead of
> threads for doing the builds called for by the DAG walk.. but sadly not
> today.
>
>
> On Mon, Jun 26, 2017 at 4:42 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>> Python -m tar 😀
>>
>> On Mon, Jun 26, 2017, 18:53 Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>> Remember though, if a flow depends on creating many independent
>>> tarballs, implementing it in python means that it will no longer run in
>>> parallel.
>>>
>>> I guess create as tar_python, and then we can poll the users if this
>>> would impact them and should we merge/replace with existing tar tool.
>>>
>>> Thanks!
>>>
>>> On Mon, Jun 26, 2017 at 12:32 PM, Daniel Holth <dholth at gmail.com> wrote:
>>>
>>>> I would be satisfied with replacing the official tar builder with a
>>>> pure Python one. The reward would be consistent cross platform behavior. It
>>>> would be faster whenever the overhead of spawning a subprocess was greater
>>>> than the speed difference between Python and native tar (probably true for
>>>> smaller tar).
>>>>
>>>> It would also be possible to build a directory structure inside the
>>>> .tar.gz without copying the files first as packaging does now, but that
>>>> would probably require deeper changes to the packaging tool.
>>>>
>>>> On Mon, Jun 26, 2017 at 2:51 PM Bill Deegan <bill at baddogconsulting.com>
>>>> wrote:
>>>>
>>>>> Daniel,
>>>>>
>>>>> Would the function be a duplicate of the existing tar tool, just pure
>>>>> python?
>>>>> Or different function?
>>>>>
>>>>> You might take a look at the zip tool which checks for the existence
>>>>> of the zip package and uses it if available, otherwise calls external zip.
>>>>> (Likely that can get dropped now as SCons used to support python back
>>>>> to 1.5.7 (I think)).
>>>>>
>>>>> It might make sense to modify the existing tar tool to do similar?
>>>>>
>>>>> -Bill
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 26, 2017 at 11:35 AM, Daniel Holth <dholth at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I love reduced parallelism. Anything wrong with writing my own tar
>>>>>> builder env = Environment(tools=['python_tar']) to overwrite the
>>>>>> stock env['BUILDERS']['Tar'] with my own pure Python version? The Python
>>>>>> tar builder will work on any platform and produce exactly the PAX format
>>>>>> tar files I want with configurable permissions & file timestamps. Then
>>>>>> hopefully it will work with the packaging tool which already does 95% of
>>>>>> what I need. Either that or add my new type to SCons.Tool.packaging.__all__
>>>>>>
>>>>>> - Daniel
>>>>>>
>>>>>> On Mon, Jun 26, 2017 at 11:50 AM Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> Daniel,
>>>>>>>
>>>>>>> See:
>>>>>>> https://bitbucket.org/scons/scons/wiki/DistTarBuilder
>>>>>>> and
>>>>>>> https://bitbucket.org/scons/scons/wiki/ArchiveBuilder
>>>>>>>
>>>>>>> One note, using python code to implement the builder rather than
>>>>>>> shelling out to run a command will make the builder subject to the GIL and
>>>>>>> thus reduce parallelism.
>>>>>>>
>>>>>>> -Bill
>>>>>>>
>>>>>>> On Mon, Jun 26, 2017 at 8:21 AM, Daniel Holth <dholth at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello, I would like to generate .tar.gz archives cross platform and
>>>>>>>> on Windows by using the Python tar module instead of the tar command,
>>>>>>>> similar to how the zip builder uses the zipfile module. Has anyone
>>>>>>>> developed such a builder?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170627/235570a9/attachment-0001.html>