[Scons-users] Is there a pure-python tar builder?

[Bill Deegan]

At some point we'll take a look at using multiprocessing instead of threads
for doing the builds called for by the DAG walk.. but sadly not today.


On Mon, Jun 26, 2017 at 4:42 PM, Daniel Holth <dholth at gmail.com> wrote:

> Python -m tar 😀
>
> On Mon, Jun 26, 2017, 18:53 Bill Deegan <bill at baddogconsulting.com> wrote:
>
>> Remember though, if a flow depends on creating many independent tarballs,
>> implementing it in python means that it will no longer run in parallel.
>>
>> I guess create as tar_python, and then we can poll the users if this
>> would impact them and should we merge/replace with existing tar tool.
>>
>> Thanks!
>>
>> On Mon, Jun 26, 2017 at 12:32 PM, Daniel Holth <dholth at gmail.com> wrote:
>>
>>> I would be satisfied with replacing the official tar builder with a pure
>>> Python one. The reward would be consistent cross platform behavior. It
>>> would be faster whenever the overhead of spawning a subprocess was greater
>>> than the speed difference between Python and native tar (probably true for
>>> smaller tar).
>>>
>>> It would also be possible to build a directory structure inside the
>>> .tar.gz without copying the files first as packaging does now, but that
>>> would probably require deeper changes to the packaging tool.
>>>
>>> On Mon, Jun 26, 2017 at 2:51 PM Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>>> Daniel,
>>>>
>>>> Would the function be a duplicate of the existing tar tool, just pure
>>>> python?
>>>> Or different function?
>>>>
>>>> You might take a look at the zip tool which checks for the existence of
>>>> the zip package and uses it if available, otherwise calls external zip.
>>>> (Likely that can get dropped now as SCons used to support python back
>>>> to 1.5.7 (I think)).
>>>>
>>>> It might make sense to modify the existing tar tool to do similar?
>>>>
>>>> -Bill
>>>>
>>>>
>>>>
>>>> On Mon, Jun 26, 2017 at 11:35 AM, Daniel Holth <dholth at gmail.com>
>>>> wrote:
>>>>
>>>>> I love reduced parallelism. Anything wrong with writing my own tar
>>>>> builder env = Environment(tools=['python_tar']) to overwrite the
>>>>> stock env['BUILDERS']['Tar'] with my own pure Python version? The Python
>>>>> tar builder will work on any platform and produce exactly the PAX format
>>>>> tar files I want with configurable permissions & file timestamps. Then
>>>>> hopefully it will work with the packaging tool which already does 95% of
>>>>> what I need. Either that or add my new type to SCons.Tool.packaging.__all__
>>>>>
>>>>> - Daniel
>>>>>
>>>>> On Mon, Jun 26, 2017 at 11:50 AM Bill Deegan <
>>>>> bill at baddogconsulting.com> wrote:
>>>>>
>>>>>> Daniel,
>>>>>>
>>>>>> See:
>>>>>> https://bitbucket.org/scons/scons/wiki/DistTarBuilder
>>>>>> and
>>>>>> https://bitbucket.org/scons/scons/wiki/ArchiveBuilder
>>>>>>
>>>>>> One note, using python code to implement the builder rather than
>>>>>> shelling out to run a command will make the builder subject to the GIL and
>>>>>> thus reduce parallelism.
>>>>>>
>>>>>> -Bill
>>>>>>
>>>>>> On Mon, Jun 26, 2017 at 8:21 AM, Daniel Holth <dholth at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello, I would like to generate .tar.gz archives cross platform and
>>>>>>> on Windows by using the Python tar module instead of the tar command,
>>>>>>> similar to how the zip builder uses the zipfile module. Has anyone
>>>>>>> developed such a builder?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Daniel
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20170626/d1af6878/attachment.html>