Repository navigation
tarfile in stream mode always set zlib compression level to 9 #70441
Description
Activity
PatrikDufresne commented
on Feb 1, 2016 PatrikDufresnemannequinMannequinAuthorWhen using tarfile.open(mode='w|gz'), the compression level is hard-coded to 9. Seed _Stream._init_write_gz():
self.zlib.compressobj(9,-
In regards to zlib, I would start by replacing the value of 9 by zlib.Z_DEFAULT_COMPRESSION. This is the default value and zipfile is using it. Why using something different.
-
Then, I would also love to control the compression level when calling tarfile.open()
-
It looks like the default has been hard-coded to 9 ever since tarfile was added to Python. The gzip module is also hard-coded to 9 since it was added. If tarfile is changed, maybe gzip should too.
Why would you want to use zlib’s default (apparently 6)? Memory usage or speed perhaps? If we do change the default, maybe it is best to only do it in 3.6. I don’t see it as a bug fix, and there is a chance it could break someone’s code.
To be able to control the compression level, perhaps you can already do it by wrapping the tar stream with GzipFile (untested):
gz_writer = GzipFile(fileobj=raw_writer, mode="wb", compresslevel=...) tar_writer = tarfile.open(fileobj=gz_writer, mode="w|") tar_writer.addfile(...) tar_writer.close() gz_writer.close()
If the default is changed, it certainly makes sense to add an easy compression level parameter, to be able to restore the old behaviour.
Actually it’s not really obvious from the signatures, but in the middle of the tarfile.open() documentation it says “. . . tarfile.open() accepts the keyword argument _compresslevel_”, so it should already be possible.
*compresslevel* takes effect for modes 'w:gz', 'r:gz', 'w:bz2', 'r:bz2', 'x:gz', 'x:bz2'. For stream modes, 'r|gz', 'w|gz', 'r|bz2', 'w|bz2', the *compresslevel* doesn't make sense. It seems not hard to make it possible but I'm not sure it's worth it or there is any reason it's hard-coded.
wimglenn commented
on Jun 28, 2017 wimglennmannequinMannequinThis issue also got me. compresslevel kwarg works fine for tarfile.open(..., mode='w:gz') but raises exception for tarfile.open(..., mode='w|gz')
I want to use stream compression, and compresslevel=1 is more than enough for my use case, the default of 9 is way too slow.
jarondl commented
on Aug 1, 2017 jarondlmannequinMannequinI have submitted a PR on GitHub #2962
The PR broke tests on systems without the optional bz2 module.
====================================================================== ERROR: test_wrong_compresslevels (test.test_tarfile.CompressLevelRaises.test_wrong_compresslevels) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Lib/test/test_tarfile.py", line 1620, in test_wrong_compresslevels tarfile.open(tmpname, "w:bz2", fobj, compresslevel=0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Lib/tarfile.py", line 1654, in open return func(name, filemode, fileobj, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Lib/tarfile.py", line 1732, in bz2open raise CompressionError("bz2 module is not available") from None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tarfile.CompressionError: bz2 module is not available ----------------------------------------------------------------------- added a commit that references this issue
on Jun 26, 2022 Thanks, looks like this has been completed and the test has been fixed
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: