[ptweezy]

After decompress(data, max_length) returns max_length bytes, needs_input can be True although the next decompress(b"", max_length) call returns more data. The docstring of decompress() says the opposite:

If this limit is reached and further output can be produced, self.needs_input will be set to False.

When the output buffer is full, isal_inflate() keeps decoding into its own buffer, so it can read all of the input before the output is returned. decompress() sets needs_input whenever no input is left, like CPython's BZ2Decompressor, which it is based on.

Readers that stop at the end of the file when needs_input is True fail on valid data. CPython's DecompressReader, which bz2 and lzma use for files, raises EOFError: Compressed file ended before the end-of-stream marker was reached. It does so for 1,000 bytes of data read with read(100), and for 100,000 bytes read with read(8192). zlib._ZlibDecompressor reads the same streams.

This PR sets needs_input only when the output buffer was not full, as CPython's LZMADecompressor does. When the output buffer was full and no output is left, the next decompress() call returns b"" and sets needs_input.

The new test fails without the fix. testDecompressorChunksMaxsize could not catch this. Like the CPython test_bz2 test it comes from, it passes all but 64 bytes of the stream in the first call, which is more input than ISA-L reads while the output is full.

Checklist

• Pull request details were added to CHANGELOG.rst
• Documentation was updated (if needed)

[rhpvorderman]

Excellent. Thanks. The igzip_lib decompressor was initially made for the python _GzipReader object, but has been superseded by the C _GzipReader.