Comparison with isal

[danielhrisca]

Hello Ruben,

the latest beta release of zlib NG https://github.com/zlib-ng/zlib-ng/releases/tag/2.1.0-beta1 promises some big speed improvements

Is there a way to compare it against python-isal?

[rhpvorderman]

Hi @danielhrisca , sorry for my late reply. I hadn't yet turned on notifications for this repository. (I have now).

You could install the latest development version to see if it makes any difference. On my machine, isal still beats zlib-ng when reading, but the result is much more close. Writing isal easily wins, but it only has low compression levels. So zlib-ng is certainly worth considering when higher compression levels are desired.

[bdraco]

develop is a bit snapper. It shaves another few milliseconds off websocket response times with large payloads

[bdraco]

I opened #15 to update to 2.1.4

It would be nice to get a new release with it

[rhpvorderman]

The latest release now has the latest zlib-ng bundled.

[rhpvorderman]

@danielhrisca I have run some benchmarks, tables below. The conclusions:

• compression: isal requires up to 80% less runtime (compared to zlib), zlib-ng up to 30% less runtime. For compressing lots of data isal is the clear winner. No contest.
• Decompression, isal requires up to 55% less runtime. zlib-ng up to 40%. Isal is the clear winner.
• CRC: crc32 on par. zlib-ng has a much faster adler32 implementation. Though both libraries are much faster than zlib.
• Compression ratio. It is clear that by supporting a much broader range of compression options, zlib-ng is the clear winner here. Though for the higher compression ratios, other algorithms such as lzma become more interesting as well. zlib-ng has a level 1 strategy that hardly compresses at all. It is much better to use lz4 or zstd for such use cases.

Does this adequately answer your question?

Python-isal

CRC32
name    isal    zlib    ratio
0b      0.03    0.03    0.78
8b      0.03    0.04    0.78
128b    0.04    0.13    0.28
1kb     0.09    0.35    0.25
8kb     0.55    1.84    0.3
16kb    1.03    3.5     0.3
32kb    2.02    6.81    0.3
64kb    3.96    13.46   0.29
Adler32
name    isal    zlib    ratio
0b      0.1     0.04    2.55
8b      0.1     0.04    2.2
128b    0.11    0.07    1.52
1kb     0.13    0.29    0.45
8kb     0.39    2.09    0.19
16kb    0.67    4.17    0.16
32kb    1.28    8.14    0.16
64kb    2.35    16.1    0.15
zlib compression
name    isal    zlib    ratio
0b      1.89    1.82    1.04
8b      2.19    2.09    1.05
128b    3.18    3.93    0.81
1kb     5.74    10.55   0.54
8kb     21.32   56.38   0.38
16kb    36.09   144.42  0.25
32kb    68.57   320.67  0.21
64kb    139.58  669.22  0.21
zlib decompression
name    isal    zlib    ratio
0b      0.63    0.2     3.22
8b      0.7     0.25    2.82
128b    1.68    1.48    1.14
1kb     3.41    3.47    0.98
8kb     12.45   15.54   0.8
16kb    22.34   29.16   0.77
32kb    41.96   90.79   0.46
64kb    91.36   202.09  0.45
gzip compression
name    isal    zlib    ratio
0b      2.3     2.49    0.92
8b      2.67    2.75    0.97
128b    3.84    4.79    0.8
1kb     6.52    11.52   0.57
8kb     21.28   56.96   0.37
16kb    37.89   144.95  0.26
32kb    71.17   325.47  0.22
64kb    147.7   678.43  0.22
gzip decompression
name    isal    zlib    ratio
0b      1.15    1.49    0.77
8b      1.2     1.62    0.74
128b    2.17    2.89    0.75
1kb     4.03    5.07    0.79
8kb     13.22   19.13   0.69
16kb    23.03   34.04   0.68
32kb    42.67   87.37   0.49
64kb    92.43   209.87  0.44
zlib Compress instantiation
name    isal    zlib    ratio
        0.12    0.91    0.13
zlib Decompress instantiation
name    isal    zlib    ratio
        0.09    0.09    0.99
Gzip Writer instantiation
name    isal    zlib    ratio
        7.19    5.16    1.39
Gzip Reader instantiation
name    isal    zlib    ratio
        4.16    2.75    1.51
zlib sizes
name    -1      0       1       2       3       4       5       6       7       8       9
0b      8.0     11.0    8.0     8.0     8.0     8.0     8.0     8.0     8.0     8.0     8.0
8b      2.0     2.375   2.0     2.0     2.0     2.0     2.0     2.0     2.0     2.0     2.0
128b    0.672   1.086   0.695   0.695   0.695   0.68    0.672   0.672   0.672   0.672   0.672
1kb     0.522   1.011   0.535   0.533   0.529   0.524   0.522   0.522   0.522   0.522   0.522
8kb     0.488   1.001   0.515   0.509   0.503   0.493   0.49    0.488   0.489   0.489   0.489
16kb    0.482   1.001   0.517   0.51    0.503   0.489   0.483   0.482   0.482   0.482   0.482
32kb    0.48    1.0     0.522   0.514   0.505   0.488   0.482   0.48    0.48    0.48    0.48
64kb    0.476   1.0     0.519   0.512   0.503   0.486   0.48    0.476   0.474   0.474   0.474
isal sizes
name    0       1       2       3
0b      119.0   8.0     8.0     8.0
8b      15.875  2.0     2.0     2.0
128b    1.695   0.703   0.703   0.68
1kb     0.886   0.531   0.531   0.532
8kb     0.721   0.492   0.493   0.489
16kb    0.692   0.488   0.488   0.483
32kb    0.67    0.489   0.487   0.483
64kb    0.657   0.487   0.485   0.483

python-zlib-ng

CRC32
name    zlib_ng zlib    ratio
0b      0.03    0.03    0.86
8b      0.04    0.04    0.81
128b    0.05    0.13    0.37
1kb     0.1     0.34    0.29
8kb     0.56    1.82    0.31
16kb    1.04    3.47    0.3
32kb    2.02    6.74    0.3
64kb    3.96    13.33   0.3
Adler32
name    zlib_ng zlib    ratio
0b      0.03    0.04    0.74
8b      0.03    0.04    0.73
128b    0.03    0.07    0.5
1kb     0.05    0.29    0.17
8kb     0.2     2.05    0.1
16kb    0.35    4.04    0.09
32kb    0.65    8.03    0.08
64kb    1.21    16.03   0.08
zlib compression
name    zlib_ng zlib    ratio
0b      2.9     1.8     1.61
8b      3.46    2.02    1.71
128b    5.33    3.93    1.36
1kb     12.13   10.76   1.13
8kb     57.73   58.52   0.99
16kb    117.09  157.31  0.74
32kb    259.22  344.15  0.75
64kb    566.55  802.19  0.71
zlib decompression
name    zlib_ng zlib    ratio
0b      0.22    0.2     1.1
8b      0.29    0.25    1.18
128b    1.58    1.45    1.08
1kb     3.45    3.58    0.96
8kb     15.8    17.64   0.9
16kb    27.75   29.93   0.93
32kb    52.47   91.23   0.58
64kb    152.34  198.83  0.77
gzip compression
name    zlib_ng zlib    ratio
0b      3.34    2.47    1.35
8b      4.26    2.75    1.55
128b    6.23    4.76    1.31
1kb     12.92   11.55   1.12
8kb     59.68   59.47   1.0
16kb    129.3   159.53  0.81
32kb    263.81  346.7   0.76
64kb    596.29  761.75  0.78
gzip decompression
name    zlib_ng zlib    ratio
0b      1.11    1.53    0.73
8b      1.27    1.63    0.78
128b    2.61    3.04    0.86
1kb     4.74    5.16    0.92
8kb     15.85   19.98   0.79
16kb    28.41   35.22   0.81
32kb    53.98   92.16   0.59
64kb    131.31  206.19  0.64
zlib Compress instantiation
name    zlib_ng zlib    ratio
        11.56   0.91    12.66
zlib Decompress instantiation
name    zlib_ng zlib    ratio
        0.12    0.08    1.51
Gzip Writer instantiation
name    zlib_ng zlib    ratio
        40.53   9.02    4.5
Gzip Reader instantiation
name    zlib_ng zlib    ratio
        4.22    2.74    1.54
zlib sizes
name    -1      0       1       2       3       4       5       6       7       8       9
0b      8.0     11.0    8.0     8.0     8.0     8.0     8.0     8.0     8.0     8.0     8.0
8b      2.0     2.375   2.0     2.0     2.0     2.0     2.0     2.0     2.0     2.0     2.0
128b    0.672   1.086   0.695   0.695   0.695   0.68    0.672   0.672   0.672   0.672   0.672
1kb     0.522   1.011   0.535   0.533   0.529   0.524   0.522   0.522   0.522   0.522   0.522
8kb     0.488   1.001   0.515   0.509   0.503   0.493   0.49    0.488   0.489   0.489   0.489
16kb    0.482   1.001   0.517   0.51    0.503   0.489   0.483   0.482   0.482   0.482   0.482
32kb    0.48    1.0     0.522   0.514   0.505   0.488   0.482   0.48    0.48    0.48    0.48
64kb    0.476   1.0     0.519   0.512   0.503   0.486   0.48    0.476   0.474   0.474   0.474
zlib_ng sizes
name    -1      0       1       2       3       4       5       6       7       8       9
0b      8.0     11.0    8.0     8.0     8.0     8.0     8.0     8.0     8.0     8.0     8.0
8b      2.0     2.375   2.0     2.0     2.0     2.0     2.0     2.0     2.0     2.0     2.0
128b    0.68    1.086   0.805   0.688   0.68    0.68    0.68    0.68    0.656   0.656   0.672
1kb     0.522   1.011   0.785   0.53    0.522   0.522   0.522   0.522   0.52    0.52    0.522
8kb     0.476   1.001   0.745   0.489   0.481   0.478   0.476   0.476   0.472   0.472   0.492
16kb    0.47    1.001   0.734   0.486   0.476   0.471   0.471   0.47    0.466   0.466   0.489
32kb    0.467   1.0     0.722   0.487   0.476   0.469   0.468   0.467   0.464   0.463   0.488
64kb    0.461   1.0     0.715   0.487   0.474   0.466   0.464   0.461   0.457   0.457   0.482

[rhpvorderman]

I will close this issue as I am cleaning up all issues that do not require further action. Feel free to reopen when something new comes up!