Big performance difference between `python-zstd` and `zstd-rs` speeds

[scrottty]

Hey! I'm getting a 6x processing speed difference between the python package python-zstd and this package when compressing a single image that is ~13MB in size. The python variant compresses in ~0.38 seconds whilst rust is taking ~2.4 seonds. This same issue occurs when comparing the zstd-cli tool with rust

We have tried to implement this package in all of the different ways we can. And we have also tried to match the python implementation as close as possible but are continuing to get the same result. Both are using the same version of zstd under the hood.

Is there any chance you can see where we have gone wrong to be getting such a big performance lose compared to python and the cli? Please let me know if you need any other information!

Here are our testing scripts and results. Our test device is an embedded system running Yocto linux.

Rust

main.rs:

use std::{fs, path::Path, time::Instant};

use zstd::zstd_safe::CCtx;
use zstd::zstd_safe::CParameter;

fn main() {
    let img_path = Path::new("./img_raw");
    let img_data = fs::read(img_path).unwrap();

    println!("Input size: {:?} bytes", img_data.len());

    for _ in 0..=20 {
        let t0 = Instant::now();
        let mut cctx = CCtx::create();
        cctx.init(1).unwrap();
        let cparam = CParameter::NbWorkers(1);
        cctx.set_parameter(cparam).unwrap();

        let mut output = Vec::new();
        output.reserve(img_data.len());
        let data_size = cctx.compress2(&mut output, &img_data).unwrap();
        println!(
            "Compression time: {:?} | Compression size: {:?}",
            t0.elapsed(),
            data_size
        );
    }
}

cargo.toml:

[package]
name = "zstd_test"
version = "0.1.0"

[dependencies]
zstd = { version = "0.13.2", features = ["zstdmt"] }

console output

sh-5.2# ./zstd_test
Input size: 13120000 bytes
Compression time: 2.424943091s | Compression size: 4141879
Compression time: 2.425405005s | Compression size: 4141879
Compression time: 2.420405395s | Compression size: 4141879
Compression time: 2.418717218s | Compression size: 4141879
Compression time: 2.419599167s | Compression size: 4141879
Compression time: 2.418817959s | Compression size: 4141879
Compression time: 2.419841419s | Compression size: 4141879
Compression time: 2.419766838s | Compression size: 4141879
Compression time: 2.423758679s | Compression size: 4141879
Compression time: 2.419085892s | Compression size: 4141879
Compression time: 2.423615787s | Compression size: 4141879
Compression time: 2.428020042s | Compression size: 4141879
Compression time: 2.419602857s | Compression size: 4141879
Compression time: 2.420533826s | Compression size: 4141879
Compression time: 2.421025891s | Compression size: 4141879
Compression time: 2.423241083s | Compression size: 4141879
Compression time: 2.420323494s | Compression size: 4141879
Compression time: 2.421580927s | Compression size: 4141879
Compression time: 2.424723099s | Compression size: 4141879
Compression time: 2.424607418s | Compression size: 4141879
Compression time: 2.420573007s | Compression size: 4141879

Python

zstd_tests.py

import zstd
import time

img_raw = open("./img_raw", "rb").read()

print(f"Input size: {len(img_raw)} bytes")

for i in range(20):
    t0 = time.time()
    comp = zstd.compress(img_raw, 1, 1)
    print(f"Compression time: {time.time() - t0}s | Compression size: {len(comp)}")

python packages

sh-5.2# python3 -m pip list
Package               Version
--------------------- ------------
...
zstd                  1.5.2.1

sh-5.2# python3 --version
Python 3.12.4

console output

sh-5.2# python3 zstd_tests.py
Input size: 13120000 bytes
Compression time: 0.3925755023956299s | Compression size: 4141879
Compression time: 0.3913264274597168s | Compression size: 4141879
Compression time: 0.38705897331237793s | Compression size: 4141879
Compression time: 0.38625478744506836s | Compression size: 4141879
Compression time: 0.38326406478881836s | Compression size: 4141879
Compression time: 0.3831746578216553s | Compression size: 4141879
Compression time: 0.3832221031188965s | Compression size: 4141879
Compression time: 0.38354063034057617s | Compression size: 4141879
Compression time: 0.3833439350128174s | Compression size: 4141879
Compression time: 0.383317232131958s | Compression size: 4141879
Compression time: 0.38402223587036133s | Compression size: 4141879
Compression time: 0.38319969177246094s | Compression size: 4141879
Compression time: 0.3832590579986572s | Compression size: 4141879
Compression time: 0.3833591938018799s | Compression size: 4141879
Compression time: 0.38364577293395996s | Compression size: 4141879
Compression time: 0.38327574729919434s | Compression size: 4141879
Compression time: 0.3832426071166992s | Compression size: 4141879
Compression time: 0.38323450088500977s | Compression size: 4141879
Compression time: 0.383345365524292s | Compression size: 4141879
Compression time: 0.3834688663482666s | Compression size: 4141879

zstd_cli

For good measure I also ran the benchmark on the cli tool
console output

sh-5.2# zstd -b1 ./img_raw -o cli_test_output
 1#img_raw           :  13120000 ->   4143404 (x3.166),   35.2 MB/s,  104.3 MB/s

35.2 MB/s for a 13.12MB image comes to ~0.37s. This is in line with the python results

[NobodyXu]

Perhaps using lto = "fat" and --release would help?

IIRC since the code is split among zstd-sys, zstd-safe and zstd, there might be some missed optimization that can be claimed back by using fat LTO across crates (rust defaults to thin LTO within crate).

[NobodyXu]

It could be the memory allocator.

IIRC python 3.11 supports optional mimalloc support, python/cpython#90815, it could be that the python version you use uses mimalloc.

Compared to the glibc or musl malloc, mimalloc is a lot more efficient.

[scrottty]

Thanks for your replies!

Perhaps using lto = "fat" and --release would help?

IIRC since the code is split among zstd-sys, zstd-safe and zstd, there might be some missed optimization that can be claimed back by using fat LTO across crates (rust defaults to thin LTO within crate).

I did just test using only the lower level zstd-sys commands and got the same slow results. Does this answer this as its only using one of the crates? If not i can give compiling it like this ago

It could be the memory allocator.

IIRC python 3.11 supports optional mimalloc support, python/cpython#90815, it could be that the python version you use uses mimalloc.

Compared to the glibc or musl malloc, mimalloc is a lot more efficient.

I am using Python3.12 which appears to use mimalloc as you said. If it is indeed the memory allocator is there any remedy for it?

[NobodyXu]

If it is indeed the memory allocator is there any remedy for it?

You could use mimalloc for rust as well, mimalloc would be a simple drop-in replacement.

Not sure if it would replace the allocator used by C.

Alternatively you could install libmimalloc.so and use LD_PRELOAD to use mimalloc.-

[scrottty]

Ok so I tried this and got the same result as before. I will note that my installation of the libmimalloc.so was a bit hacky, our system doesnt have a package manager. So I installed it on another system and copied it over.

Below are the tests and the results:
main.py

use std::ffi::c_void;
use std::{fs, path::Path, time::Instant};

use zstd::zstd_safe::zstd_sys::{
    ZSTD_CCtx_setParameter, ZSTD_cParameter, ZSTD_compress2, ZSTD_createCCtx, ZSTD_initCStream,
};

use mimalloc::MiMalloc;

#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;

fn main() {
    let img_path = Path::new("/vyoma/test_scripts/img_raw");
    let mut img_data = fs::read(img_path).unwrap();

    println!("Input size: {:?} bytes", img_data.len());

    for _ in 0..=20 {
        let t0 = Instant::now();

        let mut cctx = unsafe { ZSTD_createCCtx() };
        unsafe {
            ZSTD_initCStream(cctx, 1);
        }
        unsafe {
            ZSTD_CCtx_setParameter(cctx, ZSTD_cParameter::ZSTD_c_nbWorkers, 1);
        }
        let mut output = Vec::new();
        output.reserve(img_data.len());
        unsafe {
            ZSTD_compress2(
                cctx,
                output.as_mut_ptr(),
                img_data.len(),
                img_data.as_mut_ptr() as *mut c_void,
                img_data.len(),
            );
        }

        println!(
            "Compression time: {:?} | Compression size: ???",
            t0.elapsed(),
            // data_size
        );
    }
}

output

sh-5.2# LD_PRELOAD=/usr/lib/libmimalloc.so.2 ./hello_world_rust
Input size: 13120000 bytes
Compression time: 2.437150634s | Compression size: ???
Compression time: 2.429171483s | Compression size: ???
Compression time: 2.43176177s | Compression size: ???
Compression time: 2.428406566s | Compression size: ???
Compression time: 2.431454117s | Compression size: ???
Compression time: 2.428295235s | Compression size: ???
Compression time: 2.432154654s | Compression size: ???
Compression time: 2.429074973s | Compression size: ???
Compression time: 2.430695889s | Compression size: ???
Compression time: 2.428498607s | Compression size: ???
Compression time: 2.430916991s | Compression size: ???
Compression time: 2.428063662s | Compression size: ???
Compression time: 2.431278435s | Compression size: ???
Compression time: 2.428181474s | Compression size: ???
Compression time: 2.431276635s | Compression size: ???
Compression time: 2.432433326s | Compression size: ???
Compression time: 2.430921342s | Compression size: ???
Compression time: 2.431368166s | Compression size: ???
Compression time: 2.43074851s | Compression size: ???
Compression time: 2.430214834s | Compression size: ???
Compression time: 2.433638589s | Compression size: ???

[NobodyXu]

Hmmm I think

comp = zstd.compress(img_raw, 1, 1)

Might not be equivalent to the rust code you have.

let mut cctx = unsafe { ZSTD_createCCtx() };
        unsafe {
            ZSTD_initCStream(cctx, 1);
        }
        unsafe {
            ZSTD_CCtx_setParameter(cctx, ZSTD_cParameter::ZSTD_c_nbWorkers, 1);
        }
        let mut output = Vec::new();
        output.reserve(img_data.len());
        unsafe {
            ZSTD_compress2(
                cctx,
                output.as_mut_ptr(),
                img_data.len(),
                img_data.as_mut_ptr() as *mut c_void,
                img_data.len(),
            );
        }

It's obvious that the rust code is over-allocating, the output is only ~3MB but it's allocating ~13MB.

I don't know what that one-line python is doing and how it allocates that memory.

Also

        let mut output = Vec::new();
        output.reserve(img_data.len());

Vec::reserve might reserve more than the bytes you specified.

To reserve exactly what you need, you could use Vec::reserve_exact.

[NobodyXu]

https://github.com/sergey-dryabzhinsky/python-zstd/blob/eba9e633e0bc0e9c9762c985d0433e08405fd097/src/python-zstd.c#L102

Looking at python zstd source code, it uses ZSTD_compressBound for the buffer size.

[NobodyXu]

BTW your rust code doesn't call ZSTD_freeCCtx which python version does.

[scrottty]

Damn i was really hoping that would be it! I tried a few combinations of your suggestions but nothing changed.

I tried both using ZSTD_compressBound and hardcoding a size a bit bigger than the compressed size (5mb) for Vec::reserved and the size passed to the compress function but nothing worked 😞

(also tried ZSTD_freeCCtx but also nothing)

[NobodyXu]

Vec::reserved and the size passed to the compress function

Maybe try using Vec::reserve_exact

[scrottty]

Yeh i did. Also no luck

[NobodyXu]

Maybe it's the zstd version that is different, or the way they are built?

[gyscos]

Overallocating could cost a couple of ms, but far from the 2s difference we're seeing here.

The difference in compilation is that python-zstd uses DYBAMIC_BMI2=0 to disable runtime cpu feature detection, which again could be non-zero cost but far from 2s per run.

You are running in release mode, right? We don't manually specify the C compiler opt level, so I think it inherits it from the rust side.

You can try running your test binary through perf to see how the timings are distributed.

[scrottty]

Overallocating could cost a couple of ms, but far from the 2s difference we're seeing here.

The difference in compilation is that python-zstd uses DYBAMIC_BMI2=0 to disable runtime cpu feature detection, which again could be non-zero cost but far from 2s per run.

You are running in release mode, right? We don't manually specify the C compiler opt level, so I think it inherits it from the rust side.

No I wasn't and that it fixed it! Sorry for my ignorance, what is the difference that would cause such a different performance difference?

[NobodyXu]

such a different performance difference?

Because the vendored libzstd will have the same optimization level as rust.

Unless you are using pkg-config to link with a system-wide libzstd, zstd-sys will compile a vendored version of libzstd, which uses the same optimization level as rust.