Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't decompress data which is compressed by rust code #235

Open
WindSoilder opened this issue Aug 12, 2024 · 2 comments
Open

Can't decompress data which is compressed by rust code #235

WindSoilder opened this issue Aug 12, 2024 · 2 comments

Comments

@WindSoilder
Copy link

Here is simplest reproducing example:

import zstandard
zstandard.decompress(b'\x28\xb5\x2f\xfd\x00\x58\x11\x00\x00\x7b\x7d')

It raises an error: ZstdError: could not determine content size in frame header

More context

I'm trying to rewrite a client application in rust, it sends compressed data to server, then server decompresses it. Unfortunally the server failed to decompress data.

Here is how I do it in client side:

use std::io::Cursor;
use zstd;

fn main() {
    let body = zstd::encode_all(Cursor::new("{}".as_bytes()), 3).unwrap();
    for x in body.iter() {
        print!("\\x{x:x?}");
    }
}

And I copied the body and decompressed it in python, and it failed.


If I tried to compress data({} in my example) in python, and decompressed in rust, it successes. So I think it's the issue in python side.

In rust, I'm using zstd-rs for compressing/decompressing

@g2p
Copy link

g2p commented Sep 23, 2024

See #150; you must pass a max_output_size to decompress in this case.

Docs

@aarshivv
Copy link

aarshivv commented Feb 5, 2025

@WindSoilder , If the size of input is known, you can pass that to encoder, that way the content size will be added to the frame header and you will be able to decompress in python too without any issue (no need to pass max_output_size in this case.
Sample rust code:

    let input_bytes = serde_json::to_vec(&input)?;
    let mut output = Vec::new();
    let mut encoder = zstd::Encoder::new(&mut output, 0)?;
    encoder.set_pledged_src_size(Some(input_bytes.len() as u64))?;   // This adds the needed content size in frame header
    encoder.write_all(&input_bytes)?;
    encoder.finish()?;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants