How do I parse a binary format consisting of a hierarchy of chunks with different types?

[philpax]

Hey there! I'm looking into loading the map from GTA: Vice City in Bevy, and I'm interested in using binrw as a solution for my soon-to-be file format reading woes. The primary format I'd have to read is the RenderWare binary stream, detailed here: https://gtamods.com/wiki/RenderWare_binary_stream_file#File_Format

This format consists of hierarchical chunks; each chunk has a header (type, size in bytes including children, library stamp) (three u32s), and is followed by its children (also chunks), and then followed by the data for the chunk itself.

What's the best way to represent this in binrw - specifically, how do I choose the variant automatically based on the type?

[jam1garner]

This (sections/chunks in a file with a chunk length in bytes) is a usecase I want to make more ergonomic, maybe some way to use the take io::Read adapter.... (see #121)

Anyways, how to deal with it in existing binrw, a few options:

1. (Preferred imo) big enum with per-chunktype magic on each variant, and maybe a bit of repetition in the body
2. br(repr(u32)) enum for type, then when you want to do per-type parsing you can use pre_assert on each variant of the underlying chunk contents parser to select which variant should be used for parsing

In either case you should have a fallthrough/default "unknown chunk type" variant that either reads in the raw bytes or skips them

Example 1

enum Chunk {
    // ...

    #[br(magic = 14u32)]
    FrameList {
        #[br(temp)]
        size: u32,
        version: u32,

        #[br(temp)]
        frame_count: u32,

        #[br(count = frame_count)]
        frames: Vec<Frame>,
    },

    // ...

    Unknown {
        chunk_type: u32,
        chunk_size: u32,
        version: u32,

        #[br(count = chunk_size)]
        data: Vec<u8>,
    },
}

if you do something like the above, then if you find a chunk you don't know how to parse yet then it handles it gracefully

Example 2

if you're more interesting in doing the other way (a bit less repetition w/ header fields)....

#[binread]
struct Chunk {
    kind: ChunkType,
    size: u32,
    version: u32,

    #[br(pad_size_to = size)]
    #[br(args(kind, size, version))]
    body: ChunkData,
}

#[binread]
#[derive(Debug, PartialEq)]
#[br(repr(u32))]
enum ChunkType {
    // ...
    FrameList = 14,
    // ...
}

use ChunkType::*;

#[binread]
#[br(import(kind: ChunkType, size: u32, version: u32))]
enum ChunkData {
    #[br(pre_assert(kind == FrameList))]
    FrameList {
        #[br(temp)]
        frame_count: u32,

        #[br(count = frame_count)]
        frames: Vec<Frame>
    },
}

[philpax]

I've gotten around to actually trying this out, and Unknown works perfectly (which is awesome), but I'm not sure what the best way to handle the recursive type would be:

pub enum Section {
    Unknown {
        chunk_type: u32,
        chunk_size: u32,
        version: u32,

        // amount of remaining data to read, in bytes
        #[br(temp, calc = chunk_size - 12)]
        data_size: u32,

        // what count am I to use here? `data_size` is in bytes
        #[br(count = data_size)]
        children: Vec<Section>,
    },
    // ...
}

especially since the other variants feature data after the children:

pub enum Section {
    // ...
    #[br(magic = 0x10u32)]
    Clump {
        #[br(temp)]
        size: u32,
        version: u32,

        // amount of remaining data to read, in bytes
        #[br(temp, calc = chunk_size - 12)]
        data_size: u32,

        #[br(count = data_size)]
        children: Vec<Section>,

        atomic_count: u32,
        light_count: u32,
        camera_count: u32,
    },
}

[philpax]

uh, ignore the above comment for now - I think I've misunderstood the format. I'll come back once I know what I'm looking at...