Any API to achieve pair of headers and contents layout?

[ileixe]

Hi, I was struggled with binary parsing, and this library looks awesome. We have somewhat non standard binary layout and am looking for the API that I can use of.

Binary layout like below

|------------------------------|
| u64: Number of Programs (N)  |
| -----------------------------|
| Vec<Header>                  | x N
|------------------------------|
| Vec<Program>                 | x N
|------------------------------|

As you can see, Header and Program are decided via number of entries. The problem here is that actual Program content (code, data) should be decided via Header information.

I was trying to implement

#[derive(BinRead)]
pub struct Header {
    code_size: u32,
    data_size: u32,
}

#[derive(BinRead)]
pub struct Program {
    #[br(count = ?)]   # This size should be decided via Header (code_size)
    code: Vec<u8>,
    #[br(count = ?)]  # This size should be decided via Header (data_size)
    data: Vec<u8>,
}

#[derive(BinRead)]
pub struct Metadata {
    count: u64,
    #[br(count = count)]
    headers: Vec<Header>,
    #[br(count = count)]
    programs: Vec<Program>,
}

but did not find any way to pass length information from Header to Program. Is there any way to solve this layout? I was trying to use import attribute but as a Program requires only one Header which should be correctly paired, I could not achieve it.

Thanks

[csnover]

Hi,

Probably you will want to write and use a helper function that creates the programs field from an iterator of Header values:

#[derive(BinRead, Clone, Copy)]
pub struct Header {
    code_size: u32,
    data_size: u32,
}

#[derive(BinRead)]
#[br(import_raw(header: Header))]
struct Program {
  #[br(count = header.code_size)]
  code: Vec<u8>,
  #[br(count = header.data_size)]
  data: Vec<u8>,
}

#[derive(BinRead)]
struct Metadata {
  count: u64,
  #[br(count = count)]
  headers: Vec<Header>,
  #[br(parse_with = from_iterator_args(headers.iter()))]
  programs: Vec<Program>,
}

fn from_iterator_args<'it, R, T, Arg, Ret, It>(it: It) -> impl FnOnce(&mut R, &ReadOptions, ()) -> BinResult<Ret>
where
  T: BinRead<Args = Arg>,
  R: Read + Seek,
  Arg: Clone + 'static,
  Ret: FromIterator<T> + 'static,
  It: Iterator<Item = &'it Arg> + 'it,
{
  move |reader, options, _| {
    it.map(|arg| T::read_options(reader, options, arg.clone())).collect()
  }
}

The arguments need to be cloned instead of passed by reference because binrw hasn’t yet implemented GATs for the associated BinRead::Args type. Once that is done then that could be removed too.

This is probably a sufficiently common pattern that it makes sense to add something like this helper to the library in the future, but in the meantime I think this should do what you need.

[csnover]

It turns out that something I thought was a bug in the current release version of binrw was a bug I introduced myself when working on a feature that hasn’t landed yet, so I’ve revised the example to remove the Fn-vs-FnOnce workaround which is actually not needed.

[ileixe]

Your code works like a charm. Thanks a lot!

Btw, I want to understand how it works and felt somewhat missing link and could not find proper reference. (e.g. How import_raw know header?) Could you explain a bit about them?

[csnover]

import_raw is described in the documentation in the section raw arguments.

This code:

#[derive(BinRead)]
#[br(import_raw(header: Header))]
struct Program { /* ... */ }

…generates this implementation of the BinRead trait:

impl BinRead for Program {
  type Args = Header;

  fn read_options<R: Read + Seek>(reader: &mut R, options: &ReadOptions, header: Self::Args) -> BinResult<Self> {
    /* ... */
  }
}

The from_iterator_args function generates an actual parsing function; the parse_with directive replaces the default BinRead implementation of Vec<T> and instead calls the generated function to parse the field.

[ileixe]

hi @csnover. Your answer worked very well, but now I'm having similar problem how to make it possible for write.

I was trying to make it possible following some code example on my own, but it's still bit unclear to me. So could you please give me an advice to make the code work?

This is what I am trying.

use std::io::{Seek, Write};

use binrw::{BinRead, BinResult, BinWrite, Endian};

#[derive(BinRead, BinWrite, Debug, Clone, Copy, Default)]
pub struct Header {
    pub pe_id: u32,
    pub task_size: u32,
    pub code_size: u32,
    pub data_size: u32,
}

#[derive(BinWrite, Debug, Default)]
// #[br(import_raw(header: Header))]
#[bw(import_raw(header: Header))]
pub struct ProgramData {
    // #[br(count = header.code_size + header.task_size)]
    pub code: Vec<u8>,
    // #[br(count = header.data_size, align_after = 0x8)]
    pub data: Vec<u8>,
}

#[derive(BinWrite, Debug)]
pub struct ProgramBinary {
    /// Number of headers
    pub count: u64,
    /// Binary metadata headers
    // #[br(count = count)]
    pub headers: Vec<Header>,
    /// Program data per header
    // #[br(parse_with = custom_parser(headers.iter()))]
    #[bw(write_with = write_custom(headers.iter()))]
    pub programs: Vec<ProgramData>,
}

// fn parse<'it, R, T, Arg, Ret, It>(it: It) -> impl FnOnce(&mut R, Endian, ()) -> BinResult<Ret>
// where
//     T: BinRead<Args<'it> = Arg>,
//     R: Read + Seek,
//     Arg: Clone + 'static,
//     Ret: FromIterator<T> + 'static,
//     It: Iterator<Item = &'it Arg> + 'it,
// {
//     move |reader, endian, _| it.map(|arg| T::read_options(reader, endian, arg.clone())).collect()
// }

fn write_custom<'it, T, W, Arg, It>(it: It) -> impl FnOnce(&T, &mut W, Endian, ()) -> BinResult<()>
where
    T: BinWrite<Args<'it> = Arg>,
    W: Write + Seek,
    Arg: Clone + 'static,
    It: Iterator<Item = &'it Arg> + Clone + 'it,
{
    move |programs, writer, endian, _| {
        it.clone().map(|arg| T::write_options(programs, writer, endian, arg.clone())).collect()
    }
}

fn main() {}

I changed lifetime to follow latest release and write_custom is the function that compiler complaints like below

error[E0382]: use of moved value: `__binrw_generated_write_function`
  --> src/main.rs:23:10
   |
23 | #[derive(BinWrite, Debug)]
   |          ^^^^^^^^
   |          |
   |          value used here after move
   |          move occurs because `__binrw_generated_write_function` has type `impl for<'a, 'b> FnOnce(&'a Vec<ProgramData>, &'b mut W, binrw::Endian, ()) -> Result<(), binrw::Error>`, which does not implement the `Copy` trait
   |
   = note: this error originates in the derive macro `BinWrite` (in Nightly builds, run with -Z macro-backtrace for more info)

It looks somehow related to your previous Fn vs FnOnce workaround but I could not catch more.

Any advice would be very appreciated. Thanks

[ileixe]

I saw this is indeed a bug for BinWrite side. BinRead works well below

            let __binrw_generated_read_function = (parse(headers.iter()));
            let __binrw_generated_options_programs = __binrw_generated_var_options;
            let mut programs: Vec<ProgramData> = __binrw_generated_read_function(
                    __binrw_generated_var_reader,
                    __binrw_generated_options_programs,
                    <_>::default(),
                )
                .map_err(|err| binrw::error::ContextExt::with_context(

But BinWrite generates invalid codes as __binrw_generated_write_function used after moved.

        let __binrw_generated_write_function = binrw::__private::write_fn_type_hint(
            write_custom(headers.iter()),
        );
        let __binrw_generated_args_programs = binrw::__private::write_function_args_type_hint::<
            Vec<ProgramData>,
            W,
            _,
            _,
        >(__binrw_generated_write_function, ());
        __binrw_generated_write_function(
            {
                let __binrw_generated_saved_position = binrw::io::Seek::stream_position(
                    __binrw_generated_var_writer,
                )?;
                &((programs))
            },
            __binrw_generated_var_writer,
            __binrw_generated_var_endian,
            __binrw_generated_args_programs,
        )?;