Kaitai build error

[maraMaraschino]

In trying to create an example file that works alongside the instructions under the kaitai struct runtime library for Awkward, I have generated the file "ParametricParser.ksy" in a separate repository which is meant to be able to parse key-value pairs with lengths of 3 or 8.
When following the instructions, I simply replace the command:
./kaitai-struct-compiler -t awkward --outdir src-animal example_data/schemas/animal.ksy
With:
$ ./kaitai-struct-compiler -t awkward --outdir src-parametric ~/dataReaderWriter/kaitai/ksy/ParametricParser.ksy
And then, instead of running:
awkward-kaitai-build src-animal/animal.cpp -b build
I run:
$ awkward-kaitai-build src-parametric/kv_pairs.cpp -b build
Which produces two errors:
error: no matching function for call to ‘kv_pairs_t::kv_pair_t::kv_pair_t(awkward::LayoutBuilder::Record<std::map<long unsigned int, std::__cxx11::basic_string<char> >, awkward::LayoutBuilder::Field<0, awkward::LayoutBuilder::ListOffset<long int, awkward::LayoutBuilder::Numpy<unsigned char> > >, awkward::LayoutBuilder::Field<1, awkward::LayoutBuilder::ListOffset<long int, awkward::LayoutBuilder::Numpy<unsigned char> > > >&, int, kaitai::kstream*&, kv_pairs_t*, kv_pairs_t*&)’

error: no matching function for call to ‘kv_pairs_t::kv_pair_t::kv_pair_t(awkward::LayoutBuilder::Record<std::map<long unsigned int, std::__cxx11::basic_string<char> >, awkward::LayoutBuilder::Field<0, awkward::LayoutBuilder::ListOffset<long int, awkward::LayoutBuilder::Numpy<unsigned char> > >, awkward::LayoutBuilder::Field<1, awkward::LayoutBuilder::ListOffset<long int, awkward::LayoutBuilder::Numpy<unsigned char> > > >&, int, kaitai::kstream*&, kv_pairs_t*, kv_pairs_t*&)’
The two errors point to lines 36 and 45 of the generated file respectively.

[ManasviGoyal]

Can you share the generated C++ files too? It would make easier to investigate. From what I see, it looks like the current implementation doesn't support parametric types and will need an implementation for the same.

Have you tried running other tests from the example_data directory besides animal.ksy? You can just run the make file for that. Just to make sure there aren't any other issues. Thanks!

[maraMaraschino]

Here's the link for kv_pairs.cpp for ParametricParser.ksy and animal.cpp for animal.ksy.
I just tried to run ./kaitai-struct-compiler -t awkward --outdir src-midas example_data/schemas/midas.ksy for the midas file as a test and it ran into error: key not found: scdms
Should I test it out for each of the .ksy files in example_data just to see which ones are working and which ones aren't?

[maraMaraschino]

Ok, I ran the generate and build commands for the other .ksy files in example_data, and all of the other ones worked.
midas.ksy, lists.ksy, scdms_no_enums_v8.ksy, scdms.ksy, and scdms_v8.ksy all had warnings when generating the source and header files, but none of the files besides ParametricParser.ksy had problems with building awkward-kaitai with the generated .cpp files.

[ManasviGoyal]

Here's the link for kv_pairs.cpp for ParametricParser.ksy and animal.cpp for animal.ksy. I just tried to run ./kaitai-struct-compiler -t awkward --outdir src-midas example_data/schemas/midas.ksy for the midas file as a test and it ran into error: key not found: scdms Should I test it out for each of the .ksy files in example_data just to see which ones are working and which ones aren't?

midas is expected to give the error since we don't support the import feature yet.

[maraMaraschino]

Here's the link for kv_pairs.cpp for ParametricParser.ksy and animal.cpp for animal.ksy. I just tried to run ./kaitai-struct-compiler -t awkward --outdir src-midas example_data/schemas/midas.ksy for the midas file as a test and it ran into error: key not found: scdms Should I test it out for each of the .ksy files in example_data just to see which ones are working and which ones aren't?

can you share the kv_pairs.h too?

Yeah, no problem. Here is that for you.

[ManasviGoyal]

@maraMaraschino Do you have a data file for this ParametricParser.ksy that I can use for testing? I see the issue. The order of the arguments in the constructor call is wrong. Looks like a simple fix if that is the only issue.

Maybe you can try reversing the arguments lines 36 and 45 in kv_pairs.cpp like 3, kv_pair_recordbuilder instead and see if that fixes the issue? It it does, I'll make the fix in Scala code too. If it doesn't then we will need to see what else is wrong.

[maraMaraschino]

Here is the bin file generated by the Parametric Generator.
Also, just making sure that I'm understanding correctly before I make a change, lines 36 and 45 should change to:
m_short_pairs->push_back(new kv_pair_t(3, kv_pair_recordbuilder, m__io, this, m__root));
m_long_pairs->push_back(new kv_pair_t(8, kv_pair_recordbuilder, m__io, this, m__root));

[ManasviGoyal]

Here is the bin file generated by the Parametric Generator. Also, just making sure that I'm understanding correctly before I make a change, lines 36 and 45 should change to: m_short_pairs->push_back(new kv_pair_t(3, kv_pair_recordbuilder, m__io, this, m__root)); m_long_pairs->push_back(new kv_pair_t(8, kv_pair_recordbuilder, m__io, this, m__root));

Yes, exactly!

[maraMaraschino]

It looks like that worked! Thank you. I'll try to keep doing this with other .ksy files I've made for additional examples.

[ManasviGoyal]

It looks like that worked! Thank you. I'll try to keep doing this with other .ksy files I've made for additional examples.

Great. I'll add the fix tomorrow so that the code gets generated correctly.

[ManasviGoyal]

@maraMaraschino Instead of having the src-animal and src-ParametricParser in the main repository, I think it will be better if you can make a branch with the files you have been working on. This will keep things clean in main. I am removing them for now. But if you need them you can add them in a separate branch. Thanks!

Also, I have added the fix, so you should be able to generate the files correctly without changing anything. But it is failing when you pass the kv_pairs_data.bin in Python as in the generated code the params are not being filled in a LayoutBuilder. I can try and see if I have time next week to fix that.