Update tflite-micro-compiler to work with recent tflite-micro

[panickal-xmos]

Hi,

We at XMOS have been doing some work internally to move tflite-micro-compiler up to work with latest tflite-micro.

We would be happy to contribute the changes. It contains some hardware-specific changes at the moment. If a PR is of interest, we can put some effort into cleaning it up. Would that be of interest?

Best regards,
Deepak

[rafzi]

Hello and thanks for reaching out!

There already were efforts to update this tool by @yair-ehrenwald and @andrewstevens-infineon.

I'd suggest that you send a PR including your specific changes without much effort to the master branch so that we can all take a look at your approach and can find the most suitable solution.

[panickal-xmos]

Thank you @rafzi! Will tidy it up a bit and put up a PR.

[andrewstevens-infineon]

Hi Deepak

Sounds great. Please do add PR with your goodies.

Here @ Infineon currently have a pretty extensive internal fork that is long overdue to be push-ed as PR. This adds better arena optimization, generation of static OpData (user_data) record and support for Prepare-phase selection of kernel variants (to allow link-time elimination of unused kernel variants.

Current plan is to get it out after the first productive release (part of Modus Toolbox 3.0 / MTB-ML 2.0 for our PSoC product-line) and refactor so it bases on latest+greated tflite(u) without a lot of in-house dependencies. Sounds like your PR would cover some of this.

If you're interest in smore details I've promised to do a brief walk-through at the next tflite(u) SIG.
I'd be delighted to collaborate/coordinate on further development / ideas. There's plenty more useful stuff to do...

[panickal-xmos]

Thank you @andrewstevens-infineon! It sounds like you have made much more progress with the fork. :) Will be great to see your PR. I have got ours working with the latest tflite-micro with few changes. Due to the addition of MicroContext recently in tflite-micro, we apply a small patch to tflite-micro, to avoid linking in all of that. Would be happy to work together on future ideas!

I was not aware of tflite-micro SIG. I'll join the next SIG micro meeting. Looking forward to the walk-through!

[andrewstevens-infineon]

Glancing at your changes looks like we've been covering similar ground w.r.t ScratchBuffer handling. Our version bases on tflite(u) from 2022-02-22 (same version ARM used for the -03 release of their ethos-U SW stack).

However the approach is a little different - rather than expand the "peeking" into the interpreter internals we went the route of patching the kernel to add callback hooks we can intercept at relevant entry-points. The hope being that these would probably be more stable than the internals implementing them.

[panickal-xmos]

I have put up the PR here. Please have a look through.

Tflite-micro has introduced MicroContext class, which is now the entry point to the Allocate methods.
Linking it in would link in the allocator, micro_graph etc, thereby losing the benefit of reducing code size.
To avoid this, two changes are done.

• We apply a small patch to define a different MicroContext with function pointers, and for some methods we use.
• We build a separate tflite-micro library without including some core files in tflite-micro. This is then used to link with the compiled model executable. To demonstrate this, and to test github actions workflow is running properly, I have added a patch for tflite-micro Makefile.

Missing functionality
Some ops are not supported at the moment, as they link in unsupported methods.
Haven't looked into them yet, as they are not an immediate priority for us.
These are:

• AssignVariable
• CallOnce
• If
• ReadVariable
• UnidirectionalSequenceLstm
• While.cc
• VarHandle