channel problem

[rose-jinyang]

Hello
How are you?
Thanks for contributing to this project.
I converted my PyTorch model to the TFLite model as your method.
But this TFLite model is different from the original TFLite model from the Keras model.

This figure is the structure of the base TFLite model converted from the Keras model.

The below figure is the structure of the TFLite model converted from the Pytorch model.

The below figure is the structure of the ONNX model converted from the Pytorch model.

The main problem is that the TFLite model converted from the Pytorch model runs more slowly than the base TFLite model converted from the Keras model.

As u know, the PyTorch model uses channel-first as tensor ordering and the Tensorflow uses channel-last.
I looked at the structure of the TFLite model converted from the PyTorch model.
Comparing with the base TFLite model converted from the Keras model, there are many Transpose layers in the TFLite model converted from the PyTorch model.
I think that this affects the model inference speed.
Can we get the TFLite model of using channel-last ordering from the PyTorch model?

[sithu31296]

Hi Thanks for your interest in this repo.

PyTorch and ONNX are written in channel first format.
PyTorch has beta support for NCHW format. Please check here. But this will not convert the model to NCHW format. Although ONNX has some support for NHWC format, but ONNX-tf has no support for NHWC. So, as far as I know, you cannot do that.

But if your main concern is speed, run onnx model with onnxruntime. I see a performance gain from 100ms to 1ms with just running onnx model with onnxruntime rather than using pytorch model.

If you have time, please also check out this guide for optimizing pytorch model.

[PINTO0309]

I don't mean to interrupt this thread, but there is a way. You can use the tool I created to convert it to NHWC. Since it is still a WIP, there are occasional bugs. Eliminate all Transpose layers.

PyTorch (NCHW) -> ONNX (NCHW) -> OpenVINO (NCHW) -> optimized -> saved_model (NHWC) -> tflite (NHWC)
https://github.com/PINTO0309/openvino2tensorflow

sample image
https://github.com/PINTO0309/openvino2tensorflow#8-model-structure

[sithu31296]

That is a great tool! But I have a question. That channel conversion you implemented is done in OpenVINO to tf model step. Right? Even official onnx_tf doesn't have any support for NHWC. Is there any speed comparison you tested?

[PINTO0309]

That channel conversion you implemented is done in OpenVINO to tf model step. Right?

Yes. That's right.

Even official onnx_tf doesn't have any support for NHWC. Is there any speed comparison you tested?

Unfortunately, it generates a model that is far more optimized and faster than the onnx-tf developed by Microsoft. Sorry. I didn't record the benchmark because it is very troublesome. It's fast enough to feel, since it doesn't contain any transpose. However, only GroupConvolution is known to be very slow because tflite does not support it. The tool I created reads all the weights and structures during the conversion process and automatically transposes NCHW to NHWC.

By the way, if you want to reverse-translate tfite (NHWC) files, the following tool is also available.

"Generate saved_model, tfjs, tf-trt, EdgeTPU, CoreML, quantized tflite, ONNX, OpenVINO, Myriad Inference Engine blob and .pb from .tflite. Support for building environments with Docker. It is possible to directly access the host PC GUI and the camera to verify the operation. NVIDIA GPU (dGPU) support."
https://github.com/PINTO0309/tflite2tensorflow

Here are 122 models that I have committed to transformations generated from PyTorch and other sources. You can try them out.
https://github.com/PINTO0309/PINTO_model_zoo

Specially optimized TFLite runtime installer
https://github.com/PINTO0309/TensorflowLite-bin

[sithu31296]

Thanks for your answer! I will check it out your works!

[PINTO0309]

I am looking forward to your contribution. I am a lone committer.

[lee-man]

I also face this problem when I try to deploy the model on Android phones using Google's official demo. When I try to replace the TF models, the channel ordering is different so I cannot directly reuse other codes.

[PINTO0309]

@lee-man
You can freely change the channel order and the shape of Transpose and Reshape.
https://github.com/PINTO0309/openvino2tensorflow#6-7-replace-weights-or-constant-values-in-const-op-and-add-transpose-or-reshape-or-cast-just-beforeafter-the-operation-specified-by-layer_id

[lee-man]

@lee-man
You can freely change the channel order and the shape of Transpose and Reshape.
https://github.com/PINTO0309/openvino2tensorflow#6-7-replace-weights-or-constant-values-in-const-op-and-add-transpose-or-reshape-just-before-the-operation-specified-by-layer_id

Got it. I will check your tool later. Thanks for your help!

[MPolaris]

You can use this tools, it will auto transform NCHW to NHWC:
https://github.com/MPolaris/onnx2tflite