[Xylo IMU] Losing performance when inferencing on XyloIMU

[PeterRolfes]

I have an issue with the XyloIMU neuromorphic chip when inferencing on it. The whole notebook including the results is here: https://github.com/PeterRolfes/XyloIMU-Train

The idea is to simply train a spiking neural network using rockpool on some gearbox kaggle data: https://www.kaggle.com/code/abhinavraj24/data-gear-box-fault-detection

The training itself worked just fine as it produced accuracies of ~97% on training and test data. The problems occurs when i am trying to inference on the XyloIMU, because the accuracy suddenly drops to ~82%.

Can someone help me to get the performance close to the original one or explain why it drops that heavily?

Thanks in advance!

[DylanMuir]

Hi @PeterRolfes , thanks for reaching out about this. Generally speaking I would expect to see some drop in accuracy due to quantisation error, but this is a larger drop than I would expect. I assume that 97% is the simulation test accuracy, is that correct?

You could try running with XyloSim, and see if there is a difference between XyloSim and XyloSamna (there shouldnt' be).

What quantisation method are you using? You could try both channel_quantise and global_quantise, and see if you obtain better performance.

Finally you could investigate the internal activity of the simulation and quantised network, to see if you can identify where the difference arises.

[PeterRolfes]

Hello @DylanMuir, thank you for the quick response. 97% is the accuracy that is achieved by just running the inference from that SynNet on my MacBook. By the way, I also tried WaveSenseNet, am I right that this one is not usable on the XyloIMU due to the inherent dilations?

When i am using XyloSim, the loop can't be executed at all. I found out that it was possible when i changed the data preprocessing such that there are no negative values any more. But then my accuracy dropped significantly again, even worse than 82%.

I did try any combinations of the different quantization methods. That didn't improve the results either.

What do you mean by investigating the internal activity exactly?

[DylanMuir]

Hi @PeterRolfes , thanks for your reply.

It's concerning about XyloSim not working --- what error do you observe?

When using the SynNet network, how are you encoding your IMU data?

[DylanMuir]

By "investigating the internal activity", I mean passing record = True when evolving the network, and then visualising the internal state traces that are returned.

[PeterRolfes]

Hi and thanks again @DylanMuir,

using XyloSim to infer produces following error:
TypeError: evolve(): incompatible function arguments. The following argument types are supported:
1. (self: xylosim.v2.XyloLayer, arg0: list[list[int]]) -> list[list[int]]

The problem might just be, that i am not using the IMU data at all, i am using the data from the kaggle link previously provided:
https://www.kaggle.com/code/abhinavraj24/data-gear-box-fault-detection

Sorry if that caused a misunderstanding, that is on me

In preprocessing i am basically calculating some features and then quantize them to Integer as that is the only format Xylo accepts. At least that is what I experienced.

[PeterRolfes]

By "investigating the internal activity", I mean passing record = True when evolving the network, and then visualising the internal state traces that are returned.

Will do thank you :)