Zigbee2mqtt on home assistant fails to set state of switches frequently

[fergalish]

What happened?

Recenty, I since upgrading beyond 2.0.x, I find that the z2m add-on intermittently crashes in an unusual manner. HA continues to receive updates about the states of sensors (e.g. temperature, humidity etc) but can no longer set the state of switches (e.g. switch.turn_on service etc). This is happening now every few hours and is causing problems (specifically for climate control; automation can't switch devices on or off). Changing the state of a switch via the z2m device page also fails at this point.

The issue is temporarily fixed by restarting the z2m add-on from the HA add-ons page. Restarting z2m from its own web-page fails to successfully restart- HA does not reconnect to the devices. The watchdog is enabled but does not restart z2m.

I've tried multiple fixes - reboots, eliminating nice-but-not-necessary add-ons & integrations to reduce resource usage, placed the Zigbee dongle on a powered USB2 hub, throttled a particularly chatty zigbee device (Owon PC321), and so on. No success.

I recently upgraded to Z2M 2.1.3.

N.B. I tried the ember driver when I upgraded some months ago from ConbeeII to Dongle-E, but had problems and so remained with EZSP driver. I don't think this should be relevant as everything was working fine until I upgraded beyond z2m 2.0.x in early February. Trying ember driver now fails with:

[2025-03-02 12:54:49] info: 	zh:ember: ======== Ember Adapter Starting ========
...
[2025-03-02 12:54:50] error: 	z2m: Error while starting zigbee-herdsman
[2025-03-02 12:54:50] error: 	z2m: Failed to start zigbee-herdsman
[2025-03-02 12:54:50] error: 	z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start_crashes-runtime.html for possible solutions
[2025-03-02 12:54:50] error: 	z2m: Exiting...
[2025-03-02 12:54:51] error: 	z2m: Error: Adapter EZSP protocol version (8) is not supported by Host [13-16].

One possible hint of the problem is the size of the "node index.js" process in the z2m container, occupying 10.4G of virtual memory! However swap usage remains at zero and the system can remain functional for several hours even with the 10.4G VSIZE before eventually crashing.

Tasks: 416 total,   1 running, 414 sleeping,   0 stopped,   1 zombie
%Cpu(s):  1.9 us,  1.2 sy,  0.0 ni, 95.4 id,  1.5 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   7945.2 total,   1233.7 free,   1831.4 used,   4906.8 buff/cache     
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   6113.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                      
 818369 root      20   0   10.4g 122932  51456 S   1.0   1.5   0:07.61 node index.js                                                                                                                
   1589 root      20   0 3724420 151976  55168 S   0.7   1.9   6:26.06 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock                                                       
   2179 fergal    20   0  501416  12928  10496 S   0.7   0.2   0:19.98 /usr/libexec/gvfs-udisks2-volume-monitor                                                                                     
      1 root      20   0  316724  11968   8424 S   0.3   0.1   0:09.88 /sbin/init splash                                                                                                            
    282 root      20   0  108472  57236  55180 S   0.3   0.7   3:10.85 /lib/systemd/systemd-journald                                                                                                
   1337 fergal    20   0   20072  10624   8576 S   0.3   0.1   0:08.39 /lib/systemd/systemd --user                                                                                                  
   4544 root      20   0  321996 109936  27776 S   0.3   1.4   0:34.72 python3 -m supervisor                                                                                                        
   6381 root      20   0   20032   6016   4096 S   0.3   0.1   0:56.29 bash /usr/bin/bashio /etc/s6-overlay/s6-rc.d/mariadb-post/run                                                                
 262167 root      20   0 1320632 656272 107776 S   0.3   8.1   4:39.78 python3 -m homeassistant --config /config                                                                                    
 675333 63112     20   0 3235176  98860  95532 S   0.3   1.2   0:06.05 /lib/systemd/systemd-journal-gatewayd                                                                                        
 822392 root      20   0   11840   4992   2816 R   0.3   0.1   0:00.07 top                                                                                                                          

What did you expect to happen?

An automation to, eg, switch a radiator valve on or off would successfully switch the valve.
In the attached log I was luckily testing just at the right moment. You can see:

[2025-03-02 08:58:37] debug: z2m:mqtt: Received MQTT message on 'zigbee2mqtt/Breakfast Radiator Valves/set' with data 'ON'

This was successful. Moments later:

[2025-03-02 08:58:49] debug: z2m:mqtt: Received MQTT message on 'zigbee2mqtt/Playroom Radiator Valves/set' with data 'OFF'

was unsuccessful. All zigbee switches were now unresponsive and I restarted the z2m add-on.

How to reproduce it (minimal and precise)

Sorry, no idea. The problem is intermittent.

Zigbee2MQTT version

2.1.1-1

Adapter firmware version

6.10.3.0 build 297

Adapter

/dev/serial/by-id/usb-ITEAD_SONOFF_Zigbee_3.0_USB_Dongle_Plus_V2_20240220203028-if00

Setup

RPi5 debian 12 with HA Supervised installation, Sonoff 3.0+ Dongle-E

Debug log

log.log

[Koenkk]

You are using the old ezsp driver, it's recommended to use the new ember drivers (info), that may fix your instability

[fergalish]

Thanks @Koenkk . I tried the new ember driver but it fails to start:

[2025-03-02 12:54:49] info: 	zh:ember: ======== Ember Adapter Starting ========
...
[2025-03-02 12:54:50] error: 	z2m: Error while starting zigbee-herdsman
[2025-03-02 12:54:50] error: 	z2m: Failed to start zigbee-herdsman
[2025-03-02 12:54:50] error: 	z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start_crashes-runtime.html for possible solutions
[2025-03-02 12:54:50] error: 	z2m: Exiting...
[2025-03-02 12:54:51] error: 	z2m: Error: Adapter EZSP protocol version (8) is not supported by Host [13-16].

If upgrading firmware on the Dongle-E is the only way to proceed, I can try that (never did that before, am nervous about it). BUT I reiterate that the system was stable until a few weeks ago with existing firmware & ezsp driver. Something in a recent software upgrade is causing the problem.

I think CC1352P2_CC2652P_launchpad_coordinator_20240710 is the correct firmware now?

edit: Just realising the ZBDongle-E stick I have is not a CC2652 chip, so I assume that's not the correct firmware and nor is this the correct method to flash.

[chris-1243]

For your dongle-E, you may want to check this link for a suitable firmware

https://github.com/darkxst/silabs-firmware-builder/releases

This one to flash via a web browser

https://darkxst.github.io/silabs-firmware-builder/

@Nerivec is also providing tools and firmware in a bit more experimental way as described here

• First flash a v12 firmware (7.3.x) and start Z2M without any modification in configuration.yaml.
• Wait for Z2M to run and stop it.
• Modify your adapter section in your configuration and set ember.
• Flash a v13 (7.4.5) or v14 (8.0.2) firmware.
• Start Z2M and you should be fine with the ember driver.

[fergalish]

Thanks @chris-1243 . I have flashed the 7.3.3 firmware successfully. All devices are still present (phew!) and working (double-phew!) and the Z2M 'About' page shows:

Coordinator type    EZSP v12
Coordinator revision    7.3.3.0 build 239

I stopped it and changed the add-on config to use the ember driver and this fails with the same problem as before full log attached:

[2025-03-03 10:02:13] info: 	zh:ember: ======== Ember Adapter Starting ========
[2025-03-03 10:02:13] info: 	zh:ember:ezsp: ======== EZSP starting ========
[2025-03-03 10:02:13] info: 	zh:ember:uart:ash: ======== ASH Adapter reset ========
[2025-03-03 10:02:13] info: 	zh:ember:uart:ash: RTS/CTS config is off, enabling software flow control.
[2025-03-03 10:02:13] debug: 	zh:ember:uart:ash: Opening serial port with {"path":"/dev/serial/by-id/usb-ITEAD_SONOFF_Zigbee_3.0_USB_Dongle_Plus_V2_20240220203028-if00
","baudRate":115200,"rtscts":false,"autoOpen":false,"parity":"none","stopBits":1,"xon":true,"xoff":true}
[2025-03-03 10:02:13] info: 	zh:ember:uart:ash: Serial port opened
[2025-03-03 10:02:13] info: 	zh:ember:uart:ash: ======== ASH starting ========
[2025-03-03 10:02:13] error: 	zh:ember:uart:ash: Received frame with CRC error
[2025-03-03 10:02:13] debug: 	zh:ember:uart:ash: ---> [FRAME type=RST]
<snip>
[2025-03-03 10:02:14] debug: 	zh:ember:ezsp: <=== [FRAME: ID=0:"VERSION" Seq=0 Len=7]
[2025-03-03 10:02:14] error: 	z2m: Error while starting zigbee-herdsman
[2025-03-03 10:02:14] error: 	z2m: Failed to start zigbee-herdsman
[2025-03-03 10:02:14] error: 	z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start_crashes-runtime.html for possible solutions
[2025-03-03 10:02:14] error: 	z2m: Exiting...
[2025-03-03 10:02:14] error: 	z2m: Error: Adapter EZSP protocol version (12) is not supported by Host [13-16].

The suggested link seems not relevant.

[chris-1243]

[2025-03-03 10:02:14] error: z2m: Error: Adapter EZSP protocol version (12) is not supported by Host [13-16].

This is expected. Your original firmware was quite old. You had to move to a v12 (7.3.3) version. The aim is to be able to create a coordinator_backup.json compatible with ember.
By the way, you did not completely follow what I wrote.😉

• Downlaod latest firmware for you dongle-E (8.0.2). Follow the links provided above or check this one
• Stop Z2M and flash your dongle once again. As you have a full backup of your network, no need to worry.
• Set ember in your configuration and start Z2M...

This time, you should be fine and completely up to date