0.0.24 has introduced duplicate entities

[andrewmooreio]

Version of the custom_component

0.0.24

Describe the bug

I've just updated from 0.0.23 to 0.0.24 and all existing entities in the "Home" device are no longer provided by the integration. They've now been replaced by new entities with the same names, appended with _2:

It hasn't impacted my eddi, zappi or harvi devices, only the "Home" device. This is very frustrating as I have other entities, such as a Reimann sum integral, that are now broken.

I don't know if it's related to the update or coincidence, but any advice on possible cause would be appreciated.

Thank you.

Debug log

N/A

[pickeringar]

I have encountered the same problem - happened when updating HA core and restarting HA with 0.0.24.

As a work around found if you delete the integration and re-add it then the problem is resolved until the next re-start. NOTE: deleting and re-adding the integrtaion requires HUB serial number and API key.

[tomrush]

I was just about to report this as I am seeing it too. I just rolled back to 0.0.23 for the moment. I did check the home assistant logs but nothing looked to be erroring. However I will have a bit more of a check over the weekend when I get some free time.

[a-wolter]

I can confirm this issue

[CTMULOW]

Same issue broken my Energy page!

[MattCheetham]

Same here.

[JohnTis]

Suggest clicking the Subscribe feature to this thread so it is left clear for updates / solutions. Adding Same here tends to cloud the issue.

[Voz-DeCarabas]

Glad I found this thread. I thought I had done something horribly wrong.

[tomrush]

I have done some investigation into this and it is just hub sensors that cause the issue.

In 0.0.24 the unique id in MyenergiHubSensor has changed here the word hub has been added.

If I remove the word hub and restart the integration the correct fields are updated however the entities with the _1 are still present.

Would someone with a bit more knowledge of the integration be able to put a fix in place for this?

[water-escape]

I was just about to report this as I am seeing it too. I just rolled back to 0.0.23 for the moment. I did check the home assistant logs but nothing looked to be erroring. However I will have a bit more of a check over the weekend when I get some free time.

How do i roll back. Same issues here.

[pleaseremove]

I have the same issue, all my duplicate entities have my in the middle of them.

so it was sensor.myenergi_home_grid_export_today and is now sensor.myenergi_my_home_grid_export_today

I dont want to just swap onto the new entity as it means I loose my grid consumption graphs.

[ntompson]

Also getting this issue. I don't have a problem with an extra my as noted by @pleaseremove - just the hub entities with a _2 appended. Deleting the integration and reintegrating solved the problem (for now - I assume it will break at next restart, which I haven't attempted).

[smoketabs]

Same issue here. Sample of duplicate sensors.

[lozzd]

+1, have rolled back - the names are all identical just with _2 appended. The new ones are updating fine but I don't want to lose the historical data in the original sensors - unclear why HA created those new ones?

[trizmark]

The 'hub' was added into the unique_id of the hub as HA was complaining that the integration does not generate unique IDs. It's caused by the vHub functionality of myenergi devices. The issue was reported in #375

[LeiChat]

Am I correct in thinking, if we temporarily disabled the integration, renamed the entities to the new convention, then installed the upgrade via HACS, it would hopefully not create the duplicates and would retain the history/statistics?

Any custom templating (card-mod, etc.) we have might need to also have entities references manually renamed.

Or is there an easier way to deal with the duplicate issue that will not impact the history/statistics, please?

[trizmark]

It could possibly work, I'll try to test it.
This issue didn't come up in my case as with my libbi install the vHub functionality was moved from my eddi to my libbi. (I also only don't rely on HA to keep statistics for me; I got all my data going into VictoriaMetrics for long-term storage)

[lozzd]

I don't think that will work, in my case the hub metrics have the identical name before and after the upgrade.. Except since that's now a duplicate, HA adds _2 to the new ones. Even in developer tools I can't see any real difference between the states (even the friendly name is the same)

[trizmark]

It nearly works. I updated the unique_id generation to create the _2 entities. Disabled the integration, delete the _2 entity and update the original one to have a sensor id with _2.
The only problem is that this update moves the entity from the myenergi integration to 'Sensor', which causes a _3 entity to be created when the integration is re-enabled.

[dshipp]

Is there any update on a fix or workaround for this issue please? My dashboards are currently broken with unavailable entities and I don’t want to lose the historic data.

[trizmark]

Here's my fix that I just tested on a new install of HA.

• Installed 0.0.23, configured it
• Waited for an hour to have historical data (to verify that I haven't lost anything)
• Upgraded to 0.0.24, got the _2 entities

The fix:

• Stop HA
• In [HA_root_dir]/.storage directory make a backup of the core.entity_registry file
• Remove the _2 entires from the core.entity_registry
• Add the -hub to the unique id of the original entities
• Start HA

After upgrading to 0.0.24, I had these entities:

grep myenergi_my_home core.entity_registry 
        "entity_id": "sensor.myenergi_my_home_hub_firmware",
        "entity_id": "sensor.myenergi_my_home_hub_serial_number",
        "entity_id": "sensor.myenergi_my_home_power_grid",
        "entity_id": "sensor.myenergi_my_home_power_export",
        "entity_id": "sensor.myenergi_my_home_power_import",
        "entity_id": "sensor.myenergi_my_home_voltage_grid",
        "entity_id": "sensor.myenergi_my_home_frequency_grid",
        "entity_id": "sensor.myenergi_my_home_grid_import_today",
        "entity_id": "sensor.myenergi_my_home_grid_export_today",
        "entity_id": "sensor.myenergi_my_home_green_energy_today",
        "entity_id": "sensor.myenergi_my_home_generated_today",
        "entity_id": "sensor.myenergi_my_home_power_generation",
        "entity_id": "sensor.myenergi_my_home_power_charging",
        "entity_id": "sensor.myenergi_my_home_home_consumption",
        "entity_id": "sensor.myenergi_my_home_hub_firmware_2",
        "entity_id": "sensor.myenergi_my_home_hub_serial_number_2",
        "entity_id": "sensor.myenergi_my_home_power_grid_2",
        "entity_id": "sensor.myenergi_my_home_power_export_2",
        "entity_id": "sensor.myenergi_my_home_power_import_2",
        "entity_id": "sensor.myenergi_my_home_voltage_grid_2",
        "entity_id": "sensor.myenergi_my_home_frequency_grid_2",
        "entity_id": "sensor.myenergi_my_home_grid_import_today_2",
        "entity_id": "sensor.myenergi_my_home_grid_export_today_2",
        "entity_id": "sensor.myenergi_my_home_green_energy_today_2",
        "entity_id": "sensor.myenergi_my_home_generated_today_2",
        "entity_id": "sensor.myenergi_my_home_power_generation_2",
        "entity_id": "sensor.myenergi_my_home_power_charging_2",
        "entity_id": "sensor.myenergi_my_home_home_consumption_2",

When deleting the _2 entities, make sure that you get the whole section of each entity. E.g

{
  "aliases": [],
  "area_id": null,
  "capabilities": null,
  "config_entry_id": "30e6b38d40eb3670b719cb77ca55f9c5",
  "device_class": null,
  "device_id": "b5bf0a7b065aa55cb310ff719481cf61",
  "disabled_by": null,
  "entity_category": null,
  "entity_id": "sensor.myenergi_my_home_hub_serial_number_2",
  "hidden_by": null,
  "icon": null,
  "id": "5663ed28393a85d39b4ff5b331f5d0ab",
  "has_entity_name": false,
  "name": null,
  "options": {
    "conversation": {
      "should_expose": false
    }
  },
  "original_device_class": null,
  "original_icon": "mdi:numeric",
  "original_name": "myenergi My home Hub serial number",
  "platform": "myenergi",
  "supported_features": 0,
  "translation_key": null,
  "unique_id": "30e6b38d40eb3670b719cb77ca55f9c5-hub-24047164-serial_number",
  "previous_unique_id": null,
  "unit_of_measurement": null
},

Unique ID update example, from:
"unique_id": "30e6b38d40eb3670b719cb77ca55f9c5-24047164-energy_green",
to
"unique_id": "30e6b38d40eb3670b719cb77ca55f9c5-hub-24047164-energy_green",

You've got 14 updates to make.
Is it pretty? No. Does it work? Yes.

[LeiChat]

Thanks @trizmark.
What tool do you use to make the backup and changes please?
I have Studio Code installed but that only gives access to the config files.

[trizmark]

@LeiChat I'm running HA in a container and I've got the /config directory mapped from the host OS. In my case I simply copied the file with cp and edited it vi (very old-school, I know).
Are you running HAOS and got the VSCode plugin installed?

[LeiChat]

I am, yes.
Perhaps I need to use Terminal rather than VSCode.

[trizmark]

Hmmm... I only used HAOS briefly, but looking at the very first screenshot here, it looks like .storage is visible.

This post in that topic suggests that you can show the hidden files/dirs.

[LeiChat]

Thank you.
Clicking the little X to delete the **/.storage pattern from the "Files: Exclude" list made the folder appear in the Explorer. 👍

[LeiChat]

Oops. You of course cannot use VSCode when HAOS Core is stopped. 🤦‍♂️
Will remind myself how to use PuTTY. Thanks for your help.

[lozzd]

You've got 14 updates to make. Is it pretty? No. Does it work? Yes.

Perfect! it did work. Now I am up to date with my historical data. Thank you.
And I also learnt something about HA data structure/storage too so even better.

[NJTrance]

same issue, rolled back to 0.0.23

[MattCheetham]

@trizmark

The fix:

• Stop HA

Been trying to do this but I'm running HAOS. I'm using the VSCode plugin and have found and edited the file etc but I can't work out how to stop HA so I can edit this before resuming. Editing while running just results in it getting overwritten on reboot.

Edit: Well, I did it but I probably don't recommend what I did. I edited the file in VSCode while it was still running, saved the file then pulled the power from the Pi instead of using shutdown (Figured out that part of shutdown is overwriting that file). When I rebooted it stuck and is now working as expected.

[mwiseler]

This just took me 3 hours of tinkering to get it working, but thank you to @trizmark for posting the fix.