white list not working

[btenney100]

I have enabled the whitelist and added one id. it still finds 30 other devices over a week.

Thanks in advance, I'm sure I am missing something simple.

[scuppasteve]

I agree, i would appreciate an example from someone of a working whitelist format. So i can stop adding so many garbage devices.

[thejeffreystone]

My Whitelist looks like this. And when enabled should only send the Devices with the ids in the Whitelist field. If I don't put anything, and turn on debug, then every time it reads a device it puts a note that the device is not in the whitelist.

[scuppasteve]

My Whitelist looks like this. And when enabled should only send the Devices with the ids in the Whitelist field. If I don't put anything, and turn on debug, then every time it reads a device it puts a note that the device is not in the whitelist.

How does that ID work, where does it come from. my devices come from a state topic of rtl_433/Acurite-Tower/4726/A/ are you saying the whitelist ID for this item would be 4726?

[FinestDice17]

did you ever figure this out, because im reading my electric and gas meters with this but am also getting others, would like to filter out just my 2 meters. What id is this looking for?

[btenney100]

I still can not get the white list to work. I tried the example given, but it still reads all

[FinestDice17]

I still can not get the white list to work. I tried the example given, but it still reads all

I got mine working, can you paste a copy of your Mqtt data coming over and I might be able to help you locate your id.

[thejeffreystone]

It reads all in th log or they end up in Home Assistant? It is going to read all of it in the log. I guess we could filter those out.
But if its not in the whitelist it should print a line that its not in the white list.

[thejeffreystone]

Think I found a bug. Trying to squash it now.

[btenney100]

Thanks all for helping with this. I will get a copy of my log files.

On a side note, This is so bulletproof that I have abandoned my other zigbee temp / humidity sensors because this is so reliable compared to them. The only problem I ever have is when the batteries die in my sensors. Thanks again! Awesome.

[FinestDice17]

Here is how i have mine configured which is now working flawlessly at only sending over data to HA of my gas and electric meters only.....

::::::::Starting RTL_433 with parameters::::::::
[05:45:56] INFO: MQTT Host = "mqtt host ip address"
[05:45:56] INFO: MQTT port = 1883
[05:45:56] INFO: MQTT User = "optional"
[05:45:56] INFO: MQTT Password = "optional"
[05:45:56] INFO: MQTT Topic = rtl_433
[05:45:56] INFO: MQTT Retain = true
[05:45:56] INFO: PROTOCOL = -R 149 -R 160 -R 161
[05:45:56] INFO: FREQUENCY = -f 915m -f 912m
[05:45:56] INFO: Whitelist Enabled = true
[05:45:56] INFO: Whitelist = 37823822 15093018
[05:45:56] INFO: Expire After = 0
[05:45:56] INFO: UNITS = si
[05:45:56] INFO: DISCOVERY_PREFIX = homeassistant
[05:45:56] INFO: DISCOVERY_INTERVAL = 300
[05:45:56] INFO: DEBUG = true
#################################################################################
Received message: rtl_433/events : {"time": "2023-02-01 05:45:57-0500", "protocol": 149, "model": "ERT-SCM", "id": 39059630, "physical_tamper": 0, "ert_type": 4, "encoder_tamper": 1, "consumption_data": 6769, "mic": "CRC", "mod": "ASK", "freq": 915.07885, "rssi": -4.69423, "snr": 22.13518, "noise": -26.8294}
Device Id:39059630 not in whitelist
Received message: rtl_433/events : {"time": "2023-02-01 05:45:57-0500", "protocol": 149, "model": "ERT-SCM", "id": 37823822, "physical_tamper": 0, "ert_type": 4, "encoder_tamper": 1, "consumption_data": 15664, "mic": "CRC", "mod": "ASK", "freq": 915.10074, "rssi": -3.54561, "snr": 23.41345, "noise": -26.9591}
Received key time for ERT-SCM
Received key freq for ERT-SCM
Received key rssi for ERT-SCM
Received key snr for ERT-SCM
Received key noise for ERT-SCM

For Device Id:39059630 its receiving the message but but publishing it to HA
For Device Id:37823822 you see its sending Received messages for :
Received key time for ERT-SCM
Received key freq for ERT-SCM
Received key rssi for ERT-SCM
Received key snr for ERT-SCM
Received key noise for ERT-SCM

hope this helps a little.
Thank you @thejeffreystone for your work on this!

[scuppasteve]

I still don't get the whitelist working

Received message: rtl_433/events : {"time": "2023-02-06 07:41:10-0800", "protocol": 40, "model": "Acurite-Tower", "id": 1946, "channel": "A", "battery_ok": 1, "temperature_F": 72.14, "humidity": 40, "mic": "CHECKSUM", "mod": "ASK", "freq": 433.94515, "rssi": -0.118229, "snr": 24.17268, "noise": -24.2909}

I am going to assume my ID for this is 1946.

I have entered all of those in the whitelist with spaces between them, and other devices keep getting picked up. It is a PITA to have to keep cleaning up the created devices.

[thejeffreystone]

Yeah, the SDR to MQTT isn't respecting the whitelist. I haven't had a chance to get it all fixed.

[scuppasteve]

I know beggars can't be choosers, but this causes a huge knock on effect of adding so many entities that it crashes Node-Red due to so many entities in the system. So every like 3 months i have to delete like 2000 devices. Thanks for all the work so far.

[thejeffreystone]

Yeah, I get it. I've been working on updates to both versions. Hoping to have those up there this week. I need to do some more testing to make sure I don't make things worse.
I think going forward I will put beta versions up that have the changes in them so everyone could help test since I dont have all the sensors, or setups.

[btenney100]

Beta is a great idea. Would love to help test and provide feedback. I have 42 sensors. Thanks for all the work.

A

[thejeffreystone]

Ok, I pushed an update. Lots of changes that Ive been working on and testing with my setup. Finger crossed nothing breaks.

I tried to clean up the whitelist logic. And I tested it on a new system and MQTT server. It appears to work at least in terms of Home Assistant. When whitelist is on, it reports in the log that it received data from a device, but didn't send it to Home Assistant. No new devices were created in home assistant that were not in the whitelist.

But those devices do still end up in the MQTT server since that is done by the rtl_433 program I am using. If we want to prevent those entities from making it to the MQTT server then I will need to write a wrapper for the RTL_433 program and handle the sending to MQTT part.

[scuppasteve]

Awesome, thank you i will try it out. I don't know that i care about it going through MQTT, i already have lots of traffic there that i ignore. More of an issue with entity creation in my mind. Thank you for being response and the work you have put in.

[thejeffreystone]

Ok, I think the whitelist problem is not going to be fixed by anything I added...or at least once the device has been pushed to the home assistant topic in the mqtt server it's there. From that point on, unless it is removed from the home assistant topic, and removed from HA it will continue to get updates even after the whitelist has been enabled.

Because the whitelist only prevents auto discovery from happening.

So I need to rethink the whitelist setup.

[jn3va]

The problem I was having was I initially did the setup without a whitelist, mqtt_retain ON, and with expire_after set to 0. Over year running I had a collection of about 100 weather devices from neighbors etc. I whitelisted my devices and then deleted them in HomeAssistant but they started to come back since I was running with mqtt_retain turned on. The steps I used to clear these out and keep them from coming back...

1. Get all of the ID for the devices you want to keep and build your whitelist: i.e.:

3126 855 26 120

2. Populate the whitelist field, turn on whitelist and then save (Optionally set expire_after to 86400 i.e. one day, or something other than 0)
3. Open the "Acurite to Home Assistant" add-in and turn off watchdog and turn off start on boot and then stop the add in.
4. Open the MQTT Explorer tool (https://mqtt-explorer.com/ )
5. Expand the rtl_433 tree and one by one, select the devices that you want to remove. You can select a device (e.g. Acurite-5n-1) and purge everything below, but if any of the devices you want to keep are in that tree, expand the device and select the devices one-by-one.... As you select a device or a branch of the treee, at the top of the interface is a trash-can icon. Click that and MQTT-Explorer will send empty retain messages for each object in the tree you have selected. They items in the tree will not be deleted, but the values stored for each key will be cleared.

In the example, if you want to delete ALL devices below Acurite-6054M select Acurite-6054M. If you just want to delete a specific device e.g. 120, select 120. In my screenshot, 120 was actually the device I needed to keep, so I had to manually select and delete about 10 others.

6. After you have all of the items you do not want purged, restart the mosquito broker.
7. Reconnect to MQTT-Explorer and only the items you want to keep should remain under the rtl_433 tree
8. Go into Home Assistant and view the MQTT devices. One by one, select and then delete the devices that you want to purge (its tedious!)
9. Open the "Acurite to Home Assistant" add-in, re-enable watchdog and start on boot and the start the add-in

[scuppasteve]

The problem I was having was I initially did the setup without a whitelist, mqtt_retain ON, and with expire_after set to 0. Over year running I had a collection of about 100 weather devices from neighbors etc. I whitelisted my devices and then deleted them in HomeAssistant but they started to come back since I was running with mqtt_retain turned on. The steps I used to clear these out and keep them from coming back...

1. Get all of the ID for the devices you want to keep and build your whitelist: i.e.:

3126 855 26 120

2. Populate the whitelist field, turn on whitelist and then save (Optionally set expire_after to 86400 i.e. one day, or something other than 0)
3. Open the "Acurite to Home Assistant" add-in and turn off watchdog and turn off start on boot and then stop the add in.
4. Open the MQTT Explorer tool (mqtt-explorer.com )
5. Expand the rtl_433 tree and one by one, select the devices that you want to remove. You can select a device (e.g. Acurite-5n-1) and purge everything below, but if any of the devices you want to keep are in that tree, expand the device and select the devices one-by-one.... As you select a device or a branch of the treee, at the top of the interface is a trash-can icon. Click that and MQTT-Explorer will send empty retain messages for each object in the tree you have selected. They items in the tree will not be deleted, but the values stored for each key will be cleared.

In the example, if you want to delete ALL devices below Acurite-6054M select Acurite-6054M. If you just want to delete a specific device e.g. 120, select 120. In my screenshot, 120 was actually the device I needed to keep, so I had to manually select and delete about 10 others.

6. After you have all of the items you do not want purged, restart the mosquito broker.
7. Reconnect to MQTT-Explorer and only the items you want to keep should remain under the rtl_433 tree
8. Go into Home Assistant and view the MQTT devices. One by one, select and then delete the devices that you want to purge (its tedious!)
9. Open the "Acurite to Home Assistant" add-in, re-enable watchdog and start on boot and the start the add-in

So are you saying the whitelist works in that setup? I believe the developer doesn't think the whitelist function works.

[jn3va]

Yes - whitelisting works. Its just does not solve the problem of once a device is identified and gets into HA, adding your desired devices to the whitelist does not automatically remove the others. For example, A neighbor has a Acurite 606 temp sensor. My radio is detecting it and the add-in sees it, but because the device ID (112) is not in my whitelist, its not passing that in via MQTT so its not being re-added back into my HA devices.

From my add-in's log I see the devices its finding but ignoring (but also letting me know in case I missed one in the whitelist):

INFO:Device Id:112 Model: Acurite-606TX not in whitelist. Add to the Whitelist to create device in Home Assistant.

I think what Jeff meant about "not working" was his code change was not going to be able to auto-remove the already added devices.

@thejeffreystone what I'd suggest is to have people start with whitelisting enabled and a blank list. Then watch logs and see what devices are found. Maybe you could add a toggle in the config that when enabled, would add some output from of the device's measurement data on the line in the log, like the temperature, or battery, or RSSI to help people determine which device is theirs and they want to be added to the whitelist. e.g.

INFO:Device Id:112 Model: Acurite-606TX, Temp: 62.5 Battery: 100 RSSI -9.1dB Add to the Whitelist to create device in Home Assistant.
INFO:Device Id:855 Model: Acurite-5n1, Temp: 62.5 Battery: 100 RSSI -0.13dB Add to the Whitelist to create device in Home Assistant.

The 855 is my device; 112 is not. I know my device is an Acurite-5n1. If there were multiple of the same detected because a neighbor had one, knowing the temp etc. its sending to compare to the Acurite display can help know which to use/keep.

[scuppasteve]

I have all of my sensors spelled out manually in an mqtt.yaml file because i was try to stop this. I have every one of those sensors in my whitelist in this addon. Yet i keep getting new devices added. I have stopped the addon, and wiped every one of the sensors with Mqtt explorer. Since my sensors are hardcoded, is there a reason i cant just delete everyone of these devices from Home Assistant? Mine will get readded as sensors the second i reenable the addon. I have the whitelist fully enabled with every sensor listed, this should work, correct?

[jn3va]

Could you post a screenshot of your add-in config?  That might help.  The lower half of mine is posted below as an example if that helps you troubleshoot. 

I'm not sure about the relationship of your hard coded mqtt entities and the deletion of them all (I assume you mean via using the HA UI); I assume they would come back, but you should make sure to do a full backup before you try it just in case. 

Hard coding the sensors would not have any effect to stop the creation of the other entities by the add-in; hard coding is basically just saving the add-in a step.   You should not have to hard code them since the add-in would create them dynamically as they are detected.     The whitelist tells the add-in to only try and create (and send MQTT updates) for devices that match those included in the whitelist .  

ALSO - When I was taking a screenshot I noticed that there is auto_discovery toggle.   The add-in's info page does not say what this is for, but I assume it is intended to enable or disable mqtt auto-creation of entities. If you turn that off, maybe it will not auto-create any entities - in that case you would have to hard code the entities as you describe.   You'd have to wait for Jeffrey to provide some input on that to confirm for sure.

[jn3va]

p.s. On my whitelist, the items like 0d495c and 8550d495c are for soil moisture sensors (non-accurite) but the add-in is detecting them nicely when I added the appropriate protocol codes. The accurite IDs are 855 26 and 120

[scuppasteve]

Yeah, so the whitelist still isn't working. I purged all the sensors that had been detected and added, about 600 that were trash. Attached is my config. Yeah my sensors are perm mapped so this had no effect on them. I know i don't have to hard code them, i just get tired of removing sensors, so i generally turn off auto discovery.

[thejeffreystone]

Yeah, I haven't figured out the whitelist yet. It seems to act a bit weird.