MQTT sensor and history recorder

Configuration
1

Hi,

I have an external Python script which query the Netatmo API to get the rain status 5km around my home.
If it´s raining the status will be set to True, otherwise to False. In addition I query the “rain value”.
I send this information every minute via MQTT sensor to Home Assistant. This is my configuration:

sensor:
  # Netatmo Custom Rain sensor via API
  - platform: mqtt
    name: "NetatmoAPIRainValue"
    state_topic: "home-assistant/netatmo/rainvalue"
    unit_of_measurement: "mm"
  - platform: mqtt
    name: "NetatmoAPIRainStatus"
    state_topic: "home-assistant/netatmo/rainstatus"

This works quite good, but there is one problem:
Every minute the status will be sent to HA and every time HA recognize a status change and set the status again, even if the status is the same as before.
E.g. if it´s not raining the Python script is sending False every minute. And every minute there is a new record in the HA history.
For other sensors only a real status change is recorded, but in this case every status, even if it´s the same will be recorded.
If I open the history of this sensor it takes a few minutes until it is available because there are thousands of status changes in it for only a few days. But actually the status changed only a few times from True to False and the other way around.

Is this a normal behavior of HA ?
Any ideas how to optimize this?

Thank you!

2

Have your python script not publish if the state has not changed :wink:

3

Yes, that could be a solution and I also thought about that.
But that would mean that I need to store the values (at least the last value) on the script side, but I want to avoid this as I want it to keep it as simple as possible.

4

In your script you could check what the last message was that has been published to mqtt and if it differs publish, else don’t publish.

5

I cannot replicate this behavior.

I created this simple MQTT Sensor:

- platform: mqtt
  name: Test MQTT
  unique_id: test_mqtt
  state_topic: test

After reloading MQTT entities, it appeared in Developer Tools > States with a value of unknown (which is normal). I then published multiple consecutive true payloads, followed by multiple consecutive false payloads and then multiple consecutive true payloads again.

If it were to behave as you described, I should see at least a dozen entries in the database, one entry for each time I published a payload. However, the database only contains four entries, just once for each time the sensor’s state changed value (and none for when the published value was identical to the sensor’s existing value).

Screenshot from 2021-09-28 09-26-16
Screenshot from 2021-09-28 09-26-161332×648 75.7 KB

I performed the test using version 2021.09.6 and published payloads using both MQTT Explorer and Home Assistant’s MQTT Publish service.

I don’t know why your system is recording every single value, even if when it’s the same value, but it’s not reproducible on my system. What version of Home Assistant are you using?

6

Hi Taras,

I just made the same test as you and can confirm your results. I set the status several times to the same value via Developer Tools > States and only one event was logged in the database.

But, if I set the status via MQTT sensor it´s different and I get an event for every status I set, even it is the same.

If it helps, this is my Python script which runs every minute (triggered outside of HA as a Scheduled Task in my Synology):

#!/usr/bin/python3
import time
import random
import paho.mqtt.client as mqtt
import paho.mqtt.publish as publish
import patatmo
import re

def Average(lst):
    return sum(lst) / len(lst)

broker = 'diskstation'
delay = 10

credentials = {
    "password":"xxx",
    "username":"[email protected]",
    "client_id":"<netatmo client id>",
    "client_secret":"<netatmo secret>"
}

authentication = patatmo.api.authentication.Authentication(
    credentials=credentials,
)

client = patatmo.api.client.NetatmoClient(authentication)

coord_region = {
    "lat_ne" : 51.3249,
    "lat_sw" : 51.2872,
    "lon_ne" : 9.5960,
    "lon_sw" : 9.5271,
}

region_data = client.Getpublicdata(region = coord_region)

publish.single('home-assistant/netatmo/rainvalue', '0.0', hostname=broker)
publish.single('home-assistant/netatmo/rainstatus', '0.0', hostname=broker)

client = mqtt.Client("ha-client")
client.connect(broker)
client.loop_start()

# parse the response
region_str = str(region_data)
tmp_result_list = re.findall('(?<=rain_live\D\D\D)\d*\.*\d+', region_str)
	
result_list = [float(i) for i in tmp_result_list]
average = Average(result_list)
print('Average: ' + str(average))
publish.single('home-assistant/netatmo/rainvalue', str(average), hostname=broker)
if average > 0:
     print('Status: True')
     publish.single('home-assistant/netatmo/rainstatus', 'True', hostname=broker)
else:
     publish.single('home-assistant/netatmo/rainstatus', 'False', hostname=broker)
     print('Status: False')
7

Being intrigued, I checked HA source code and it explicitly, at high level so basically for everything, do not write a new state if state and attributes are the same as the previous one.

Puzzling…

8

Why does the python script attempt to publish 0.0 to two topics before it connects to the broker?

publish.single('home-assistant/netatmo/rainvalue', '0.0', hostname=broker)
publish.single('home-assistant/netatmo/rainstatus', '0.0', hostname=broker)

client = mqtt.Client("ha-client")
client.connect(broker)
client.loop_start()

#... more code ...


publish.single('home-assistant/netatmo/rainvalue', str(average), hostname=broker)
if average > 0:
     print('Status: True')
     publish.single('home-assistant/netatmo/rainstatus', 'True', hostname=broker)
else:
     publish.single('home-assistant/netatmo/rainstatus', 'False', hostname=broker)
     print('Status: False')
9

Just like to point out, if the rainstatus is only ever True or False, (On or Off) - this should be a binary_sensor and not just a sensor. I made this terrible mistake a year back. Sensor should be for stuff that can have a range of values. binary_sensor for stuff that is just on or off.

10

Don’t be so hard on yourself; it’s not a terrible mistake. :wink:

The advantage of making it a binary_sensor is that binary_sensors have a wider choice for device_class (which simplifies the translation of its state values for display in the Lovelace UI).

11

I think you are right!
These two lines are in because of some initial tests, but it seems to be that I never removed them. With these lines the status is of course always different because status is set to 0.0 and then to True/False. With the next run the same again.

Thanks a lot! I totally overlooked that!

12

Glad to hear it helped to solve the problem.

My next question was going to be why doesn’t the python script formally disconnect from the broker after it has finished publishing its payloads?

13

Good catch :slight_smile: