Allow mqtt_statestream component the ability to include/exclude entities

Feature Requests
1

It would be great if there was a way to restrict which entities have their state changes published via MQTT.

I have some sensors that are updated once per second (which report power usage for a number of branch circuits.) This data can consume a lot of space in the recorder database, so I’ve excluded it there. However, when publishing these state changes, I also get events logged in the databased as a consequence of the MQTT activity like this:

sqlite> select * from events order by created desc limit 20;
16080636|service_executed|{"service_call_id": "4410591664-162162"}|LOCAL|2017-10-18 13:18:49.538304|2017-10-18 13:18:49.541744
16080635|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_total/state", "payload": "1023.0", "retain": true, "qos": 1}, "service_call_id": "4410591664-162162"}|LOCAL|2017-10-18 13:18:49.533652|2017-10-18 13:18:49.539218
16080634|service_executed|{"service_call_id": "4410591664-162161"}|LOCAL|2017-10-18 13:18:48.553081|2017-10-18 13:18:48.565953
16080633|service_executed|{"service_call_id": "4410591664-162160"}|LOCAL|2017-10-18 13:18:48.551799|2017-10-18 13:18:48.561557
16080632|service_executed|{"service_call_id": "4410591664-162159"}|LOCAL|2017-10-18 13:18:48.550754|2017-10-18 13:18:48.559761
16080631|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_total/state", "payload": "1024.0", "retain": true, "qos": 1}, "service_call_id": "4410591664-162161"}|LOCAL|2017-10-18 13:18:48.546971|2017-10-18 13:18:48.558104
16080630|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_fridge/state", "payload": "115.0", "retain": true, "qos": 1}, "service_call_id": "4410591664-162160"}|LOCAL|2017-10-18 13:18:48.546802|2017-10-18 13:18:48.556324
16080629|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_volts/state", "payload": "123.2", "retain": true, "qos": 1}, "service_call_id": "4410591664-162159"}|LOCAL|2017-10-18 13:18:48.540044|2017-10-18 13:18:48.549988
16080628|service_executed|{"service_call_id": "4410591664-162158"}|LOCAL|2017-10-18 13:18:47.557605|2017-10-18 13:18:47.568535
16080627|service_executed|{"service_call_id": "4410591664-162157"}|LOCAL|2017-10-18 13:18:47.555074|2017-10-18 13:18:47.566540
16080626|service_executed|{"service_call_id": "4410591664-162156"}|LOCAL|2017-10-18 13:18:47.552531|2017-10-18 13:18:47.562759
16080625|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_total/state", "payload": "1026.0", "retain": true, "qos": 1}, "service_call_id": "4410591664-162158"}|LOCAL|2017-10-18 13:18:47.546467|2017-10-18 13:18:47.560990
16080624|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_fridge/state", "payload": "114.0", "retain": true, "qos": 1}, "service_call_id": "4410591664-162157"}|LOCAL|2017-10-18 13:18:47.546284|2017-10-18 13:18:47.558916
16080623|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_volts/state", "payload": "123.3", "retain": true, "qos": 1}, "service_call_id": "4410591664-162156"}|LOCAL|2017-10-18 13:18:47.542724|2017-10-18 13:18:47.549915
16080622|service_executed|{"service_call_id": "4410591664-162155"}|LOCAL|2017-10-18 13:18:46.538614|2017-10-18 13:18:46.541959
16080621|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_total/state", "payload": "1014.0", "retain": true, "qos": 1}, "service_call_id": "4410591664-162155"}|LOCAL|2017-10-18 13:18:46.533674|2017-10-18 13:18:46.539276
16080620|service_executed|{"service_call_id": "4410591664-162154"}|LOCAL|2017-10-18 13:18:45.545590|2017-10-18 13:18:45.552038
16080619|service_executed|{"service_call_id": "4410591664-162153"}|LOCAL|2017-10-18 13:18:45.544014|2017-10-18 13:18:45.550371
16080618|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_total/state", "payload": "1023.0", "retain": true, "qos": 1}, "service_call_id": "4410591664-162154"}|LOCAL|2017-10-18 13:18:45.540931|2017-10-18 13:18:45.548561
16080617|call_service|{"service": "publish", "domain": "mqtt", "service_data": {"topic": "19916/homeassistant/sensor/power_volts/state", "payload": "123.2", "retain": true, "qos": 1}, "service_call_id": "4410591664-162153"}|LOCAL|2017-10-18 13:18:45.536683|2017-10-18 13:18:45.545137

In the mean time, I’ve had to disable the MQTT eventstream component to avoid generating an extra gazillion events per day.

Ideally, the publishing of these MQTT messages wouldn’t need to be inserted into the events table of the recorder database, and I’d like to be able to control which entities are published via MQTT… In my circumstance, these power measurements are from MQTT sensors, so I don’t need to publish them again on a different topic.

2

As a follow-up to this… I think it would be great if the mqtt_statestream component could somehow manage to transmit MQTT message without the service invocation being logged in the events table of the recorder database. If you have any sort of real time rapidly changing data (like power measurements in my case), there’s a lot of traffic getting logged into the database. In my case, I was seeing about 1,000,000 events logged per day with the sqlite3 database, only retaining a few days of data over a gigabyte in size.

I think it would still be useful from a debugging standpoint to see service invocation events logged, but all those MQTT service invocations take up needless and excessive space in some circumstances.

1 Like
3

It’s almost unusable right now due to the huge amount of data logged. I even slows down my other events. I get huge delays in turning on the lights for example…

4

any news on this?
According to https://home-assistant.io/components/mqtt_statestream/ limiting the sending of events could be limited to only includes.

Ive set it up like this:

##########################################################################################
# https://home-assistant.io/components/mqtt_statestream/
##########################################################################################

base_topic: homeassistant
include:
  domains:
#    - homeassistant.setup
#    - homeassistant.core
#    - sensor
    - light
#    - switch
#publish_attributes: true
#publish_timestamps: true

and still see all sensors etc fly by.
Only other option is to stop statestreams completely? DOnt know what could be the negative effect of that tbh… probably try and find out. :wink:
as a matter of fact i now see i am completely in limbo. Previously i had a reference to the state streams in my configuration.yaml, and hence i thought to be editing ‘live’ changes to its behavior.
I now see, the reference was commented out, so i can’t understand why HAssio would be doing anything at all. other than the regular mqtt definition i do have:

broker: !secret base_url
port: !port
username: !secret mqtt_username
password: !secret mqtt_password

discovery: true
discovery_prefix: homeassistant

birth_message:
topic: ‘hass/status’
payload: ‘online’
retain: ‘true’
will_message:
topic: ‘hass/status’
payload: ‘offline’

following: https://home-assistant.io/components/mqtt/

would the discovery variable account for this then?

Cheers,
Marius

5

Neither eventstream nor statestream fit my use case, so I opted for a different solution. I’ve switched to using MQTT Discovery to send sensor data from my child Home Assistants to my main Home Assistant. I just add the sensor I want to forward to the automation (shown below) on my child instances and the sensors magically appear on the main instance. The only configuration needed on the main instance is to enable MQTT Discovery. I set the retain flag on the MQTT message that creates the sensor to ensure the sensor always exists when the main instance starts.

In theory, this should work for more than just sensors. See Howto create battery alert without creating a template for every device to see how I’m using a similar method to automatically create sensors for entities with battery_level attributes.

Switches and lights definitely need more than this automation to allow control from the main instance. I’ve only tested creating sensors from state and battery attributes so far, so there might be issues with other device types.

- alias: Sensor from State
  trigger:
    - platform: state
      entity_id:
        - sensor.sensor_1
        - sensor.sensor_2
        - sensor.sensor_3
  condition:
    - condition: template
      value_template: "{{ trigger.to_state.state != trigger.from_state.state }}"
  action:
    - service: mqtt.publish
      data_template:
        topic: "homeassistant/{{ trigger.entity_id | replace('.', '/') }}/config"
        retain: true
        payload: >-
          {
            "device_class": "sensor",
            "name": "{{ trigger.to_state.attributes.friendly_name }}",
            "state_topic": "homeassistant/{{ trigger.entity_id | replace('.', '/') }}/state"
            {%- if trigger.to_state.attributes.unit_of_measurement and trigger.to_state.attributes.unit_of_measurement is defined -%},
            "unit_of_measurement": "{{ trigger.to_state.attributes.unit_of_measurement }}"
            {%- endif %}
          }
    - service: mqtt.publish
      data_template:
        topic: "homeassistant/{{ trigger.entity_id | replace('.', '/') }}/state"
        # retain: true
        payload: "{{ trigger.to_state.state }}"
6

—edit-----

commenting out discovery didnt help, log is still being filled with all websocket publishing as before.
Ive now taken out Statestreams and Mqtt discovery.

53
531206×314 203 KB

and after that, each second multiple entries on the websocket like this:

57
571172×424 361 KB

besides the way the log looks after the latest update, im really worried the system is overloaded.

Do you recognize this as regular business or should i be making changes to, hmm , to what i wonder.

Cheers,
Marius

-----end edit-------

Well, you’ve again made me thinking again …

Since I don’t have mqtt devices connected to the HA directly but receive mqtt-events published over the broker, i think using the discovery won’t be needed, fit my bill as you so eloquently put it :wink:

It might on the contrary very well be the cause for the endless stream of those published events it discovers and resends over the system. I’ll check in later and test what happens when I disable discovery and rely fully on the mqtt sensors i created.

Maybe things will calm down then.
Be in to touch no doubt…

Will test this sensor if only for the elegant solution it brings. This is what templating was designed for …classic.

Thanks
Marius

7

Websocket is different than statestream or eventstrean. You’ve likely got an add-on connecting via the websocket interface that’s causing that traffic.

You should only need statestream or eventstream if you are attempting to connect multiple Home Assistant instances together.

8

I’ve looked into that, but only have MariaDB running for the moment and Homebridge is installed but not running.

Would either of the 2 cause all this traffic (both needing to record all events of course)?

If so, how could i identify the origin of the web socket number. Ive already logged with debug but in the given output no identifier is shown other than the ws [number].

The latter of course is a bit more generic: how to debug the web socket api, and find out which instance corresponds with each web socket [number]

Marius

9

I’m not familiar with Homebridge, but I suspect it because it appears to use the Home Assistant API.

10

Same issue here (If I understand the OP correctly). I only want Switches to be StateStreamed to my main instance from a remote one. And so:

mqtt_statestream:
  base_topic: remotesite3
#  publish_attributes: false
#  publish_timestamps: false
  include:
    domains:
    - switch

My broker sees everything from the remote site. Like power sensors, temperature changes, even the uptime stats for network devices. I only care about the switches!
I’ll try “including” them by entity to see if it makes any difference…

11

Nope. This:

mqtt_statestream:
  base_topic: remotesite3
#  publish_attributes: false
#  publish_timestamps: false
  include:
    entities:
      - switch.wemo_porch_light
      - switch.wemo_lounge_lamp
      - switch.wemo_upstairs_lamp

Tells my broker, in addition to the switch events, that “Home” still exists and that it’s “Partly Cloudy”. Sigh.