Dynamic aggregate sensor?

So I added a component to my non-HA system to publish sensor data onto MQTT topics.

EG:
/sensors/livingRoom

I can add this to the configuration.yaml

sensor:
  - platform: mqtt
    name: "Living Room Temperature"
    state_topic: "/sensors/livingRoom"
    unit_of_measurement: '°C'
    value_template: "{{ value_json.value }}"

This, of course works fine and I can add a sensor widget to my dashboard for my living room temp.

However I have around 30 of these sensors and I’d rather not, if I can avoid it, create 30 different sensor config blocks. All the sensors present the same JSON and publish to their appropriate topic. The rest of the system is zero config, if I send any UDP packet containing this form of JSON it will be considered a sensor and handled automagically including being republished on MQTT. I’m hoping to extend that zero configuration feature into HA.

So, is there any way to listen to a topic like:

/sensors/#

And automatically creates sensors based on the values within the JSON. The JSON includes:
The short name, eg: LR
The key, eg: livingRoom (also used for the topic)
The full name, eg: “Living Room”
The type, eg: “float”
The units, eg: “*C”, “%”, “Amps”, “Volts”
The timestamp, eg: 1569146290

Example:

Sep 22 11:19:24 raspberrypi Heating Hub[21767]: MQTT Attempting to publish {"key": "office", "timestamp": 1569147564.1471322, "name": "Office", "shortName": "OF", "type": "float", "value": "20.56", "units": "'C"}    ON topic /sensors/office

If you have control over this aspect then you can leverage MQTT Discovery.

Make your non-HA component publish information about each one of the 30 sensors to Home Assistant’s discovery topic. The default is:

homeassistant/<component>/[<node_id>/]<object_id>/config

This will make the 30 sensors auto-magically appear as 30 entities within Home Assistant.

The documentation is very thorough and contains many examples but here’s the gist of it:

If you were to publish this configuration payload:

{"device_class": "temperature", "name": "Living Room Temperature", "state_topic": "homeassistant/sensor/livingroomtemp/state", "unit_of_measurement": "°C", "value_template": "{{ value_json.value}}" }

to this configuration topic:

homeassistant/sensor/livingroomtemperature/config

then Home Assistant’s MQTT Discovery feature will create a new sensor called sensor.living_room_temperature.

Ah. I came to the same conclusion. Slightly reluctantly as I didn’t want to convert to HA explicitly, but I found a way to add discovery to my hub as an add-on.

In a timer I already update RRD files. So I just tagged discovery in there.

The challenge I’m current facing is “device_class”. If I have a data item which looks like temperature (units ends in C), I can send device_class = “temperature”. The same works for “ends with W” and sending power. The trouble is when I don’t know what class to make it. If I send device_class= “” or device_class=“None” I get a value not allowed error:

Exception in async_discover_sensor when dispatching 'mqtt_discovery_new_sensor_mqtt': ({'name': 'Air Pressue', 'device_class': 'sensor', 'state_topic': '/sensors/airPressure', 'value_template': '{{value_json.value}}', 'unit_of_measurement': 'Hpa', 'platform': 'mqtt'},)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/sensor.py", line 81, in async_discover_sensor
    config = PLATFORM_SCHEMA(discovery_payload)
  File "/usr/local/lib/python3.7/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.7/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python3.7/site-packages/voluptuous/schema_builder.py", line 432, in validate_mapping
    raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: value is not allowed for dictionary value @ data['device_class']

Still working on it. I can omitt device_class if it’s not set, that might work.

Correct. Not all parameters are required.

That worked. So I have discovery component that looks like the below and I now have my several dozen sensors in HA. Haven’t tried using them yet, but it looks good and they have values.

def update_ha_discovery(key, value, units, datum):
    global mqttBridge
    topic = "homeassistant/sensor/{0!s}/config".format(datum.key)
    device_class = ""
    units_of_measure = units
    if datum.units.endswith("C"):
        device_class = "temperature"
        units_of_measure = "°C"
    if datum.units.endswith("W"):
        device_class = "power"
        units_of_measure = "W"

    state_topic = "/sensors/"+datum.key
    nice_name = datum.name

    device_element = ""
    if device_class != "":
        device_element = ', "device_class":"{0!s}"'.format(device_class)

    payload = '{{"name":"{0!s}" {1!s}, "state_topic":"{2!s}",' \
              '"value_template":"{{{{value_json.value}}}}", "unit_of_measurement":"{3!s}"}}'.format(nice_name, device_element, state_topic, units_of_measure)
    mqttBridge.publish_direct(topic, payload)

Is there an easy way to clean out discovered sensors?

During setting this up I published some discovery items to the wrong topic and now I have things like:
bedroom (wrong topic)
bedroom_2 (right topic)

From the documentation:

An empty payload will cause a previously discovered device to be deleted.

Sorry. I had a brain fart, I forgot I can publish messages on the command line to do this. I was thinking I’d have to re-implement the bugged topics and remove the payload (doh).

Just thought I’d mention this didn’t work. I think it is because there were two sensors with the same name attached to the same same state and only differing by the discovery ID. So even though I fired repeated empty payloads at both discovery IDs only one of them would disappear and reappear again as soon as the new discovery message arrived. So _2 would disappear but then reappear. I could not get the non _2 one to go away.

Luckily, as it turned out a “docker-compose down” did not cause HA to save it’s state to the config_entries folder and the sensors all disappeared on restart. So I’m back to where I needed to be.

Now I have to figure out how to deal with “boolean” data types to display ON or OFF instead of 1 or 0. I would have liked to do this at the display/lovelace portion as that is where it belongs, but haven’t figured out how yet.

Still it’s progress towards integrating my current system into HA.

Could you please show an example of a sensor in HA, for which you want to show ON or OFF instead of 1 and 0? Binary Sensors for example are already configured with ON and OFF.

Take a look at the binary sensor component

There is a part in the beginning:

The way these sensors are displayed in the frontend can be modified in the customize section.

Thanks, I think I can get that to work. I currently send all sensors as just “sensor” and try and switch the device class, but I can switch to binarysensor when I see a boolean type.

The next thing I want to tackle is showing the demand states.

I think this one might be trickier as HA doesn’t, that I’m aware of, have a direct analogy.

The first thing is that they should only be displayed if they have not expired. They include a timestamp and an expiry in seconds. They currently have a “key” and a “category” but I intend to add fields like, “inReponseTo” and in the case of heating, “target”. So a demand might look like:

Key: livingRoom
Category: HEATING
InResponseTo: Schedule Presence
Target: 20
Current: 19.6
Timestamp: 123456789
Expiry: 300

Ideally that would show up in a single card called “Heating Demands” until it expires and maybe show something like:
Schedule Presence - Living Room - 19.6 for 20
Schedule Presence - Office - 20.4 for 21
Boiler - Multiple (no target)

EDIT: I’m not in a hurry for this as I have work to do propagating some of that data through the system first.

So at least it’s looking a little bit more like a functioning dash board now:
Imgur

Out of curiosity, were you publishing retained messages to the discovery topics?

I don’t think so. Is this why they disappears on a restart do you think?

Indeed, all discovered sensors disappear on restart, which is an interesting side effect or not using “retain”. I kinda like it. Of course when the discovery messages start arriving it all appears again, history included.

This sort of mirrors how I handle data anyway so all good.

Slightly alarmed to find HA persisting all this data into a SQLlite DB which is already 220Mb in size. That will get out of hand unless it’s purged occasionally. This is why I don’t use a normal database to store sensor data, it will get out of hand. I use fixed size round robin DBs (RRDS).

The database maintains a history of an entity’s states. You can specify what you want or don’t want to be recorded.

For example, here’s my config entry for recorder.

recorder:
  purge_keep_days: 5
  exclude:
    domains:
      - group
      - script
      - automation
      - updater
    entities:
      - sun.sun
      - sensor.all_doors
      - sensor.all_lights
      - sensor.all_appliances
      - sensor.last_boot
    event_types:
      - service_removed
      - service_executed
      - service_registered
      - component_loaded
      - call_service