Parse Shelly MQTT into a sensor

[safritjuh88 (Marc)]

Hi All,

I want to explore the use of a shelly device (in this particular case a 1PM) to use it as a BLE monitor.

The main goal is, when device within a certain RSSI level of the shelly, do something or set a certain room on that user.

That automation part is yet to come, and willing to sort out myself fist once this date is fed correctly into HA.

the current setup is that mosquito MQTT is installed on HA and MQTT plugin is running.
The shelly device is running the BLE script, and pushing this to HA/MQTT.

All is set so I am receiving the shelly data to HA.

The last part is to get that data into a sensor.

I’ve created a mqtt.yaml

to make the future devices where required, but I am not getting my head around how to parse the data correctly into a sensor.

I’m am not familiar with MQTT yet, but from what I’ve learned the past days is that there are topics.

In this case the topic where the data is sent to:

BLE-Tuin/events/rpc

The json formatted text that is received is as following:

{
    "src": "shellypro1pm-30c6f78116f4",
    "dst": "BLE-Tuin/events",
    "method": "NotifyEvent",
    "params": {
        "ts": 1690899791.78,
        "events": [
            {
                "component": "script:1",
                "id": 1,
                "event": "ble.scan_result",
                "data": [
                    1,
                    "87:84:d9:06:8a:51",
                    -96,
                    "AgEGAwIBohQWAaIBIz9OVbmCsI4heQKizoZMHQ==",
                    ""
                ],
                "ts": 1690899791.78
            }
        ]
    }
}

Basically what my toughts are for now, is to make a sensor based on the mac-adress. in this case 87:84:d9:06:8a:51 and write the actual RSSI level to a entity. in this case -96

Is someone able to ramp me up how to:

• Create a proper sensor. the lucky guess from what I have learned is, that this is some part to go into configuration.yaml but on my end, I included mqtt.yaml so that is clear so far.

mqtt
   sensor:
       - name: "BLE-Tuin"

• How to parse the event data into the correct sensor every time.

I hope I am clear on my question, if not, let me know and I’ll do my best to provide the needed info

As far as it is usefull.
I’m running the 1.0.0-beta6 firmware on the 1PM

[petro]

so, you’re in a unique situation where you have an event stream going to a single topic and you have multiple sensors being published to the same topic.

Your options are:

1. Create a sensor that just holds the last BLE data.
2. Create an automation that separates each event into a separate sensor, 1 for each new mac address.

Regardless what route you want to go, this is not an easy feat, even for an advanced person.

If I were in your situation, I’d go with route #2, but it requires knowledge w/ MQTT discovery and advanced automations.

[petro]

So getting back to this, this automation just might work out of the box for you. It will create a new entity each time one enters the stream.

- alias: Shelly MQTT Event Stream
  id: shelly_mqtt_event_stream
  mode: parallel
  trigger:
    - platform: mqtt
      topic: "BLE-Tuin/events/rpc"
  variables:
    root: "homeassistant"
    domain: "sensor"
    payload: "{{ trigger.payload_json }}"
    events: >
      {% set payload = trigger.payload_json | default({}) %}
      {% if payload %}
        {% set ns = namespace(items=[]) %}
        {% for event in payload.params.events %}
          {% set id, mac, rsi, key, dump = event.data %}
          {% set ns.items = ns.items + [ dict(mac=mac, rsi=rsi, ts=event.ts) ] %}
        {% endfor %}
        {{ ns.items }}
      {% else %}
        []
      {% endif %}
  action:
  - repeat:
      for_each: "{{ events }}"
      sequee:
      - variables:
          mac: "{{ repeat.item.mac }}"
          unique_id: "{{ mac | slugify }}"
          rsi: "{{ repeat.item.rsi }}"
          ts: "{{ repeat.item.ts }}"
          topic: "{{ root }}/sensor/{{ unique_id }}"
          state_topic: "{{ topic }}/state"
      - alias: MQTT Discovery
        service: mqtt.publish
        data:
          topic: "{{ topic }}/config"
          payload: >
            {{
              {
                'name': unique_id,
                'unique_id': unique_id,
                'state_topic': state_topic,
                'value_template': "{{ value_json.state }}",
                'json_attributes_topic': state_topic,
                'json_attributes_template': "{{ value_json.attributes }}",
                'unit_of_measurement': 'rssi',
                'state_class': 'measurement',
              } | to_json
            }}
          retain: True
      - alias: MQTT State Update
        data:
          topic: "{{ state_topic }}"
          payload: >
            {{
              {
                'state': rsi,
                'attributes': {'ts': ts, 'mac': mac},
              } | to_json
            }}
          retain: True

[safritjuh88 (Marc)]

Thanks for this awesome reply and the out of the box solution!

I’ve simply copied and pasted the code in automations.yaml, but there seems to be an error in the actios part.
Will have to sort that out (just for the learning curve)

I will look if I can provide the data from the shelly side in another format, but even that is a steep learning curve. For now I will stick to your #2

[petro]

Can you post the errors?

[safritjuh88 (Marc)]

Ok, I’ve got the automation “working”.
Or at least the errors are gone.

Found a type at sequence and there was no service defined at the second action.
I’ve added mqqt.publish in the service

- alias: MQTT State Update
        service: mqtt.publish #### <-- added
        data:
          topic: '{{ state_topic }}'
          payload: >
            {{
              {
                'state': rssi,
                'attributes': {'ts': ts, 'mac': mac},
              } | to_json
            }}
          retain: True

The sensors are now created succesfully, but I can’t really find the RSSI value within the sensors.

[safritjuh88 (Marc)]

And I got it working…

I’ve made some changes in the automation (RSSI instead of RSI)… bu tin that case you have to replace all entries, and not leave some behind.

For the full record…

- alias: Shelly MQTT Event Stream
  id: shelly_mqtt_event_stream
  trigger:
    - platform: mqtt
      topic: "BLE-Tuin/events/rpc"
  variables:
    root: "homeassistant"
    domain: "sensor"
    payload: "{{ trigger.payload_json }}"
    events: >
      {% set payload = trigger.payload_json | default({}) %}
      {% if payload %}
        {% set ns = namespace(items=[]) %}
        {% for event in payload.params.events %}
          {% set id, mac, rssi, key, dump = event.data %}
          {% set ns.items = ns.items + [ dict(mac=mac, rssi=rssi, ts=event.ts) ] %}
        {% endfor %}
        {{ ns.items }}
      {% else %}
        []
      {% endif %}
  action:
  - repeat:
      for_each: '{{ events }}'
      sequence:
      - variables:
          mac: '{{ repeat.item.mac }}'
          unique_id: '{{ mac | slugify }}'
          rssi: '{{ repeat.item.rssi }}'
          ts: '{{ repeat.item.ts }}'
          topic: '{{ root }}/sensor/{{ unique_id }}'
          state_topic: '{{ topic }}/state'
      - alias: MQTT Discovery
        service: mqtt.publish
        data:
          topic: '{{ topic }}/config'
          payload: >
            {{
              {
                'name': unique_id,
                'unique_id': unique_id,
                'state_topic': state_topic,
                'value_template': '{{ value_json.state }}',
                'json_attributes_topic': state_topic,
                'json_attributes_template': '{{ value_json.attributes }}',
                'unit_of_measurement': 'rssi',
                'state_class': 'measurement',
              } | to_json
            }}
          retain: True
      - alias: MQTT State Update
        service: mqtt.publish
        data:
          topic: '{{ state_topic }}'
          payload: >
            {{
              {
                'state': rssi,
                'attributes': {'ts': ts, 'mac': mac},
              } | to_json
            }}
          retain: True
  mode: parallel

This give’s me the RSSI value in the sensor!

Thanks a lot so far @petro !

[ArnoldV (Arnold)]

Would you mind sharing the script running on the Shelly 1PM? I’m keen to understand how you did this and see if i could use it for some of my ideas.

[safritjuh88 (Marc)]

There are many topics how to setup the MQTT broker and such. Assuming you’ve got that sorted already.

On the shelly I have enabled the default script aioshelly_ble_integration

If it’s not there… this is the code

// aioshelly BLE script 1.0
BLE.Scanner.Subscribe(function (ev, res) {
    if (ev === BLE.Scanner.SCAN_RESULT) {
        Shelly.emitEvent("ble.scan_result", [
            1,
            res.addr,
            res.rssi,
            btoa(res.advData),
            btoa(res.scanRsp)
        ]);
    }
});
BLE.Scanner.Start({
    duration_ms: -1,
    active: false,
    interval_ms: 320,
    window_ms: 30,
});

And the settings in the mqtt tab in the shelly:

[ArnoldV (Arnold)]

Thanks… was looking for the code running on the shelly. This helps.

[derMicro]

Hi, I “borrowed” your automation, but unfortunately my HA responds with a warning and no sensors are created.

My JSON

{
   "src":"shellyplus2pm-90380c35bba8",
   "dst":"BLE_GT_LightSocket/events",
   "method":"NotifyEvent",
   "params":{
      "ts":1693420254.30,
      "events":[
         {
            "component":"script:2",
            "id":2,
            "event":"ble.scan_result",
            "data":[
               1,
               "c0:49:ef:85:2f:02",
               -97,
               "AgEGEP+pCwEFAAsCEArASe+FLwA=",
               ""
            ],
            "ts":1693420254.30
         }
      ]
   }
}

HA-Log is

2023-08-30 20:40:00.709 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'events' when rendering '{% set payload = trigger.payload_json | default({}) %} {% if payload %}
{% set ns = namespace(items=[]) %}
{% for myevent in payload.params.events %}
{% set id, mac, rssi, key, dump = myevent.data %}
{% set ns.items = ns.items + [ dict(mac=mac, rssi=rssi, ts=myevent.ts) ] %}
{% endfor %}
{{ ns.items }}
{% else %}
[]
{% endif %}'

I just can’t see where the error is supposed to be here. HA is 2023.8.4

[safritjuh88 (Marc)]

Can you paste/add your automation?

[derMicro]

Hello, I think it was a bit late, when I just looked at the trace again because I wanted to show this with, I realized the error
root: “homeassistant” must be “homeassistanttest”, because I have a test and prod system. Now the sensors are created.

My conclusion: More coffee or less coffee?
Thank you

[safritjuh88 (Marc)]

When in doubt, always more coffee
Cheers!

[Josemi]

Hi
I am an IT guy, but not a developer
I would like to know how to filter in order to be created in HA only two devices (by known MAC BLE address), because I have had running your script for 1 day and it created more than 3.300 MQTT entities and the database increased a lot.
Would you know how to do that?
Thank you and Merry Christmas.

[simonhoy]

Hi, Josemi,

Being an “IT Guy”, I don’t need to tell you that the first 6 characters of the MAC dictate the manufacturer, so you can filter based on manufacturer quite easily. I have a couple of Shelly buttons so filter out everything that is not by Shelly (Allterco) using this slightly modified version. The IF statement checks to see if the received MAC address contains 90:ab:96. This can be updated with the MAC you are looking for.

BLE.Scanner.Subscribe(function (ev, res) {
    
    if (ev === BLE.Scanner.SCAN_RESULT) {
          
       
        if (res.addr.indexOf("90:ab:96") === 0) { //Checks to see if the Mac address contains "90:ab:96". 
       
              Shelly.emitEvent("ble.scan_result", [
              res.addr,
              res.rssi,
              btoa(res.advData),
              btoa(res.scanRsp)
          ]);
               
         
    } //end of MAC check if statement
        
    }
});

BLE.Scanner.Start({
    duration_ms: -1,
    active: true,
    interval_ms: 320,
    window_ms: 30,
});

I hope this helps.