How to select State from MQTT topic and Attributes from json?

1

Hello Everyone,
Firstly Happy New Year, and excuse my explanation as I am not a pro at coding etc…

So I have been trying to extract a JSON string from my LoRaWAN gateway into HA
This is the JSON:

{
  "applicationID": "1",
  "applicationName": "HomeAssistant",
  "deviceName": "MNL1 Tracker",
  "devEUI": "****************=",
  "rxInfo": [
    {
      "gatewayID": "LPfxEURAABE=",
      "time": "2024-01-10T02:27:05.205689230Z",
      "timeSinceGPSEpoch": null,
      "rssi": -86,
      "loRaSNR": 12.5,
      "channel": 0,
      "rfChain": 0,
      "board": 0,
      "antenna": 0,
      "location": {
        "latitude": -40.33129247197204,
        "longitude": 175.88169336318973,
        "altitude": 50,
        "source": "UNKNOWN",
        "accuracy": 0
      },
      "fineTimestampType": "NONE",
      "context": "hB4IrA==",
      "uplinkID": "C0TPvG6uQl6biBm4h1HU9A==",
      "crcStatus": "CRC_OK"
    }
  ],
  "txInfo": {
    "frequency": 903900000,
    "modulation": "LORA",
    "loRaModulationInfo": {
      "bandwidth": 125,
      "spreadingFactor": 9,
      "codeRate": "4/5",
      "polarizationInversion": false
    }
  },
  "adr": true,
  "dr": 1,
  "fCnt": 1077,
  "fPort": 5,
  "data": "BwAAAABlngBseEVYSw6J2H5Nj0DXL9GRVPdh2fHIekVYOw6J2wD0ADg+",
  "objectJSON": "{\"err\":-1,\"messages\":[],\"payload\":\"0700000000659E006C7845584B0E89D87E4D8F40D72FD19154F761D9F1C87A45583B0E89DB00F400383E\",\"valid\":false}",
  "tags": {},
  "confirmedUplink": true,
  "devAddr": "AAC9mw=="
}

To start I am trying to extract Latitude with this in my config

- device_tracker:
    name: "MNL1 Tracker"
    state_topic: "application/1/device/2cf7f1c053000409/event/up"
    value_template: "{{ value_json.rxInfo[0].location.latitude }}"

In the states part of developer tab I get unknown.
I have put the JSON into JSONPath online checker as suggested in similar posts and it returns the Latitude correctly.

Could anyone please point me to where I may be going wrong or missing something?
Cheers

2

Not sure what you mean by that, but it works for me like this in the template debugging tab

{% set value_json = {
  "applicationID": "1",
  "applicationName": "HomeAssistant",
  "deviceName": "MNL1 Tracker",
  "devEUI": "****************=",
  "rxInfo": [
    {
      "gatewayID": "LPfxEURAABE=",
      "time": "2024-01-10T02:27:05.205689230Z",
      "timeSinceGPSEpoch": null,
      "rssi": -86,
      "loRaSNR": 12.5,
      "channel": 0,
      "rfChain": 0,
      "board": 0,
      "antenna": 0,
      "location": {
        "latitude": -40.33129247197204,
        "longitude": 175.88169336318973,
        "altitude": 50,
        "source": "UNKNOWN",
        "accuracy": 0
      },
      "fineTimestampType": "NONE",
      "context": "hB4IrA==",
      "uplinkID": "C0TPvG6uQl6biBm4h1HU9A==",
      "crcStatus": "CRC_OK"
    }
  ],
  "txInfo": {
    "frequency": 903900000,
    "modulation": "LORA",
    "loRaModulationInfo": {
      "bandwidth": 125,
      "spreadingFactor": 9,
      "codeRate": "4/5",
      "polarizationInversion": false
    }
  },
  "adr": true,
  "dr": 1,
  "fCnt": 1077,
  "fPort": 5,
  "data": "BwAAAABlngBseEVYSw6J2H5Nj0DXL9GRVPdh2fHIekVYOw6J2wD0ADg+",
  "objectJSON": "{\"err\":-1,\"messages\":[],\"payload\":\"0700000000659E006C7845584B0E89D87E4D8F40D72FD19154F761D9F1C87A45583B0E89DB00F400383E\",\"valid\":false}",
  "tags": {},
  "confirmedUplink": true,
  "devAddr": "AAC9mw=="
}
%}

{{ value_json.rxInfo[0].location.latitude }}

But you probably actually want to use json_attributes_template to extract latitude and longitude

- device_tracker:
    name: "MNL1 Tracker"
    json_attributes_topic: "application/1/device/2cf7f1c053000409/event/up"
    json_attributes_template: "{{ value_json.rxInfo[0].location | tojson }}"
3

Hey Chris,
Thanks for your response.
Ah its called the template debugging tab, thats the same one that I meant but I didnt realise what you had to add to that and why.

I have used the json attributes as you suggested and this works perfectly.
Now I just need to figure out where battery levels are and also the button press functionality of the T1000 trackers we have…

Thank you very much for your help, I sort of understand how that works now…

4

Hi Again,
Could I ask your advice on another sensor I have for temp and moisture please?
Here is the json:

{
  "applicationID": "1",
  "applicationName": "HomeAssistant",
  "deviceName": "S2104 Soil Temp and Moisture",
  "devEUI": "LPfxwFMABxQ=",
  "rxInfo": [
    {
      "gatewayID": "LPfxEURAABE=",
      "time": "2024-01-11T19:45:05.129933141Z",
      "timeSinceGPSEpoch": null,
      "rssi": -76,
      "loRaSNR": 13,
      "channel": 1,
      "rfChain": 0,
      "board": 0,
      "antenna": 0,
      "location": {
        "latitude": -40.33129247197204,
        "longitude": 175.88169336318973,
        "altitude": 50,
        "source": "UNKNOWN",
        "accuracy": 0
      },
      "fineTimestampType": "NONE",
      "context": "IiFGHw==",
      "uplinkID": "Q3qK7pBORSu4zTYLurcrJA==",
      "crcStatus": "CRC_OK"
    }
  ],
  "txInfo": {
    "frequency": 904100000,
    "modulation": "LORA",
    "loRaModulationInfo": {
      "bandwidth": 125,
      "spreadingFactor": 10,
      "codeRate": "4/5",
      "polarizationInversion": false
    }
  },
  "adr": true,
  "dr": 0,
  "fCnt": 2200,
  "fPort": 2,
  "data": "AQcQnHwAAMRr",
  "objectJSON": "{\"err\":0,\"messages\":[{\"measurementId\":4103,\"measurementValue\":31.9,\"type\":\"report_telemetry\"}],\"payload\":\"0107109C7C0000C46B\",\"valid\":true}",
  "tags": {},
  "confirmedUplink": true,
  "devAddr": "AdPX9w=="
}

You will notice that there is an objectJSON item (sorry if thats the wrong term) 4 lines up from the bottom that contains sensor data, in this case a value of 31.9.
How do I go about extracting that data from the JSON?
I am a bit stuck on the syntax required.
Cheers

5 
{% set value_json = {
  "applicationID": "1",
  "applicationName": "HomeAssistant",
  "deviceName": "S2104 Soil Temp and Moisture",
  "devEUI": "LPfxwFMABxQ=",
  "rxInfo": [
    {
      "gatewayID": "LPfxEURAABE=",
      "time": "2024-01-11T19:45:05.129933141Z",
      "timeSinceGPSEpoch": null,
      "rssi": -76,
      "loRaSNR": 13,
      "channel": 1,
      "rfChain": 0,
      "board": 0,
      "antenna": 0,
      "location": {
        "latitude": -40.33129247197204,
        "longitude": 175.88169336318973,
        "altitude": 50,
        "source": "UNKNOWN",
        "accuracy": 0
      },
      "fineTimestampType": "NONE",
      "context": "IiFGHw==",
      "uplinkID": "Q3qK7pBORSu4zTYLurcrJA==",
      "crcStatus": "CRC_OK"
    }
  ],
  "txInfo": {
    "frequency": 904100000,
    "modulation": "LORA",
    "loRaModulationInfo": {
      "bandwidth": 125,
      "spreadingFactor": 10,
      "codeRate": "4/5",
      "polarizationInversion": false
    }
  },
  "adr": true,
  "dr": 0,
  "fCnt": 2200,
  "fPort": 2,
  "data": "AQcQnHwAAMRr",
  "objectJSON": "{\"err\":0,\"messages\":[{\"measurementId\":4103,\"measurementValue\":31.9,\"type\":\"report_telemetry\"}],\"payload\":\"0107109C7C0000C46B\",\"valid\":true}",
  "tags": {},
  "confirmedUplink": true,
  "devAddr": "AdPX9w=="
}
%}

{{ (value_json.objectJSON | from_json).messages[0].measurementValue }}

Key is from_jsom that transforms a “stringified” json object into an actual object that HA understands.

6

Hey Chris,
Thats great, thanks for the help. Is there somewhere that I can try an learn the JSON syntax? I am still struggling with this. The objectJSON part of the message seems to change/cycle through between the following-:

"objectJSON": "{\"err\":0,\"messages\":[{\"measurementId\":4102,\"measurementValue\":28.4,\"type\":\"report_telemetry\"}],\"payload\":\"010610F06E00001251\",\"valid\":true}",

"objectJSON": "{\"err\":0,\"messages\":[{\"measurementId\":4103,\"measurementValue\":33,\"type\":\"report_telemetry\"}],\"payload\":\"010710E8800000FA6A\",\"valid\":true}",

and also another one that I can find now in the history for the battery level… maybe this only updates once a day or something…

Heres what I have got in my mqtt.yaml-:

################################################
### SeeedStudio S2104 Soil Temp and Moisture ###
################################################

sensor:
  - name: "S2104_Battery"
    state_topic: "application/1/device/2cf7f1c053000714/event/up"
    json_attributes_topic: "application/1/device/2cf7f1c053000714/event/up"
    json_attributes_template: "{{ (value_json.objectJSON | from_json).messages[0].battery }}"
  
  - name: "S2104_Soil_Temp_4102"
    state_topic: "application/1/device/2cf7f1c053000714/event/up"
    json_attributes_topic: "application/1/device/2cf7f1c053000714/event/up"
    json_attributes_template: "{{ (value_json.objectJSON | from_json).messages[0].measurementValue }}"
    
  - name: "S2104_Soil_Moisture_4103"
    state_topic: "application/1/device/2cf7f1c053000714/event/up"
    json_attributes_topic: "application/1/device/2cf7f1c053000714/event/up"
    json_attributes_template: "{{ (value_json.objectJSON | from_json).messages[0].measurementValue }}"

and this is what I get looking at the entities…

image
image908×530 21.9 KB

Any ideas?

7

I guess you’ll have to select the proper mesurementId, so

{{ ((value_json.objectJSON | from_json).messages | selectattr('measurementId','eq',4103) | list | first).measurementValue }}
8

Hey Chris,
Thanks for the reply, I think I understand what you are doing there but I am not a programmer… you obviously are :slight_smile:

So am I correct in saying that what this does is looks at the JSON and from messages, selects the attribute ‘Measurement ID’ thats equal to 4103 then looks at its measurementValue (which is the data we want)?

What I dont understand is the list | first part ?

So as an update I tried this but still get unknown in Dev tools/Sates… but I put your code into the template debug and get the data for 4103, so not sure why its not showing the value in Dev tools/states…

Thanks for the help.

9

Correct

Technicality. It transforms what is preceding into a list, then takes the first element.

10

Ah, I see I think.
So it makes a list from the Measurement ID (which only has one item in the list being 4103) then selects it for use?
So you need to use a list even though there is only one item in the list?

Sorry for the questions, trying to learn.

I did find this Template Designer Documentation — Jinja Documentation (3.1.x) yesterday. Which made some sense ish.

11

@koying Hey Chris, am slowly learning but have got a bit stuck back on the trackers that I have been trying to integrate into HA.
I am trying to extract the id number from the events when the button on the tracker gets pressed.
What am I doing wrong with this?

{% set value_json = {
  "deduplicationId": "4f7a3af5-c630-4f73-8761-7b8235fdd8b3",
  "time": "2024-01-25T23:58:00.145856736+00:00",
  "deviceInfo": {
    "tenantId": "52f14cd4-c6f1-4fbd-8f87-4025e1d49242",
    "tenantName": "SenseCAP",
    "applicationId": "4dbc51c8-b854-4964-b499-678bf98b0086",
    "applicationName": "Home Assistant",
    "deviceProfileId": "d1dcbe7b-d2dd-4d41-bcc5-c5515a263dd5",
    "deviceProfileName": "MNL1 Tracker",
    "deviceName": "MNL1 Tracker",
    "devEui": "2cf7f1c053000409",
    "tags": {}
  },
  "devAddr": "5b49faab",
  "adr": true,
  "dr": 3,
  "fCnt": 1674,
  "fPort": 5,
  "confirmed": true,
  "data": "BgAAgABlsvWFCnu0AP2YnkgBDgBkXg==",
  "object": {
    "err": 0,
    "payload": "060000800065b2f5850a7bb400fd989e48010e00645e",
    "valid": true,
    "messages": [
      [
        {
          "measurementValue": [
            {
              "eventName": "Press once event.",
              "id": 8
            }
          ],
          "motionId": 0,
          "type": "Event Status",
          "measurementId": "4200",
          "timestamp": 1706227077000
        },
        {
          "measurementId": "4197",
          "type": "Longitude",
          "motionId": 0,
          "timestamp": 1706227077000,
          "measurementValue": 175.879168
        },
        {
          "measurementId": "4198",
          "timestamp": 1706227077000,
          "motionId": 0,
          "type": "Latitude",
          "measurementValue": -40.329656
        },
        {
          "timestamp": 1706227077000,
          "measurementId": "4097",
          "measurementValue": 27,
          "type": "Air Temperature",
          "motionId": 0
        },
        {
          "measurementId": "4199",
          "timestamp": 1706227077000,
          "motionId": 0,
          "type": "Light",
          "measurementValue": 100
        },
        {
          "timestamp": 1706227077000,
          "measurementValue": 94,
          "type": "Battery",
          "measurementId": "3000",
          "motionId": 0
        }
      ]
    ]
  },
  "rxInfo": [
    {
      "gatewayId": "2cf7f11153100032",
      "uplinkId": 902262466,
      "rssi": -29,
      "snr": 10.8,
      "channel": 2,
      "location": {
        "latitude": -40.32967305958506,
        "longitude": 175.87926864624023
      },
      "context": "z1vMIA==",
      "metadata": {
        "region_config_id": "US_902_928_FSB_2",
        "region_common_name": "US915"
      },
      "crcStatus": "CRC_OK"
    }
  ],
  "txInfo": {
    "frequency": 904300000,
    "modulation": {
      "lora": {
        "bandwidth": 125000,
        "spreadingFactor": 7,
        "codeRate": "CR_4_5"
      }
    }
  }
}
%}

{{ (value_json.objectJSON | from_json).messages[0][0].measurementValue[0].id }}

This will then go into a counter hopefully using the ID payload.

Cheers

12

Well, you don’t have a objectJSON in the JSON you show, you don’t need to parse anything

{{ value_json.object.messages[0][0].measurementValue[0].id }}
13

Hi Chris,
Duh, yes I realised that the objectJSON had changed to just object about 10mins after I posted this.
But thanks for pointing out it didnt need parsing…

14

Hey Chris,
So I thought the next bit would be quite easy but NO never that easy. Following on from getting the syntax correct for the ID I tried the following-:

- id: '1706212772320'
  alias: 2cf7f1c0530003d4_SOS Count
  description: ''
  trigger:
  - platform: state
    entity_id:
    - sensor.2cf7f1c0530003d4_sos_event
  condition:
  - condition: template
    value_template: '{{ trigger.payload_json.object.messages[0][0].measurementValue[0].id
      == 8 }}'
  action:
  - service: mqtt.publish
    metadata: {}
    data:
      qos: 0
      topic: TrackerCount
      payload: one
  - service: counter.increment
    metadata: {}
    data: {}
    target:
      entity_id: counter.soscounterone
  mode: single

Which I was thinking would test to see if the ID was equal to 8 (which is a single button press) then if it was trigger the action, in my case an MQTT message but also increment a counter by one.
I dont get any errors but the counter does not increment.
If I remove the condition, it works but with any ID not just 8.

Sorry to keep asking questions, learning slowly.

15

This is for a MQTT trigger.
You’re using a plain state trigger.

image
image783×482 19.5 KB

16

Sorry, i think my brains fried today. I dont really understand what you mean. I thought i was dealing with an mqtt message with json payload?

17

A MQTT trigger looks like:

  trigger:
    - platform: mqtt
      topic: "living_room/switch/ac"
      payload: "on"
      value_template: "{{ value_json.state }}"

You’re triggering on a state. It’s unlikely that a plain state (which is limited to 255 characters) contains a full JSON payload.

Looking above, you already have (or should) have extracted the value from MQTT in your MQTT sensors, no?
What would be the relation between sensor.2cf7f1c0530003d4_sos_event and the condition?

18

Thanks for the explanation, yes I see what you mean about the MQTT trigger, that’s fine if the state of that sensor can only be one item/attribute?

So, the sensor.2cf7f1c0530003d4_sos_event is a bit misleading I think. You can see from the image below what the JSON payload is when a double press is done on the tracker. The image shows another tracker as well.

image
image1077×263 28.7 KB

This is an actual SOS press. If you single press the button, they have used this as another feature/button but used the SOS event sensor and changed the JSON ID to 8 instead of 7. The image below shows this:
image
image1056×153 20.3 KB

So far I have managed to create an event trigger for that sensor but it then triggers on either a single or double press.
I think its extracting the separate values or comparing the json in an automation that I am stuck on.
Hope that all makes sense.

19

I have also noticed that in the State column, before the button is triggered there are , then these encapsulate the payload when triggered.
I put this into the template checker and it evaluates as false…

image
image807×473 22.1 KB

I am now wondering if the is my problem…

But remove the and it shows true…

20

[] denote an array, so it should be:

value_json[0].id