MQTT JSON Device tracker does not recognise latitude longitude in payload

Hi All,

For quite some time I’ve been using a GPS tracker that communicates over LoRa with The Things Network. This tracker works by sending its location every few minutes when it detects motion by its integrated accelerometer.
To get the location of the tracker visible in Home Assistant I’ve used the following procedure:

  1. TTN is setup to receive the data and communicates this over MQTT, the payload decoder makes sure that the latitude and longitude are reported in JSON format. This is all done in a single message. TTN used to split the attributes (bat, lat, lon, time, rssi, etc. etc.) in separate messages but no longer does this.
  2. Node-RED takes the the latitude and longitude messages from the TTN-MQTT and combines these into a single message and sends it to the internal MQTT server that runs in my home. This so that my Home Assistant instance only needs to connect with one MQTT server.
  3. In Home Assistant I’ve configured a MQTT JSON device tracker that is aimed at the topic that Node-Red forwards the messages to.

These above steps have been working perfectly for about a year or so. Then the battery of the tracker when dead and I needed to replace them. Because I was doing busy doing other things I forgot about this until a few months later. With the new batteries the device shows up in TTN, shows up on the MQTT broker but it doesn’t show in Home-Assistant anymore. I get the following log error:

Skipping update for following data because of missing or malformatted data: 
{"app_id":"yabby","dev_id":"tracker","hardware_serial":"XXXX","port":1,"counter":269,"payload_raw":"XXXX=",

"payload_fields":{"batV":4.75,"fixFailed":false,"heading":45,"inTrip":false,
"latitude":52.XXX,"longitude":5.XXX,"parked":false,"speedKmph":0,"type":"position"},

"metadata":{"time":"2020-06-14T14:37:32.078169156Z","frequency":867.7,"modulation":"LORA","data_rate":"SF11BW125","airtime":823296000,"coding_rate":"⅘",
"gateways":[{"gtw_id":"eui-fcc23dfffe0f0bc2","timestamp":1765017268,"time":"2020-06-13T17:58:24.569087Z","channel":6,"rssi":-112,"snr":-9,"rf_chain":0},{"gtw_id":"mjs-gateway-1","timestamp":3717837036,
"time":"2020-06-14T14:37:32Z",
"channel":0,"rssi":-120,"snr":-12.75,"rf_chain":0}],"latitude":52.XXX,"longitude":5.XXX,"location_source":"registry"}}

I did some research and it seems that TTN stopped posting the different attributes of the message in a separate topic. This means that I only get a single message that looks like this (privacy sensitive data randomised):

{
  "batV": 4.75,
  "fixFailed": false,
  "heading": 135,
  "inTrip": true,
  "latitude": 52.6586387,
  "longitude": 5.8706059,
  "parked": false,
  "speedKmph": 0,
  "type": "position"
}

When I look into the MQTT JSON documentation I would think that this would work as the JSON message includes the latitude and longitude. Am I missing something? Or is there something else I need to change to get things working again? Would be really nice to now also see the history of movements which was added in one of the recent Home Assistant updates.

How does the topic look that you are subscribing?

This is what it looks like when I subscribe to the the topic in the developer tools window:

Turns out that this issue is caused by TTN stopping with posting the different payload fields as separate messages:

Identified - For performance reasons we unfortunately had to disable publishing Uplink Fields to MQTT in our EU region. This means that the values from payload decodes are no longer published to individual [AppID]/devices/[DevID]/up/[Field] topics. Until this is resolved, you can subscribe to [AppID]/devices/[DevID]/up and read the payload_fields field from the uplinks you receive there.
Nov 18, 11:00 CET

So now I need to figure out how I can have Node-Red extracting these attributes so that it is readable by Home-Assistant.