Parse the things industries cloud storage

I am having a bit of a struggle to parse the json file from thethingsindustries because it returns a weird format.

Based on this api should returns a normal json but upon some investigation is sort of not true ( Storage Integration API | The Things Stack for LoRaWAN )

I wrote a simple python script to get the json and load it which triggered errors of validation, then I discovered that the api returns lines of JSON object with “\n” at the end of it.
The logic to parse this end point would be:

  • Get the data
  • split by “\n”
  • each line needs to be parsed by a json object
  • Extract the json information

I had this simple bit here which is missing a lot of information but I guess the trip is to tell value_template to split and parse? Is there another way or I need to enable mqtt and forget about this endpoint?

rest:
     resource: https://xxxxx.eu1.cloud.thethings.industries/api/v3/as/applications/bedroom-sensor/packages/storage/uplink_message
     headers:
       authorization : 'Bearer xxxxx'
     scan_interval: 60
         value_template: "{{ value_json }}"

Example of two lines of json

{"result":{"end_device_ids":{"device_id":"9876b60000118e9e","application_ids":{"application_id":"bedroom-sensor"},"dev_eui":"xxx","dev_addr":"xxxx"},"received_at":"2022-12-21T06:08:32.225189678Z","uplink_message":{"f_port":1,"f_cnt":1135134,"frm_payload":"mW3bfGBmCH//f65XzExo","decoded_payload":{"bve":0,"c02":936.8896484375,"degreesC":17.49267578125,"gasres":878906.25,"humidity":60.693359375,"iaq":99.951171875,"iaqacc":2.9998779296875,"pressure":99609.375},"rx_metadata":[{"gateway_ids":{"gateway_id":"my-home","eui":"xxxx"},"time":"2022-12-21T06:08:31.940428972Z","timestamp":54855068,"rssi":-66,"channel_rssi":-66,"snr":7.75,"received_at":"2022-12-21T06:08:31.957940284Z"},{"gateway_ids":{"gateway_id":"packetbroker"},"packet_broker":{"message_id":"01GMSMRES1YKPKVS317FW01RQM","forwarder_net_id":"000013","forwarder_tenant_id":"goldeni-p","forwarder_cluster_id":"eu1.cloud.thethings.industries","forwarder_gateway_eui":"xxxx","forwarder_gateway_id":"gi-tti-p-gate-ttog-868-000001","home_network_net_id":"000013","home_network_tenant_id":"xxxx","home_network_cluster_id":"eu1.cloud.thethings.industries"},"time":"2022-12-21T06:08:13.969Z","rssi":-114,"channel_rssi":-114,"snr":-6.5,"received_at":"2022-12-21T06:08:32.031824192Z"}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":8,"coding_rate":"4/5"}},"frequency":"868500000","timestamp":54855068,"time":"2022-12-21T06:08:31.940428972Z"},"received_at":"2022-12-21T06:08:32.020555217Z","consumed_airtime":"0.123392s","version_ids":{"brand_id":"arduino","model_id":"mkr-wan-1310","hardware_version":"1.0","firmware_version":"1.2.0","band_id":"EU_863_870"},"network_ids":{"net_id":"000013","tenant_id":"max246","cluster_id":"eu1","cluster_address":"eu1.cloud.thethings.industries","tenant_address":"xxx.eu1.cloud.thethings.industries"}}}}
{"result":{"end_device_ids":{"device_id":"9876b60000118e9e","application_ids":{"application_id":"bedroom-sensor"},"dev_eui":"xxxx","dev_addr":"xxx"},"received_at":"2022-12-21T06:08:44.555951842Z","uplink_message":{"f_port":1,"f_cnt":1135135,"frm_payload":"mm3afGBm/37/f65XzExo","decoded_payload":{"bve":0,"c02":936.8896484375,"degreesC":17.5048828125,"gasres":874511.71875,"humidity":60.64453125,"iaq":99.951171875,"iaqacc":2.9998779296875,"pressure":99609.375},"rx_metadata":[{"gateway_ids":{"gateway_id":"packetbroker"},"packet_broker":{"message_id":"01GMSMRTSRD8HTYM6WQ02PM388","forwarder_net_id":"000013","forwarder_tenant_id":"goldeni-p","forwarder_cluster_id":"eu1.cloud.thethings.industries","forwarder_gateway_eui":"xxxx","forwarder_gateway_id":"gi-tti-p-gate-ttog-868-000001","home_network_net_id":"000013","home_network_tenant_id":"xx","home_network_cluster_id":"eu1.cloud.thethings.industries"},"time":"2022-12-21T06:08:26.297Z","rssi":-119,"channel_rssi":-119,"snr":-6.5,"received_at":"2022-12-21T06:08:44.342279793Z"},{"gateway_ids":{"gateway_id":"my-home","eui":"xxxx"},"time":"2022-12-21T06:08:44.275168895Z","timestamp":67182708,"rssi":-77,"channel_rssi":-77,"snr":12,"received_at":"2022-12-21T06:08:44.293385277Z"}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":8,"coding_rate":"4/5"}},"frequency":"867100000"},"received_at":"2022-12-21T06:08:44.347624653Z","consumed_airtime":"0.123392s","version_ids":{"brand_id":"arduino","model_id":"mkr-wan-1310","hardware_version":"1.0","firmware_version":"1.2.0","band_id":"EU_863_870"},"network_ids":{"net_id":"000013","tenant_id":"xxx","cluster_id":"eu1","cluster_address":"eu1.cloud.thethings.industries","tenant_address":"xxx.eu1.cloud.thethings.industries"}}}}