The Things Network (TTN) - new adapter for v3

vincen · May 18, 2023, 8:57am

absolutely not needed, never indicated that in post linked above

it has to be put in a bridge.conf file that you’ll save in /share/mosquitto folder in HA If it’s properly done, when you restart the mqtt addon you’ll see in logs that it connects the bridge ! something similar to that:

[10:56:31] INFO: Starting mosquitto MQTT broker...
1684400191: Loading config file /share/mosquitto/bridge.conf
1684400191: Warning: Bridge dragino-ttn-bridge using insecure mode.
1684400191: Warning: Bridge boutons-inputs using insecure mode.
1684400191: Warning: Bridge tracker-tbeam-ttn-bridge using insecure mode.
1684400191: Warning: Bridge trackit-ttn-bridge using insecure mode.
2023-05-18 10:56:31: Warning: Mosquitto should not be run as root/administrator.
[10:56:32] INFO: Successfully send discovery information to Home Assistant.
[10:56:32] INFO: Successfully send service information to the Supervisor.

Jamesm1 · May 18, 2023, 9:51am

ahh, ok. starting to understand

in terms of the file location that im going to add - i can find 2 share directories. one is /share and is currently empty, and the other share directory is - usr/share/

do I put the /mosquitto/bridge.conf file in the first directory that is currently empty?

many thx for the help

Jamesm1 · May 18, 2023, 12:15pm

I created bridge.conf in /share/mosquito

the bridge file is -

connection XXXXXlora
address nam1.cloud.thethings.network
bridge_protocol_version mqttv311
remote_username my-XXX-XXX@ttn (to get on device profile page in TTS console)
start_type automatic
notifications false
try_private false
remote_password NNSXS.00000000000000000000000UI.ABCDEFGHM53ZRLKTKEL5DAXPSVZ27IDIEM7YWJUAG3PHQ (named API keys in TTS console)
bridge_insecure true
topic # in 0
cleansession true

and now im seeing in the ttn log -

invalid authorization -

{
  "name": "as.mqtt.connect.fail",
  "time": "2023-05-18T12:12:09.714448981Z",
  "identifiers": [
    {
      "application_ids": {
        "application_id": "my-hamp-soil"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
    "namespace": "pkg/identityserver",
    "name": "invalid_authorization",
    "message_format": "invalid authorization",
    "correlation_id": "370a70acd0e1435e9d5f2b2364b32f8b",
    "code": 16
  },
  "origin": "ip-10-101-14-141.us-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "unique_id": "01H0QCCN3JGQCEGD6TWGCANFAJ"

and in the mqtt add on log -

2023-05-18 08:13:18: Received CONNACK on connection local.core-mosquitto.hampttnlora.
2023-05-18 08:13:18: Connection Refused: not authorised
time="2023-05-18T08:13:48-04:00" level=debug msg="found in cache: XXXXX"
2023-05-18 08:13:18: Client local.core-mosquitto.hampttnlora closed its connection.

and in TTN - under applications - im setting up integrations - mqtt - to get the key that im using that seems to be saying invalid

Jamesm1 · May 19, 2023, 10:50am

ok, seem to be making progress…

had to turn the false to true in the mqtt config.

and now see in the mqtt logs -


[06:44:02] INFO: Starting mosquitto MQTT broker...
1684493042: Loading config file /share/mosquitto/bridge.conf
1684493042: Warning: Bridge hampttnlora using insecure mode.
2023-05-19 06:44:02: Warning: Mosquitto should not be run as root/administrator.
[06:44:03] INFO: Successfully send discovery information to Home Assistant.
[06:44:03] INFO: Successfully send service information to the Supervisor.

now the question is - where should the ttn data be visible? in the ttn integration or in the mqtt integration? I was thinking in the mqtt integration - and just log into the local mqtt broadcast - as I thought this would be bridged to show everything? unless im mis understanding.

vincen · May 19, 2023, 12:00pm

It’ll show up through the mqtt broker (forget the TTN integration and best to disable it
I strongly suggest to use an app such as MQTT Snooper on your phone (Android only here sorry) and you connect it at TTN with same credentials you used for the bridge and it’ll allow you to “see” messages sent by TTN and know how to decode it
You then use JSON logic to extract data you want ! Here are some few examples for a button, a VOC sensor and get battery level of sensor

mqtt:
  sensor:
# Boutons
    - state_topic: "v3/buttons-xxxxxxx@ttn/devices/boutons-1/up"
      name: Bouton 1
      value_template: '{{ max(0,value_json.uplink_message.decoded_payload.button_pressed|int(0)) }}'
      state_class: measurement
# Mural
    - state_topic: "v3/sonde-xxxxxx@ttn/devices/rak3172-ambient/up"
      name: “Mural VOC 16”
      value_template: '{{ value_json.uplink_message.decoded_payload.data.voc_16 }}'
      state_class: measurement
    - state_topic: "v3/sonde-xxxxx@ttn/devices/rak3172-ambient/up"
      name: “Mural Batterie”
      value_template: '{{ value_json.uplink_message.decoded_payload.data.voltage_1 }}'
      state_class: measurement
      unit_of_measurement: V

Jamesm1 · May 19, 2023, 12:28pm

ok, will try now

Jamesm1 · May 19, 2023, 2:49pm

If this is what im getting from the MQTT snooper - how do i format that to the configuration.yaml
All of this is from a sensor


    "application_ids": {
      "application_id": "my-hamp-soil"
    },
    "dev_eui": "2CF7F1C044300918",
    "join_eui": "D85D8ED2E9FB2C50",
    "dev_addr": "260C42FF"
  },
  "correlation_ids": [
    "as:up:01H0T5S01Z0WGCH3CXQNP991RA",
    "gs:conn:01H0J9SVR5QHBA1CM8W15XW8HV",
    "gs:up:host:01H0J9SVWFHSY7BYG6PZXKA6TS",
    "gs:uplink:01H0T5RZVG0T1M7Z0G3VY1Y43S",
    "ns:uplink:01H0T5RZVHAB9SZZRV45DC8K7Y",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01H0T5RZVHF3KVR5FK3WMBNFJT",
    "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01H0T5S01YFTVWHD1QSGA2TVCS"
  ],
  "received_at": "2023-05-19T14:14:17.406699928Z",
  "uplink_message": {
    "session_key_id": "AYgK9AksNKT8Ikp4C69CSw==",
    "f_port": 2,
    "f_cnt": 249,
    "frm_payload": "AQwQFAAAAAEGEBBAAAABBxDYcgAAnqA=",
    "decoded_payload": {
      "err": 0,
      "messages": [
        {
          "measurementId": 4108,
          "measurementValue": 0.02,
          "type": "report_telemetry"
        },
        {
          "measurementId": 4102,
          "measurementValue": 16.4,
          "type": "report_telemetry"
        },
        {
          "measurementId": 4103,
          "measurementValue": 29.4,
          "type": "report_telemetry"
        }
      ],
      "payload": "010C101400000001061010400000010710D87200009EA0",
      "valid": true

Where
4108 is Electrical Conductivity
4102 is soil temp
4103 is soil moisture

Jamesm1 · May 20, 2023, 10:34am

this is a snapshot of what I receive in mqtt_explorer :

 "received_at": "2023-05-20T10:14:02.345537991Z",
    "uplink_message": {
      "session_key_id": "AYgK9AksNKT8Ikp4C69CSw==",
      "f_port": 2,
      "f_cnt": 269,
      "frm_payload": "AQwQFAAAAAEGELA2AAABBxAQcgAAQMk=",
      "decoded_payload": {
        "err": 0,
        "messages": [
          {
            "measurementId": 4108,
            "measurementValue": 0.02,
            "type": "report_telemetry"
          },
          {
            "measurementId": 4102,
            "measurementValue": 14,
            "type": "report_telemetry"
          },
          {
            "measurementId": 4103,
            "measurementValue": 29.2,
            "type": "report_telemetry"

what json code do I enter for value_template to decode -

the id’s correspond to sensor fields - like 4103 would be moisture - how do I tell HA to display the measurementValue from a given measurementID?

pwgdrx · May 22, 2023, 9:23pm

Enter the full JSON output here:
https://jsonpathfinder.com/
You can then find the path to the values for the templates on the right-hand side.

vincen · May 23, 2023, 6:30am

Have you done the configuration of the sensor ? as you should be able also to setup it to use different names for different sensors which makes it easier to extract data ! What sensor are you using ?

s3rp1c0 · June 29, 2023, 6:52pm

Hi there, looks like an awesome add-on, thank you!

Can sometime tell me if this would work for an on-premise version of Things Stack Community version? It appears the add-on needs the Storage Integration component which isn’t available in the community version. I tried to re-compose my environment but throws an error:

user@stack:~/the-things-stack$ sudo docker-compose run --rm stack storage-db init
Creating the-things-stack_stack_run … done
The storage integration is not available in the open-source version of The Things Stack.
For more information, see Storage Integration | The Things Stack for LoRaWAN
error:cmd/ttn-lw-stack/commands:storage_integration_not_available (Storage Integration not available)

I guess the only alternative is through MQTT bridge or perhaps Node-Red to pull/push data?

Thanks for any insights…

Tom

vincen · June 30, 2023, 8:10am

Nope it’s not done for on-premise version and I would strongly suggest in all matters to interface it using MQTT and not integration as integration is not really great and works not really great too ! MQTT is a little more work but so much more reliable

jms3000 · September 4, 2023, 7:14am

For me, MQTT was very easy. I installed the MQTT client (not broker). I connected it to host eu1.cloud.thethings.network port 1883, with username and api-key.
Then in my configuration.yaml I defined the mqtt sensors like this:

mqtt:
- sensor:
    name: ttn_gpssensor
    state_topic: "v3/me-ttn-app1@ttn/devices/tbol100/location/solved"
    value_template: '{{ value_json["location_solved"]["location"] }}'
    unique_id: test123
    json_attributes_topic: "v3/my-ttn-app1@ttn/devices/tbol100/location/solved"
    json_attributes_template: '{{ {"received_at": value_json.received_at} | tojson }}'

For finding the right values I used the mqtt client “mosquitto_sub”:

mosquitto_sub -h eu1.cloud.thethings.network -p 8883 -t "#" -u "me-ttn-app1@ttn" -P "NNSXS.JSWH...................EFOPLN" -d

tropfen · November 5, 2023, 2:32pm

Hello angelnu,

i get the following warning message:

Logger: homeassistant.helpers.entity
Source: helpers/entity.py:702
First occurred: 08:13:58 (15 occurrences)
Last logged: 08:48:58

Entity sensor.xxx_yyyyyy_bat_mv (<class 'custom_components.thethingsnetwork.TTN_client.TtnDataSensor'>) is using self.async_update_ha_state(), without enabling force_update. Instead it should use self.async_write_ha_state(), please create a bug report at https://github.com/angelnu/home_assistant_thethingsnetwork/issues

Can you help?

Thanks in advance
Frank

vincen · November 5, 2023, 2:48pm

As previously indicated remove the TTN integration that is crap and not maintained and connect at TTN using MQTT which works great and very easily

tropfen · November 5, 2023, 3:43pm

ok, thanks
but using an integration is usually easier than access via MQTT
I have to start again to get access via MQTT.
Frank

tropfen · November 5, 2023, 5:19pm

Hello Vincèn,
i solved the mqtt-bridge configuration for mosquito. But i am not sure what i have to add here:

sensor:
  - platform: mqtt
    state_topic: "v3/mqtt usernamefromttsconsole@ttn/devices/keyofdevice/up"
    name: “Ext. Temp DS”
    value_template: '{{ value_json.uplink_message.decoded_payload.TempC_DS }}'
    unit_of_measurement: "°C"

What is meant by keyofdevice ? The TTN device ID?

Thanks in advance
Frank

tropfen · November 6, 2023, 8:27pm

i have solved the problem: (based on HA OS)

Create a directory /mosquito in the /share directory

Create a file bridge.conf in /share/mosquito

Add

connection lora-fk  
address eu1.cloud.thethings.network  
bridge\_protocol\_version mqttv311  
remote\_username xxx@ttn  (ttn mqtt user name)
start\_type automatic  
notifications false  
try\_private false  
remote\_password NNSXS.xxxx (ttn api-key)  
bridge\_insecure true  
topic # in 0  
cleansession true

in the configuration of mosquito under Customize

active: true

Restart the mqtt add-on

Add file /config/mqtt_sensor.yaml

Add

sensor:
  - name: LoRaWan_Temp_Aussen_Haus
    state_topic: "v3/xxx@ttn/devices/ttn_device_id/up"
    value_template: '{{value_json.uplink_message.decoded_payload.TempC_SHT}}'
    unit_of_measurement: "°C"
    device_class: "temperature"

Add in /config/configuration.yaml

mqtt: !include mqtt_sensor.yaml

quick reload yaml

Frank

angelnu · November 6, 2023, 8:57pm

Jep, I will take a look and see if it is an easy change. The integration is working for me so, combined with little time, does not motivate me to spend time on it. So fixing warnings or doing the polishing to replace the non working old integration in HA is low in the prio list as contributing to HA might likely require some code refactoring.

But when people raised PRs I tried to reply/merge quickly. Perhaps I should try to get it into HACS first…

ursmii · February 14, 2024, 3:08pm

I’m a newbee in HA and I’m stuck here
I followed the instructions of @angelnu and installed HACS and there in Integrations “The Things Network (new version)”
Then I tried to configure
→ Settings->Devices&services->TheThingsNetwork->CONFIGURE->devices
and I get the message “No devices available to configure”

in TTN the Storage Integration is enabled (only the slider changed) and the uplink payload formatter is set to CayenneLPP
I’m trying to get data from a TTN-Node and see in the console the data

      "decoded_payload": {
        "accelerometer_7": {
          "x": 0.249,
          "y": -0.782,
          "z": -0.149
        },
        "analog_in_4": 4.23,
        "digital_in_1": 0,
        "digital_in_2": 0,
        "digital_in_3": 0,
        "luminosity_6": 0,
        "temperature_5": 23

In the log I see

2024-02-14 15:32:49.433 DEBUG (MainThread) [custom_components.thethingsnetwork] Fetch of ttn data: 360s
2024-02-14 15:32:49.434 DEBUG (MainThread) [custom_components.thethingsnetwork] URL: https://eu1.cloud.thethings.network/api/v3/as/applications/ttn-node-ursm/packages/storage/uplink_message?last=360s&order=received_at
2024-02-14 15:32:49.538 DEBUG (MainThread) [custom_components.thethingsnetwork] TTN entry: b'{"error":{"code":3, "message":"error:pkg/auth:token (invalid token)", "details":[{"@type":"type.googleapis.com/ttn.lorawan.v3.ErrorDetails", "namespace":"pkg/auth", "name":"token", "message_format":"invalid token", "correlation_id":"5aa791cff31e4cc29ba326f26f309e45", "code":3}]}}\n'
2024-02-14 15:32:49.539 ERROR (MainThread) [custom_components.thethingsnetwork] TTN entry without result: {'error': {'code': 3, 'message': 'error:pkg/auth:token (invalid token)', 'details': [{'@type': 'type.googleapis.com/ttn.lorawan.v3.ErrorDetails', 'namespace': 'pkg/auth', 'name': 'token', 'message_format': 'invalid token', 'correlation_id': '5aa791cff31e4cc29ba326f26f309e45', 'code': 3}]}}
2024-02-14 15:32:49.539 DEBUG (MainThread) [custom_components.thethingsnetwork] Finished fetching The Things Network data in 0.105 seconds (success: True)
2

what’s wrong? do I have to enter a yaml somewere?
thanks for your kind help to an absolute beginner in HA
urs