MQTT Sensor , Smart Meter , Sensor format Problem

blacktux88 · April 19, 2023, 7:29am

Dear Community,

I have got a script on Github, the script reads an Smart Meter through P1 Serial Connection and provide me the current power usage and some other details.

I want to add MQTT support to this Script, i have tried it with ChatGPT and it seems to work.

The Raspberry reads the Sensor and sends the data with MQTT to my HASSIO Mosquito Broker.
I have installed an MQTT Browser to my Laptop and connected to the Broker to check if the data looks good. Everything looks good.

But I have a Problem to create a sensor to add this to my Home Assistant.
Maybe someone could help me to create a working sensor template.

I have activated MQTT in home assistant, but now I have no idea what to do next.

Manual creation of a sensor ends with the error json_value not found and so on.

here is the original script:

import time
import json

from serial import Serial, PARITY_NONE, EIGHTBITS, STOPBITS_ONE
from decode import decrypt_frame, convert_to_dict, check_and_encode_frame

GLOBAL_UNICAST_ENC_KEY = "MYKEY"
GLOBAL_AUTHENTICATION_KEY = "MYKEY"

if __name__ == "__main__":
    if GLOBAL_UNICAST_ENC_KEY == "":
        raise RuntimeError("Please set the GLOBAL_UNICAST_ENC_KEY")
    if GLOBAL_AUTHENTICATION_KEY == "":
        raise RuntimeError("Please set the GLOBAL_AUTHENTICATION_KEY")

    while True:
        try:
            with Serial('/dev/ttyUSB0', 115200, timeout=6.0,
                        bytesize=EIGHTBITS, parity=PARITY_NONE, stopbits=STOPBITS_ONE, rtscts=False) as ser:
                data = ser.read(511)
                if len(data) == 0:
                    print("Warte auf Smart Meter...")
                    continue
                decrypted = decrypt_frame(GLOBAL_UNICAST_ENC_KEY, GLOBAL_AUTHENTICATION_KEY, data)
                encoded_frame = check_and_encode_frame(decrypted)
                response_as_dict = convert_to_dict(encoded_frame)
                #print("Raw frame:\n" + str(data.hex()))
                #print(json.dumps(response_as_dict, indent=2))
                print("Wird aktuell geliefert: " + str(response_as_dict["1-0:1.7.0"]))
                print("Wird aktuell eingespeist: " + str(response_as_dict["1-0:2.7.0"]))
                print("Wurde geliefert (Zählerstand): " + str(response_as_dict["1-0:1.8.0"]))
                print("Wurde eingespeist (Zählerstand): " + str(response_as_dict["1-0:2.8.0"]))
        except Exception as e:
            print(e)
            time.sleep(1)
            pass

Here is the Code with MQTT added

import time
import json
import paho.mqtt.client as mqtt

from serial import Serial, PARITY_NONE, EIGHTBITS, STOPBITS_ONE
from decode import decrypt_frame, convert_to_dict, check_and_encode_frame

GLOBAL_UNICAST_ENC_KEY = "MYKEY"
GLOBAL_AUTHENTICATION_KEY = "MYKEY"
MQTT_SERVER = "MQTT_SERVER_ADDRESS"
MQTT_PORT = 1883
MQTT_USERNAME = "MQTT_USERNAME"
MQTT_PASSWORD = "MQTT_PASSWORD"
MQTT_TOPIC = "MQTT_TOPIC"

def send_to_mqtt(payload):
    client = mqtt.Client()
    client.username_pw_set(MQTT_USERNAME, MQTT_PASSWORD)
    client.connect(MQTT_SERVER, MQTT_PORT)
    client.publish(MQTT_TOPIC, payload=payload, qos=0, retain=False)
    client.disconnect()

if __name__ == "__main__":
    if GLOBAL_UNICAST_ENC_KEY == "":
        raise RuntimeError("Please set the GLOBAL_UNICAST_ENC_KEY")
    if GLOBAL_AUTHENTICATION_KEY == "":
        raise RuntimeError("Please set the GLOBAL_AUTHENTICATION_KEY")

    while True:
        try:
            with Serial('/dev/ttyUSB0', 115200, timeout=6.0,
                        bytesize=EIGHTBITS, parity=PARITY_NONE, stopbits=STOPBITS_ONE, rtscts=False) as ser:
                data = ser.read(511)
                if len(data) == 0:
                    print("Warte auf Smart Meter...")
                    continue
                decrypted = decrypt_frame(GLOBAL_UNICAST_ENC_KEY, GLOBAL_AUTHENTICATION_KEY, data)
                encoded_frame = check_and_encode_frame(decrypted)
                response_as_dict = convert_to_dict(encoded_frame)
                payload = {
                    "current_delivery": response_as_dict["1-0:1.7.0"],
                    "current_feed_in": response_as_dict["1-0:2.7.0"],
                    "total_delivery": response_as_dict["1-0:1.8.0"],
                    "total_feed_in": response_as_dict["1-0:2.8.0"]
                }
                payload_str = json.dumps(payload)
                send_to_mqtt(payload_str)
                print("Wird aktuell geliefert: " + str(response_as_dict["1-0:1.7.0"]))
                print("Wird aktuell eingespeist: " + str(response_as_dict["1-0:2.7.0"]))
                print("Wurde geliefert (Zählerstand): " + str(response_as_dict["1-0:1.8.0"]))
                print("Wurde eingespeist (Zählerstand): " + str(response_as_dict["1-0:2.8.0"]))
        except Exception as e:
            print(e)
            time.sleep(1)
            pass

Here is the Console Output on the PI

Here is the MQTT-Client output

i have tried it with this sensor config

sensor:
  - platform: mqtt
    state_topic: "smartmeter"
    name: "Smart Meter Total Consumption"
    unit_of_measurement: "kWh"
    value_template: "{{ value_json['1-0:1.8.0'] | float / 1000 }}"

But every time this has ended with the

"UndefinedError: 'value_json' is undefined" error.

Hopefully there is someone who can help me to understand how mqtt sensors work.

thanks
Patrick

Troon · April 19, 2023, 8:52am

Mistake #1. Don’t do that — although to be fair, if ChatGPT has modified that code to include MQTT output, that’s pretty impressive.

Mistake #2: the MQTT Explorer screenshot shows the topic with the typo “smtartmeter”. I guess that’s come from wherever you define MQTT_TOPIC.

Mistake #3: I think the ["1-0:1.8.0"] isn’t in the MQTT message, although it’s not possible to tell as you haven’t pasted the MQTT topic output in full, in text. I think you want:

    value_template: "{{ value_json['total_delivery']['Wirkenergie Lieferung (Lieferung an Kunden) (+A)']['value'] | float(0) / 1000 }}"

In future, please post text rather than screenshots. I can’t tell, for example, how many spaces there are between Kunden) and (+A) — if it’s not just one space, you need to adjust the template above to match.

Try fixing those and come back if it’s still not working.

blacktux88 · April 19, 2023, 9:12am

Thanks for the answer,

you are right to modify the script with chatgpt isnt the right but a way near to get it work.

i have corrected the typo , now its “smartmeter”

and i have changed the value_template.

still get the error

sensor:
  - platform: mqtt
    state_topic: "smartmeter"
    name: "Smart Meter Total Consumption"
    unit_of_measurement: "kWh"
    value_template: "{{ value_json['total_delivery']['Wirkenergie Lieferung (Lieferung an Kunden) (+A)']['value'] | float(0) / 1000 }}"

UndefinedError: ‘value_json’ is undefined

i would like to understand what this means.
is this an format error. or does homeassistant does not find the value.

here is the “german” output

Wird aktuell geliefert: {'Momentane Wirkleistung Bezug (+A)': {'value': 2977, 'unit': 'W'}}
Wird aktuell eingespeist: {'Momentane Wirkleistung Lieferung (-A)': {'value': 0, 'unit': 'W'}}
Wurde geliefert (Zählerstand): {'Wirkenergie Lieferung (Lieferung an Kunden) (+A)': {'value': 12915846, 'unit': 'Wh'}}
Wurde eingespeist (Zählerstand): {'Wirkenergie Bezug (Lieferung an EV) (-A)': {'value': 61, 'unit': 'Wh'}}

this is the output of the MQTT Client

{"current_delivery": {"Momentane Wirkleistung Bezug (+A)": {"value": 2851, "unit": "W"}}, "current_feed_in": {"Momentane Wirkleistung Lieferung (-A)": {"value": 0, "unit": "W"}}, "total_delivery": {"Wirkenergie Lieferung (Lieferung an Kunden) (+A)": {"value": 12915942, "unit": "Wh"}}, "total_feed_in": {"Wirkenergie Bezug (Lieferung an EV) (-A)": {"value": 61, "unit": "Wh"}}}

thanks

Troon · April 19, 2023, 9:20am

It means that it doesn’t have a response to work with.

Does it really have “code here” on the end?!

Rather than posting the client output, can you post the message as received by the broker? If not, just a screenshot of MQTT Explorer showing the value in total_delivery that you’re trying to get to?

Also, I have just realised you’re using the old MQTT sensor format. Should be like this (docs):

mqtt:
  sensor:
    - name: "Smart Meter Total Consumption"
      state_topic: "smartmeter"
      unit_of_measurement: "kWh"
      value_template: "{{ value_json['total_delivery']['Wirkenergie Lieferung (Lieferung an Kunden) (+A)']['value'] | float(0) / 1000 }}"

blacktux88 · April 19, 2023, 9:27am

code here was an insert error sorry.

the MQTT Broker Output is the longer line

{
"current_delivery": {"Momentane Wirkleistung Bezug (+A)": {"value": 488, "unit": "W"}}, "current_feed_in": {"Momentane Wirkleistung Lieferung (-A)": {"value": 0, "unit": "W"}},
"total_delivery": {"Wirkenergie Lieferung (Lieferung an Kunden) (+A)": {"value": 12916118, "unit": "Wh"}},
"total_feed_in": {"Wirkenergie Bezug (Lieferung an EV) (-A)": {"value": 61, "unit": "Wh"}}
}

for me the current_delivery and the current_feed_in is the most important value

the problem with the new or old format is big for me , because i have no working example.

thanks

Troon · April 19, 2023, 9:36am

Ah, OK. I thought you wanted the total one because that’s what your initial code was for.

So in your configuration.yaml, you want this — make sure you only have one top-level mqtt: heading in the file:

mqtt:
  sensor:
    - name: "Smart Meter Current Delivery"
      state_topic: "smartmeter"
      unit_of_measurement: "kW"
      value_template: "{{ value_json['current_delivery']['Momentane Wirkleistung Bezug (+A)']['value']|float(0)/1000 }}"

    - name: "Smart Meter Current Feed In"
      state_topic: "smartmeter"
      unit_of_measurement: "kW"
      value_template: "{{ value_json['current_feed_in']['Momentane Wirkleistung Lieferung (-A)']['value']|float(0)/1000 }}"

If that’s still not working, you need to check your MQTT installation.

blacktux88 · April 19, 2023, 9:38am

thanks for the help but still the error

UndefinedError: ‘value_json’ is undefined

so in think i have to check my mqtt installation

thanks

Troon · April 19, 2023, 9:41am

No, you don’t paste this in the template editor. That’s just a tool for testing templates. You need to edit your configuration file:

See here:

blacktux88 · April 19, 2023, 9:59am

holy s*

sorry , i thought the testing template section shows me if the template is working and shows me the value !!!

its working now

thanks @Troon

Troon · April 19, 2023, 10:03am

It does, but the template is only the section in {{ }}. You can use it for testing like this — I have set the value_json variable “manually” to the MQTT topic content, then validated the template used in the sensor:

blacktux88 · April 19, 2023, 10:05am

thanks this info would help me so much for future !

thanks