Glow / Hildebrand Display - Local MQTT Access - Template Help

[mogons]

@JohnWH, I don’t think this is a glitch in the system. I just found the MQTT listening debugger and looking at the messages form the CAD I am getting:

{
    "electricitymeter": {
        "timestamp": "2022-06-23T14:36:59Z",
        "energy": {
            "export": {
                "cumulative": 0,
                "units": "kWh"
            },
            "import": {
                "cumulative": 3625.742,
                "day": 2.236,
                "week": 14.057,
                "month": 105.876,
                "units": "kWh",
                "mpan": "REDACTED",
                "supplier": "REDACTED",
                "price": {
                    "unitrate": 0.2729,
                    "standingcharge": 0.4105
                }
            }
        },
        "power": {
            "value": -2.244,
            "units": "kW"
        }
    }
}

You can see the current power is showing as -2.244kW (as my solar panels are working nicely), but the export is showing 0 kWh and I know my meter is exporting. Assuming the broker is seeing the raw data then it looks like the CAD is not reporting properly

[GadgetUK (Stuart)]

More likely you’re meter / supplier hasn’t enable the export registers on the meter.

Works fine for me, I’m on Octopus

[glitter-ball]

I’m now looking at using the lifetime cumulative registers from the meters and doing the day totals within HA, as I then have much more control. I’ve just added raw points in a graph to see how those behave over a few days. @JaneatGlow has suggested it’s a limitation of the data from the meters.

Like others, I’m with Octopus, but I’m not sure that’s relevant.

[glitter-ball]

There is an outlier filter (one of many) available in HA which is very good. It will process the last n samples and exclude any data more than x% away from the median. I can see that being very handy to filter out negative / zero glitches.

I’m using it here to filter out glitches from my Dallas sensor…

- platform: filter
  name: "Filtered return temp"
  unique_id: "return_temp"
  entity_id: sensor.temperature_sensor_28_be2a_7791_0402
  filters:
        - filter: outlier
          window_size: 4
          radius: 10.0

[Redpizza]

I presume that if I move across to this local MQTT from the cloud MQTT, and use your sensor naming, then I will lose all my previous Energy stats?

Would I need to rename “smart_meter_electricity_import” to the sensor name I currently have setup in the Energy panel? Is there a way to leave all the sensor names the same and then just set my old name to the new name via something like a template sensor, ie sensor.daily_home_electricity == sensor.smart_meter_electricity_import ?

I really don’t want to lose 6 months worth of Energy stats.

[duffjp]

I have tried out the update to capture 0 responses provided by @robertalexa and that worked well for any issues during the day. However, as he pointed out, that still gave me the issue with my gas meter re-providing the previous days value for the first 30 minutes. As such I have modified the code to cover a that cross over and the delay with the gas meter updating

I created a debug entity in order to prove it and this is the code:

    - name: "Smart Meter Gas Debug Calc: Import (Today)"
      unique_id: "smart_meter_gas_import_day_debugcalc"
      state_topic: "glow/REDACTED/SENSOR/gasmeter"
      device_class: "energy"
      unit_of_measurement: "kWh"
      state_class: "measurement"
      value_template: >
        {% if value_json['gasmeter']['energy']['import']['day'] == 0 %}
          {% if now() > now().replace(hour=0).replace(minute=40).replace(second=0).replace(microsecond=0) %}
            {{ states('sensor.smart_meter_gas_debug_calc_import_today') }}
          {% else %}
            {{ value_json['gasmeter']['energy']['import']['day'] }}
          {% endif %}
        {% else %}
          {% if now() < now().replace(hour=0).replace(minute=40).replace(second=0).replace(microsecond=0) %}
            {{ 0 }}
          {% else %}
            {{ value_json['gasmeter']['energy']['import']['day'] }}
          {% endif %}
        {% endif %}
      icon: "mdi:fire"

I changed the threshold from being a minute passed midnight, to being 40 minutes. This gives the gas meter plenty of time to reset and feed the real data through to the IHD. I then added code to force a zero value in the first 40 minutes so you don’t carry the previous day’s peak value through to the new day. I’ve been using it for a couple of weeks and seems to work.

I’m testing the equivalent on electric aswell, as I’m seeing the same thing on the 00:00:00 entry. Whilst it doesn’t appear to feed into the Energy stats, if you have other reporting of the peak value for the day, it does impact it, so I wanted to ensure the day is zero from the first entry. It will have its first run tonight and I’ll feed back at some point. Note, I have set “40” back to “1” on the electric as it is generally correct and want to have as much accurate data as possible.

[Redpizza]

@duffjp So I replaced my template for yours last night, as I was getting the previous days gas total being added into the current days total. All works perfectly in the Energy dashboard, but this is what I am getting in the history now for that sensor:

I’m getting blanks up until the first actual gas reading of the day comes through via mqtt (gas comes on at 07:30, but it take half an hour to show up, so first reading via mqtt is 08:00).

[ungrim97 (Mike Eve)]

Hi,

I am looking at your calculation for charge including standing charge as the Energy dashboard doesn’t handle that

I am trying to understand why your calculation of “Total kWh imported today” * “Current unit rate” + “standing charge”. It seems to me that this only works if you have a fixed unit rate per kWh, and that if you have a variable rate (E7 Octopus Go etc) then it would treat your unit rate as fixed to whatever it is when it calculates it

Can you walk me through it if thats not the case. I am getting desperate to get my standing charges represented in the dashboard without resorting to crazy things like creating an additional meter or something

[mkdr36]

if you are on octopus go or E7 with a peak and off peak tariff, you will need to install the ocotpus API integration which has a live tariff entity for electricity which pulls data from ocotpus not from your meter. SMETS2 meters dont really work (or dont get configured) for peak and off peak so it usually only displays one rate.

If you install the API integration and replace the ‘current unit rate’ in your template sensor with the octopus live tariff entity, it should show the accurate and current cost.

[ungrim97 (Mike Eve)]

Thanks for that. Its at least an alternative. I was hoping for something that wouldn’t require Cloud interaction.

[tomcoleman]

Hi i have put the code into mqtt.yaml uploaded that to my include directory.

Everything seems to be working but when i go to the energy section i get the following error.

Something about last reset missing?

Last reset missing

The following entities have state class 'measurement' but 'last_reset' is missing:

* sensor.smart_meter_electricity_import_today

What code do i need to put in to retifcy this ?

I put this into my configuration.yaml and seems to have gone away?

  customize_glob:
    sensor.smart_meter_electricity_import_today:
      unit_of_measurement: "kWh"     
      device_class: energy
      state_class: measurement
      last_reset: "1970-01-01T00:00:00+00:00"

[robjameslee (Rob Lee)]

Does anyone have an example of adding the energy-devices-graph to the provided dashboard yaml that will make the graph go full-width. I’m not familiar with layout-card and despite trying various things I’ve not managed to make the card to full page width like the other components/panels. Currently it looks like the attached.

Appreciate any sample yaml from someone who knows layout card.

[garan]

Thanks for sharing your hard work.

One minor improvement suggestion is to say “Please make sure to use the correct device ID, using capital letters, and replace the word REDACTED with it.” I typed mine in and wondered if it was case sensitive so knew what to tweak when it didn’t work using lower case letters.

[stevekdavis (Steve Davis)]

thanks for the hard work on this. One question, where can I get the custom grid-layout element so I can add the same lovelace dash as you’ve used?

[megakid (James)]

Hi all - is there anything I need to add to the HACS integration? Someone mentioned above negative values but it already filters out zeros on some fields. Perhaps I missed it but what are the negative values?

[JohnWH (John W)]

Hi James (@megakid),
I am the user who initially requested assistance in templating the local MQTT data and also the person who’s electricity readings experience random extremely negative ‘spikes’. I detailed the problem - see entry #16 within this thread. Unfortunately once or twice per day I am still seeing massive negative spikes on the Bright App (as this data isn’t filtered)
In order to avoid logging these within HA I subsequently followed the excellent suggestion from @robertalexa (see entry #17) so if the value is less than 0 it uses the previous value instead.

I would be able to use the HACS integration instead of the manual lines of template code if it could incorporate a similar means of avoiding negative spikes.

[Platypus (Nick)]

Hi James,

Many thanks to you and Robert for all the hard work on this.

I’m on Octopus Go i.e. two separate time based tariffs + standing charge.

If I set the energy dashboard to use an entity tracking total costs and use Smart Meter Electricity: Cost (Today) my electricity costs are overestimated as at the 04:30 switch over the peak rate is applied to electricity that I used during the 00:30 to 04:30 off peak period.

I can sort of get around this by using Smart Meter Electricity: Import Unit Rate as an entity with current price but this then doesn’t include the standing charge so I end up with an underestimate……

Any chance that a variable tariff option could be added to the HACS integration?

[rav101 (Ravi Jain)]

Hey,

Thanks for this as it almost perfectly helped get me set up. I’ve successfully completed the first part and now have mqtt set up and the Smart Meter readings coming through for my electricity (I don’t have gas on smart meter at the moment). However, when I copy through the second template to get my electricity cost as well i get the following error when checking my configuration.yaml file in developer settings:

Error loading /config/configuration.yaml: while parsing a block collection
in "/config/sensor.yaml", line 3, column 5
expected <block end>, but found '?'
in "/config/sensor.yaml", line 4, column 5`

I have added this to a separate sensor.yaml to my configuration.yaml with only this included but the same happens if I add the lines to the bottom of my configuration.yaml file. If I remove if, my configuration.yaml file validates in developer settings properly.

I was wondering if anyone else has seen anything similar and could point me to a fix. I’m very new to HA and I’m mostly following instructions rather than knowing exactly what each line of code is doing.

[Amardeep]

Hi, not sure if this issue has already been mentioned, but I noticed a problem when trying to get electricity cost working in the Energy Panel.

I’m using @robertalexa 's custom component, and I’m tracking cost using the “Smart Meter Electricity: Cost (Today)” entity as instructed. The cost it showing as “£0.00”, though. I’ve had a look at the MQTT payload using MQTT Explorer - it is updating regularly and shows the following:

{
  "electricitymeter": {
    "timestamp": "2022-08-30T10:53:29Z",
    "energy": {
      "export": {
        "cumulative": 0,
        "units": "kWh"
      },
      "import": {
        "cumulative": 3066.055,
        "day": 0.005,
        "week": 0.018,
        "month": 0.348,
        "units": "kWh",
        "mpan": "XXXXXXXXXXXXX", # REDACTED
        "supplier": "British Gas",
        "price": {
          "unitrate": 0.291,
          "standingcharge": 0.5054
        }
      }
    },
    "power": {
      "value": 0.674,
      "value": 0.623,
      "units": "kW"
    }
  }
}

According to electricitymeter>import>day, I have only used 0.005 kWh of electricity today. The Bright app, however, shows that I have used 5.71 kWh. The actual kWh readings are out by a factor of 1000.

Bright app Energy Usage:

Bright app Cost:

Any idea how I can get an accurate reading from my IHD?

Thanks!

---

Edit: weirdly, although the MQTT payload says I’ve used 0.005 kWh, the Sources item in the Energy panel actually shows it correctly (note: in the time between the Bright app screenshots above and the Home Assistant Energy panel image below, usage had actually gone up from 5.71 kWh to 6.03 kWh). Cost still zero, though.

[BertrumUK (Neil B)]

That is strange. Is this pulling the MQTT straight from the IHD? And if so you could try rebooting it. If you have access to the Glow MQTT feed, what figures do they show?