Energy management using MQTT data

[Joelabq]

Hey all,

Been banging my head on a specific issue when trying to configure energy management in 2021.8. I already have hardware using the Brultech GreenEye Monitor (GEM) which then pushes data to their Dashbox which has it’s own graphs. I wouldn’t mind having some of the data in HA instead, so I enabled the MQTT from the dashbox to my HA instance. I am getting the data, and can add it to the energy configuration. However, the data never shows up.

Here is what I have in my Configuration.yaml:

sensor:
  - platform: mqtt
    name: "Solar generated"
    unique_id: "solar_generated"
    device_class: "energy"
    state_class: "measurement"
    unit_of_measurement: "kWh"
    state_topic: "dashbox/0001/c29/kwh"
  - platform: mqtt
    name: "House Consumption"
    unique_id: "house_consumption"
    device_class: "energy"
    state_class: "measurement"
    unit_of_measurement: "kWh"
    state_topic: "dashbox/00001/c1/kwh"

And I always end up with:

Any clues? It’s beyond the two hour period, I configured this last weekend and have made no progress understanding the blackbox of energy management that seems to be in HA.

Thanks for any help!

[123 (Taras)]

@Joelabq

made no progress understanding the blackbox of energy management that seems to be in HA

The black box isn’t all that opaque. The sensors are missing information for “last_reset”, specifically last_reset_topic and last_reset_template. There’s a discussion about it in this topic.

MQTT Sensor : how to define last_reset_value_template

Clever! It uses the same topic for state_topic, json_attributes_topic, and last_reset_topic. Because state_topic will regularly receive payloads, it will cause last_reset_value_template to be evaluated the moment a payload is received via the shared topic. The actual content of the payload is irrelevant for last_reset_value_template because it doesn’t use it and is hard-coded with the Unix Epoch. Applying this simple principle to Marius’ example, we get this: - platform: mqtt state_t…

Add this to the 1st sensor:

    last_reset_topic: "dashbox/0001/c29/kwh"
    last_reset_value_template: '1970-01-01T00:00:00+00:00'

and this to the 2nd sensor:

    last_reset_topic: "dashbox/00001/c1/kwh"
    last_reset_value_template: '1970-01-01T00:00:00+00:00'

[jdwhite (Jason White)]

You need to include the last_reset attribute. Energy: Not showing the expected entities for consumption

[Joelabq]

Sorry, I do have that in my configuration under global since you cannot add last_reset to the mqtt definition directly without HA giving an error. Without that, I would not have been able to add the sensors at all to the energy management configuration.

[123 (Taras)]

Overlooked to mention that the development team has re-engineered the concept of “last_reset” and it will be handled differently in 2021.9.0.

[123 (Taras)]

@Joelabq

since you cannot add last_reset to the mqtt definition directly without HA giving an error

I don’t know what you’re doing wrong, but you are doing something wrong because the two options I mentioned (above) are clearly documented and used successfully by others (see posted link).

---

EDIT

Read you post again and if you meant you tried to literally include a last_reset option in the MQTT Sensor’s configuration, then that will definitely not work. You must use the two options I described above and they will be used to generate a last_reset attribute for the sensor.

[Joelabq]

@123

last_reset_value_template: '1970-01-01T00:00:00+00:00'

The issue I had is the same one in the post you linked, which is where I set it in “global”:

homeassistant:
  customize: !include customize.yaml
  customize_glob:
    sensor.house_consumption:
      last_reset: '1970-01-01T00:00:00+00:00'
    sensor.solar_generated:
      last_reset: '1970-01-01T00:00:00+00:00'

If I set it in the mqtt sensor I get:Invalid config for [sensor.mqtt]: Single-level wildcard must occupy an entire level of the filter for dictionary value @ data['last_reset_topic']. Got datetime.datetime(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc). (See /config/configuration.yaml, line 32). Please check the docs at https://www.home-assistant.io/integrations/mqtt

I saw that there were changing the last_reset stuff in 9.0 but I am not sure that will help whatever I have done wrong since I am able to see it in the energy configuration and set it up. I think I must be missing something else. I will try to swap to the " last_reset_value_template" thing here in a few minutes and see if that changes the outcome.

Thanks!

[Joelabq]

So I did not get an error using the last_reset_value_template and last_reset_topic, but now the sensor is not something I can use in the energy management configuration. I removed my house consumption and solar production sensors and now neither of them are available to add back unless I go back to my global config I had posted in my last post…

I now have this:

sensor:
  - platform: mqtt
    name: "Solar generated"
    unique_id: "solar_generated"
    device_class: "energy"
    state_class: "measurement"
    unit_of_measurement: "kWh"
    state_topic: "dashbox/0001/c29/kwh"
    last_reset_topic: "dashbox/0001/c29/kwh"
    last_reset_value_template: '1970-01-01T00:00:00+00:00'
    
  - platform: mqtt
    name: "House Consumption"
    unique_id: "house_consumption"
    device_class: "energy"
    state_class: "measurement"
    unit_of_measurement: "kWh"
    state_topic: "dashbox/0001/c1/kwh"
    last_reset_topic: "dashbox/0001/c1/kwh"
    last_reset_value_template: '1970-01-01T00:00:00+00:00'

[123 (Taras)]

@Joelabq

but now the sensor is not something I can use in the energy management configuration

Again, something wrong somewhere because (here it comes) it works for me.

• I added your MQTT Sensor configuration to my system.
• Reloaded ‘Manually configured MQTT entities’.
• Used MQTTExplorer to publish some bogus values to their respective topics.
• The two sensors show the bogus values.

• In Configuration > Energy > Individual Devices, both sensors appear as eligible entities.

[Joelabq]

Well… I re-added it again, and now its appearing… I can see that data is coming in to that sensor but it was before. Now I guess I have to wait an hour or two? or is there another way to see that it is at least collecting data properly?

thanks again!

[123 (Taras)]

Glad to hear you sorted out the initial glitch.

I believe it collects data every hour so, yes, you’ll need to give it a bit of time before the graph is updated.

Once you have confirmed it is operating properly, please consider marking my post above with the Solution tag. It will automatically place a check-mark next to the topic’s title which signals to other users that this topic has been resolved. It will also place a link below your first post that leads to the solution post. All of this helps users find answers to similar questions. For more information, refer to guideline 21 in the FAQ.

[Joelabq]

I’ll be sure to do that

So far after nearly 2hrs there is nothing different in the energy panel. There is no animation in the energy distribution, no numbers associated with the sources, and nothing in the graphs. I am seeing data on those entities regularly. Maybe I am not patient enough, but thats what I thought last time when I had it configured using the globals.

[123 (Taras)]

If you have never seen anything in the graph ever, even after adding several different sensors, then that’s a separate problem (from merely a misconfigured MQTT Sensor). It’s difficult to say what would cause that (I can’t replicate the problem).

[Joelabq]

I’ve added a channel plus added the the two sensors to the individual channels to see what happens and nothing on the energy dashboard. I can click on them in the individual and see the data in the cards but nothing calculates. Thats why I feel it’s a black box, I can find nothing in the documentation on how this thing works in order to understand what I am doing wrong. Thanks for trying.

[123 (Taras)]

Not sure what you mean by “channel” (I haven’t see that term used in the Energy integration). Did you add sensor.solar_generated and sensor.house_consumption under “Individual Devices”?

[Joelabq]

Yeah sorry, that terminology is from the Dashbox/GEM energy monitor I have. A channel is a CT device on a circuit. I did add solar_generated and house_consumption under individual devices as a test and see nothing on the panel. I feel like something is not triggering the longer term storage/statistical stuff as I also added a stat card and do not get what I am expecting. The data I am getting is kwh since the last packet sent. Not sure if that is what HA is expecting?

[123 (Taras)]

@Joelabq

data I am getting is kwh since the last packet sent. Not sure if that is what HA is expecting?

Yes, it has to be energy, expressed in Wh or kWh, otherwise it doesn’t qualify as an acceptable resource for the Energy integration (some users mistakenly try to include power sensors, expressed in W or kW, but those are ignored by the integration).

[Joelabq]

In this case, my Dashbox returns two MQTT endpoints, one is in watts and the other is in kWh. So… it should work?

[Joelabq]

So this might be something that I should have remembered when I was asking this question… Logs… for some reason this slipped my mind until I was looking today… this is the error I am getting and I was unable to find an answer when searching for one:

Logger: homeassistant.components.recorder
Source: components/sensor/recorder.py:283
Integration: Recorder (documentation, issues)
First occurred: 12:12:00 PM (1 occurrences)
Last logged: 12:12:00 PM

Error while processing event StatisticsTask(start=datetime.datetime(2021, 8, 29, 17, 0, tzinfo=datetime.timezone.utc)): unsupported operand type(s) for +=: 'NoneType' and 'float'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 623, in _run_event_loop
    self._process_one_event_or_recover(event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 632, in _process_one_event_or_recover
    self._process_one_event(event)
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 738, in _process_one_event
    self._run_statistics(event.start)
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 721, in _run_statistics
    if statistics.compile_statistics(self, start):
  File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 303, in wrapper
    return job(instance, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 154, in compile_statistics
    platform_stats.append(platform.compile_statistics(instance.hass, start, end))
  File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 283, in compile_statistics
    _sum += new_state - old_state
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'

Any thoughts?

[123 (Taras)]

The Energy integration is summing the values of several sensors and, at one point, at least one of the sensors failed to have a proper numeric value (‘NoneType’ which is neither int or float). It can’t add a float value to NoneType and that’s what generated the error message.

Perhaps one of your energy sensors has an unknown or unavailable value in its history.