Energy dashboard missing some bars

[mariolpantunes (Mário Antunes)]

Recently I acquired a sensor for the electricity meter. The sensor page is in PT, but is a tasmota that was adjusted to the meters used in Portugal.
It is connected through MQTT and I have the values in Home Assistant and the database:

However, the energy dashboard has a strange behaviour:

There are mentions of similar issues, but all appear to fix themselves after one update or by changing the browser.
HomeAssistant is being deployed in an RPI4 with the following docker-compose:

version: '3'
services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:stable"
    restart: unless-stopped
    environment:
      - TZ=Europe/Lisbon
    volumes:
      - ./config/:/config
      - /etc/localtime:/etc/localtime:ro
    privileged: true
    network_mode: host
    depends_on:
      - mosquitto
      - mariadb
      - influxdb

  mosquitto:
    image: eclipse-mosquitto
    container_name: mosquitto
    volumes:
      - ./mosquitto:/mosquitto
    restart: unless-stopped
    network_mode: host

  mariadb:
    container_name: mariadb
    image: mariadb
    restart: unless-stopped
    ports:
      - "13306:3306/tcp"   # So we can access the WebUI
    environment:
      - TZ=Europe/Lisbon
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_HA_DATABASE}
      - MYSQL_USER=${MYSQL_HA_USER}
      - MYSQL_PASSWORD=${MYSQL_HA_PASSWORD}
    volumes:
      - ./mariadb/data:/var/lib/mysql
      - ./mariadb/config/:/etc/mysql/conf.d

  influxdb:
    container_name: influxdb
    image: influxdb
    restart: unless-stopped
    ports:
      - "8086:8086/tcp"   # So we can access the WebUI
    environment:
      - TZ=Europe/Lisbon
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=${INFLUXDB_USER}
      - DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUXDB_PASSWORD}
      - DOCKER_INFLUXDB_INIT_ORG=${INFLUXDB_ORG}
      - DOCKER_INFLUXDB_INIT_BUCKET=${INFLUXDB_BUCKET}
    volumes:
      - ./influxdb/data:/var/lib/influxdb2
      - ./influxdb/config/:/etc/influxdb2
    ulimits:
      nofile:
        soft: 32768
        hard: 32768

And the configuration file for home assistant is the following:

# Loads default set of integrations. Do not remove.
default_config:

# Text to speech
tts:
  - platform: google_translate

automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

http:
  use_x_forwarded_for: true
  trusted_proxies:
    - 127.0.0.1
    - ::1
    - 192.168.1.0/24

recorder:
  db_url: !secret mariadb
  purge_keep_days: 10   # default

history:

influxdb:
  api_version: 2
  ssl: false
  host: 127.0.0.1
  port: 8086
  token: !secret influxdb_token
  organization: !secret influx_org
  bucket: homeassistant
  tags:
    source: HomeAssistant
  tags_attributes:
    - friendly_name
  default_measurement: units
  ignore_attributes:
    - icon
  exclude:    # Customise to fit your needs
    entities:
      - zone.home
    domains:
      - persistent_notification
      - person

Since the sensor values are in the database, I don’t think the issue is in the deployment.
I also have no errors in the logs, nor issues in the statistics dev menu.

Any idea?

[tom_l]

Why is your sensor so unreliable?

This graph should be continuous, without gaps:

Do you have wifi reception issues?

[mariolpantunes (Mário Antunes)]

The sensor read a value every 5 minutes.
The jumps are mostly 5 minutes apart.
There are some jumps of 10 minutes.
Can try to improve the WiFi connection to the device.

[tom_l]

@mariolpantunes

The sensor read a value every 5 minutes

You may have to increase your mqtt birth and will message timing.

[mariolpantunes (Mário Antunes)]

Is that an option for the client of the server?
Where should I change that?

[tom_l]

It is a client setting. If your meter sensor is Tasmota based, then you should be able to use this in the device web console:

Unfortunately it seems it is limited to 100 seconds.

As of three weeks ago someone is working on a way to disable LWT altogether: Option to disable LWT reporting for DeepSleep devices (Home Assistant integration) · arendst/Tasmota · Discussion #16031 · GitHub which could be another solution when it is eventually implemented.

The only other way is to define the sensor manually in Home Assistant rather than using discovery.

[mariolpantunes (Mário Antunes)]

I tried that, it may help with the values.
However, is that the issue with the energy dashboard?

Some hours have passed, and there are a lot of values for energy (even considering the gaps), but the bars are not being calculated.

[mariolpantunes (Mário Antunes)]

The developer from the sensor (EasyHan) helped track some bugs and we fixed them.
There were some improvements in the connectivity and resolution.
However, the main issues still remain.

The strangest part is that on the MQTT side the meter sends updates periodically (every couple of seconds):

There are several values on the Mosquitto Broker that are not displayed on HA.
One issue may be the absolute value, in most cases the value published is similar or as a difference of 0.001 unit.

[nikito7]

Something is filtering the values.

That may happen using only influx.

But influx + mariadb, I don’t know.

Here all is working correctly.

edit:

Easy HAN send all values every 10 seconds.

Maybe a json size limit?

If json is corrupted, it shows in logs.

[mariolpantunes (Mário Antunes)]

Thanks for the suggestion, I already tried without the influxdb database.
However, the problem still exists.
There are some strange messages on the MQTT logs (after activating the debug option)

> 2023-08-11 09:57:06.515 DEBUG (MainThread) [homeassistant.components.mqtt.models] Rendering incoming payload '{"Time":"2023-08-11T09:58:17","EB1":{"CH":9,"CM":58,"CS":15,"VL1":233.5,"Current":1.3,"TEI":17002.821,"TEE":0.000,"TET1":4842.038,"TET2":4036.091,"TET3":8124.692,"Power":137,"APE":0,"Factor":0.455,"Frequency":50.0,"Tariff":3,"CT1":4.60,"CT4":4.60,"LP1_Y":2023,"LP1_M":8,"LP1_D":11,"LP1_HH":9,"LP1_MM":45,"LP3_IMP":0.056},"HAN":{"s0":0.056,"s1":5.015,"s2":0.000,"ck":"09:58:15"}}' with variables {'entity_id': 'sensor.eb1_export', 'name': 'Export', 'this': <template TemplateStateFromEntityId(sensor.eb1_export)>} with default value 'default' and Template<template=({% set x = value_json.EB1.TEE|float(0) %} {% if x > 0 %} {{ x }} {% endif %}) renders=4967>
> 2023-08-11 09:57:06.515 DEBUG (MainThread) [homeassistant.components.mqtt.sensor] Ignore empty state from 'tele/edpbox1/SENSOR'

Is the message being ignored?

[nikito7]

Only the sensor.eb1_export

[mariolpantunes (Mário Antunes)]

@nikito7 thanks.
I updated my system, based on the suggestions from this forum and others, most of the changes can be seen here.

After running the new system (a clean installation of home assistant) for a couple of days, there is strange behaviour:

Let us focus on the missing energy column from 10 to 11, as seen in images 2 and 3 there are values on the recorder to make the energy estimation. I computed the difference and from 10h to 11h the energy consumption is 0.55 and from 11h to 12h the energy consumption is 0.52. adding both values together we get 1.07 which is the value of the column in 11h to 12h.

Bellow there is an image of the consumption measured from 10h to 12h.

The problem is not missing data, but just missing columns, from some reason the energy display merges some columns.

What are the requirements for these columns?

1. There is a minimal number of points per hour? Given the low consumption that I am reading maybe there are not enough points (distinct points, EasyHan sensor sends data every 10 seconds).
2. Requires data on the hour (meaning a data point at 10,11,12)?
3. Any other requirements?

Can I create a virtual sensor that uses linear interpolation to fix the gaps and allows the energy display to properly draw all columns?

[nikito7]

Only you have that issue.

Or maybe is the “unknown” values.

But it only will be “unknown” if value is zero

    - name: "EB1 Import"
      unique_id: EB1_TotEneImp
      state_topic: "tele/edpbox1/SENSOR"
      value_template: >
          {% set x = value_json.EB1.TEI|float(0) %}
          {% if x > 0 %}
            {{ x }}
          {% endif %}
      unit_of_measurement: "kWh"
      device_class: energy
      state_class: total_increasing
      device:
        identifiers: nikito7-EB1

[nikito7]

[nikito7]

Maybe HA Statistics engine fail