Short vs. Long Term Statistics?

[josephny (Joseph)]

I use Emporia Vue to monitor electric usage, with the associated integration and it has been working great.

When I use Plotly, I can only see a few weeks of data.

But, when I use the entity history, I can see many months of data. It says source: Long Term Statistics.

What is a good way to chart or graph 5 minute peaks (of energy usage) across a several month period?

Plotly showing the 5-minute averages:

History of several months, but I don’t think it is showing the average peaks every 5 minutes:

History card explorer seems to show the 1 hour peaks, even thought the entity is the 1-minute entity:

Thank you.

[tom_l]

After the recorder purge keep days setting (10 days by default) the 5 minute LTS data is discarded. Only the hourly max, min and average LTS data are kept forever.

ApexCharts can plot LTS too.

[josephny (Joseph)]

Thanks, Tom.

Unfortunately, I can’t say I fully understand.

Is there a way to keep the 1-minute data that is collected by the device?

[Ildar_Gabdullin]

“Normal” history is kept within “purge interval” (10 days by default, may be changed).
Statistical data which is re-calculated every 5 minutes - is also kept within “purge interval” (let’s call it “short term LTS”).
Statistical data which is re-calculated hourly - kept forever (so called LTS).

History page shows “augmented history” (“normal” history + LTS for periods beyond “purge interval”) - for sensors with LTS stored, and shows only “normal” history for other entities (which are not excluded from Recorder).
History in more-info shows either “short-term LTS” (for sensors with LTS), “normal” history (for other entities) or nothing (for entities which are excluded from Recorder).
Some custom cards may show LTS (or “short-term LTS”).

If you need a “normal” history for some entity or “short-term LTS” (5-minutes) for a period longer than 10 days - you need to change “purge_interval”.

[josephny (Joseph)]

Wow, that is a great and detailed explanation.

If I wanted to keep 1 or 5 minute data for a whole bunch of sensors for several months, what would be the smartest way to acheive this?

I assume that the purge interval defaults to 10 days because storing and manipulating huge amounts of data is resource-intensive, and therefore changing the purge interval to 360 days (for example) is not advised.

In configuration.yaml I have:

recorder:
db_url: “mysql://homeassistant:@core-mariadb/homeassistant?charset=utf8mb4”
commit_interval: 10

If it is okay to do this, would I just add:

purge_keep_days: 360

Or is there a smarter way? The desire for this data is rare, but every so often I would like to see the peak usage in the past several months.

Thank you.

[Ildar_Gabdullin]

There are different opinions.

Not sure about any advantages of MariaDB nowadays.

Start with defining rules for Recorder for what entities should be stored.
Then start increasing “purge interval” to 1 month, then more…
Some people set “purge_interval” for 1 year & then complain like “DB is damaged etc”.
Probably if you keep only really needed data in DB - it will be more stable, no idea, I am not a DB expert.

(my humble experience: DB 1.3GB, 30 days, ~3K entities, some of them are not stored in DB)

Also, some people store important entities in InfluxDB for a longer period.

[tom_l]

I would not recommend keeping more than 30 days in the recorder database. Based purely on issues raised on this forum.

What do you want this data for?

If it is only visualisation then your best option is to export the data to a time series database like InfluxDB.

I keep two years worth of selected data this way for charting and comparison. It totals about 4GB and performs well. See: Update from 7.4 to 8.1 , DB can't migrate - #10 by tom_l

Here’s a quick tutorial: https://youtu.be/m9qIqq104as?si=qjYOBbFAiChsUtwR

The only thing I’d add to that is to use includes in your InfluxDB configuration so that you are only saving data you are actually interested in. e.g.

influxdb:
  host: a0d7b954-influxdb
  port: 8086
  database: homeassistant
  username: homeassistant
  password: !secret influxdb_pssword
  max_retries: 3
  default_measurement: state
  include:
    entities:
      - sensor.12v_bus_voltage
      - sensor.12v_current
      - etc...