Custom Integration to import long term statistics from a file like csv or tsv

drcoolzic · January 4, 2026, 3:48pm

Perfect this makes sense and perfectly answer my question. Many thanks.

I have installed your component and I will now do some testing.
Currently I have an old Home assistant installation (using MariaDB) that I want to move to a new machine and of course I want to keep the statistics. The main problem is with Zigbee devices that probably will have a new IDs on the new system

drcoolzic · January 5, 2026, 1:30pm

I have ran some tests on exporting statistic data with success.

But I have strange value in the unit field for temperature?

instead of displaying °C it display Â°C
I thought this problem could have been a misinterpretation from Excel, but if I look at the output data directly in hexadecimal format there is an extra c2 char in front of the expected b0 43

any idea why?

Is there an easy way to specify that we want all the statistic entities without having to create a list of all entities which is a pain?
Is there an easy way to specify that we want all data withouut specifying start and end date (for example by not specifying them)?

I have used Yaml to run the test but if I switch back to UI mode it seems like the date picker does not show for end_time ? So no way to specify it from UI?

klaj · January 6, 2026, 2:51pm

Thats probably a UTF-8 problem. Try opening with e.g. Notepad++ and see if its fine there.

Not yet. Please create an issue at the repo.

Can it be that the end time in Yaml was wrong? Ah, I see, there are two zeros missing in the end_time (01-05), thats why.

klaj · January 11, 2026, 4:40pm

I released a new version of the integration, v3.0.1.

What’s Changed

Added

Delta Import Support

Import counter statistics (state_class increasing or total_increasing) using delta values (the change of the value in one hour) instead of absolute sum/state values:

Converts delta values to absolute sum/state values using existing database values as reference points
Useful because you
- Do not need to calculate sum and state
- Do not need to align sum and state values of the import with the sum and state values in the database, which can be a pain.
Supports these scenarios:
1. Import before oldest database value
  - Use case: E.g. the sensor was not available in Home Assistant before, but there are other sources available
2. Import after newest database value
  - Use case: E.g. the sensor was offline and recorded wrong values
3. Import in the middle of existing range (overwrites with new deltas)
  - Use case: E.g. manually add consumption data for hours not yet in the database.

It is recommended to use the delta feature to import counters

Validation

Timestamp validation to prevent future values

Changed

Import Behavior

Import is now synchronous: The import action waits until all data is committed to the database before returning

Provides feedback on import completion
Eliminates race conditions when chaining imports

Export

Already with v2.0.0 its possible to export long term statistics. With v3.0.1:

Export now includes calculated delta column for counter statistics
Delta column shows the change between consecutive hours
Exported files can be directly used for delta imports (after removing sum/state columns)

Migration Notes

This is a backward-compatible change. Existing imports using sum/state columns continue to work as before. The new delta import functionality is automatically activated when a delta column is detected in the import file.

No action required for existing users. New delta import feature is opt-in by using the delta column.

Needs Home Assistant 2025.12.0 or higher

Pull requests and issues

feat: delta for counters by @klausj1 in https://github.com/klausj1/homeassistant-statistics/pull/156
fix: solve python lib incompatibility by @klausj1 in homeassistant 2025.12.0 incompatible with Python 3.13 ? · Issue #155 · klausj1/homeassistant-statistics · GitHub
fix: skip import of timestamps that are in the future by @klausj1 in Skip import of timestamps that are in the future · Issue #154 · klausj1/homeassistant-statistics · GitHub

lucyconnuk · January 12, 2026, 9:50am

Hi there. I tried to install the latest version (v3.0.1) of your integration using HACS and I’m getting the error “This integration does not support configuration via the UI. If you followed this link from the Home Assistant website, make sure you run the latest version of Home Assistant.”

I followed these instructions:

My Home Assistant is up to date - versions are:

Installation method-Home Assistant OS
Core-2026.1.0
Supervisor-2026.01.0
Operating System-16.3
Frontend-20260107.0

Not sure whether this is a bug, or whether there’s something wrong / different about my setup. I will try and install manually but wondered what you thought.

Thanks for all your hard work!

lucyconnuk · January 12, 2026, 12:59pm

Ignore this - I did get it installed via HACS. Not sure what I did differently…?

alexd_ha · January 18, 2026, 10:38pm

Hey, thanks for this integration it looks very promising.

I wanted to do some tests with external entities (using delta) but everytime I’m facing an error message

Entity 'import:foo': No statistics found in database for this entity

I had the feeling integration would create it if I use : in name. Did I miss something ?

klaj · January 19, 2026, 11:56am

Thats true, in this case the integration should create the entity automatically.

But as you are doing an import with delta values, this is not possible when there is no entry in the database at all (at least not yet), so you have to first import at least one value with sum and state. Further imports can be done with deltas.

The reason for this is that sum and state are stored in the database, not delta, and I try to align whats in the database with the imported delta. Otherwise I would have to “invent” state and sum.

Please create an issue in the repo, at least the error message has to be improved, and I will think about to “invent” state and sum if nothing is in the database.

alexd_ha · January 19, 2026, 9:19pm

Thanks, it makes perfect sense indeed. I fixed my issue by creating a sensor first.
As you said, a basic error message saying it can’t be initialized based on delta would help (or an option to init at 0 may be ?)

Thanks for your work on this integration, it helped me a lot for what I wanted to do (+ on how HA is working internally !)

klaj · January 25, 2026, 12:52pm

Version v4.0.0 released

Major improvements

Entity Export

Optional entities parameter: Export all statistics when entities is omitted or empty
- Previously required; now defaults to exporting all available statistics
- entities parameter now accepts glob patterns using * (e.g. sensor.*_temperature). Useful when exporting a subset of statistic IDs without enumerating them all.
Optional start_time parameter: Export from earliest available statistics when omitted
- Allows exporting complete historical data without knowing the exact start date
Optional end_time parameter: Export up to most recent statistics when omitted
- Allows exporting all data up to present without specifying end date

Thx a lot to @drcoolzic for this!

Misc

Timezone parameter now optional, uses HA timezone as default
More checks to avoid errors when importing
Improved documentation
Some bug fixes

For details, check changelog.

drcoolzic · January 25, 2026, 6:52pm

Thanks Klaus for releasing 4.0
Note: There is also an output option to automatically split the output into separate sensors and counters files for easy import in target system

drcoolzic · February 3, 2026, 12:56pm

There are two changes I would like to propose for this integration, but before submitting them as issues or pull requests, I think it would be good to discuss them.

The first change concerns the fact that it is required to specify the separator character for both reading and writing statistics, as well as the fact that the default separator is a tab.

It is customary for TSV files to use tabs as separators by default and for CSV files to use commas as separators by default. So, I propose the following approach for both writing and reading statistics:

if the optional separator is specified, then that separator is used
if the separator is not specified, then commas are used as the separator for CSV files and tabs are used as the separator for TSV files

The second change does not affect the code, but simply the documentation. In the current documentation, statistics are divided into two types called sensor and counter. Unfortunately, Home Assistant does not define terminology for these two types, so we can effectively use whatever we want. I think that the choice of counter is appropriate and accurately reflects their use, but the use of the term sensor does not seem appropriate to me because it is misleading. There are two reasons for this: on the one hand, counters are mostly in the sensor domain, and sensor can belong to different domains. As a reminder, the two types of statistics are defined by the value of the state_class attribute.

I therefore propose using the following terms:

measurement: if the state_class is equal to measurement or measurement_angle
counter: if the state_class is equal to total or total_increasing

klaj · February 3, 2026, 6:46pm

I totally agree to both proposed changes.

drcoolzic · February 3, 2026, 7:07pm

OK if this is fine with you I will include the terminology modifications in my PR
For the separator I need to evaluate and see if I can stick it in the same PR

drcoolzic · February 9, 2026, 12:11pm

New features in 4.1

Exporting Statistics Inventory

Export a metadata-only inventory of all long-term statistics. This is useful for:

Migration planning: Compare statistics between old and new Home Assistant instances
Database analysis: Understand what statistics exist and their sizes
Cleanup: Identify deleted or orphaned statistics

Inventory Output

A summary block, followed by a table with one line for each entity in the statistics DB

# Total statistic_ids: 47
# Measurements: 28
# Counters: 19
# Total samples: 183
# Global start: 2025-06-29 07:00:00
# Global end: 2026-02-04 15:00:00
# Internal statistic_ids: 38
# Deleted statistic_ids: 0
# External statistic_ids: 9

statistic_id	unit_of_measurement	source	category	type	samples_count	first_seen	last_seen	days_span
sensor.cnt_all_changed	kWh	recorder	Internal	Counter	6	12/29/2025 8:00	2/4/2026 15:00	37.3
sensor.cnt_all_changed_new	L	recorder	Internal	Counter	1	2/4/2026 15:00	2/4/2026 15:00	0
sensor.cnt_part_overlap_new	m3	recorder	Internal	Counter	8	12/29/2025 8:00	2/4/2026 15:00	37.3
sensor.cnt_some_changed	kWh	recorder	Internal	Counter	1	2/4/2026 15:00	2/4/2026 15:00	0
sensor.go_echarger_238557_wh	kWh	recorder	Internal	Counter	1	2/4/2026 15:00	2/4/2026 15:00	0
sensor.imp_after	kWh	recorder	Internal	Counter	7	12/29/2025 8:00	2/4/2026 15:00	37.3
sensor.imp_before	kWh	recorder	Internal	Counter	8	12/28/2025 8:00	2/4/2026 15:00	38.3
sensor.imp_inside_same_newest_1	kWh	recorder	Internal	Counter	13	12/29/2025 8:00	2/4/2026 15:00	37.3
sensor.imp_partly_before	kWh	recorder	Internal	Counter	16	12/29/2025 5:00	2/4/2026 15:00	37.4
…

Automatic selector definition based on file type

The delimiter definition is now optional

If not specified, it is inferred from the filename extension:
- .csv uses comma (,) by default
- .tsv uses tab (\t) by default

Note The only file extensions allowed are .csv or .tsv otherwise an error is generated

drcoolzic · February 13, 2026, 11:19am

Thanks to @geoffreycoan for reporting errors on export inventory services. The deleted statistics information was not reported correctly. The actual version only rely on the database tables but correct solution also requires to consult the core.entity_registery.
For more information about deleted statistics please read my documentation on the subject Part 5 - Find & Fix Statistics Errors - Understand and Use Home Assistant Statistics
Release 4.1.1 Will fix this problem and add a new category: orphan statistic
For more information about orphan statistics please read my doc Part 5 - Find & Fix Statistics Errors - Understand and Use Home Assistant Statistics