Why you (probably) don't need InfluxDB in HA

InfluxDB is a time-series database. If you’re storing a time series, you therefore need InfluxDB.

This is the (circular) argument for using InfluxDB with HA. However you probably don’t actually need it if you’re already using properly set-up MariaDB. By “properly set up MariaDB” I don’t mean SQLite, and I don’t mean MariaDB with the default HA configuration which is set up for gutless devices like Raspberry Pis and acts as a kind of straightjacket for performance on anything that’s more powerful than a Pi, which in practice is almost everything.

The following isn’t meant as a guide for MariaDB tuning but just some quick notes, this is the default config vs. the config for anything more powerful.

Pi Not-a-Pi
Buffer Pool Size 0.1GB 8GB
Log File Size 48MB 512MB
Temp Table Size 16MB 512MB
I/O Capacity 200 2000
I/O Threads 4R, 4W 8R, 8W
Table Cache 64 4000

Then change the commit_interval from the default 5s (some docs even say 1s) to 30s. Finally, keep an eye on what’s being recorded:

SELECT states_meta.entity_id, COUNT(*) cnt
FROM states LEFT JOIN states_meta ON ( states.metadata_id = states_meta.metadata_id )
GROUP BY states_meta.entity_id
ORDER BY cnt DESC
LIMIT 20;

and exclude anything and everything you don’t need. You should only be recording historical data for things you actually care about, not everything in existence. Since you’re now storing a lot less data, you can change the retention period to a much larger value without having an explosion of storage space used.

Looking at things from an InfluxDB perspective, it’s a time-series database but it’s not a very good one in terms of doing what you need with HA. Years ago when I worked with an in-house historical data store you configured it to downsample from 1-minute intervals to 5-minute intervals after a week and to 60-minute intervals after a couple of months. This was all automated and transparent, the same query would return 1-minute, 5-minute, or 60-minute data depending on how far back in the past you went. In fact it’s pretty close to what HA does for you as well, storing short-term stats binned into 5-minute intervals and downsampling to hourly and keeping the data forever after purge_keep_days have elapsed.

At this point I should say I’m not an InfluxDB expert, I’d just seen the circular argument in various places and thought I needed InfluxDB and expected it to work like the historical data store mentioned above. However, InfluxDB doesn’t work like this, you need to manually downsample using its awkward query language and store the results in another bucket or set of buckets if you’re downsampling more than once and then know that if you’re querying one date range you get the data from this bucket and another date range from that bucket, or perhaps an overlap across two or more buckets.

Does this sound familiar? It’s exactly the same thing you’d be doing if you simply let HA take care of things for you via the already-present MariaDB, only now you’re running a second database alongside MariaDB with its own quirks and complications, with a second lot of data storage that partly duplicates stuff already in MariaDB, and having to do all the data management yourself.

This isn’t to say that InfluxDB isn’t useful in general, for example if you need to store historical data and don’t already have a database available, it’s just not very useful in HA compared to just using the already-present MariaDB store for everything and letting HA take care of things.

I am not completely sure about everything in HAs LTS, but I think when data is over the threshold for time to live in the Short term database, then it is downsampled to 5 minutes intervals and stored in the LTS database, where it will stay until a manual purging happens.

So downsampling is happening and I do not think data is represented twice in the databases.

Downsampling is the one of many features.

Timeseries focused databases are designed to be specialized in handling timeseries. They are optimized to handle writes and reads and aggregate of big amount of data like no any other db.

They provide tools for making analytical tasks over those data.
It’s very important for fetching them even for visualization. I bet you cannot achieve lot of those tasks on MariaDB, sqlite aside (like resampling data on the fly, adding missing measurement points, inter/extrapolating etc)

Timeseries databases might provide hyper-efficient methods for transparent compression.

Speaking about downsampling, they provide more flexible control over downsampling and fetching downsampled data.

One of most important things when you store data for ling time is reliability. you don’t collect years worth data just to lose them accidentally. Sqlite doesn’t fit here with its fragility.

Finally they are often designed with multiple clients using them simultaneously (incl writes) in mind, unlike sqlite.

So, if you seriously need store and then work out on years worth timeseries data, using dedicated timeseries database is just a proper choice.

btw Influx is not the only one option. There are more. Personally I love TimescaleDB. It’s built over postgresql (as extension) so you can have all pg features and mix them with timescale hypertables (columnar store) with all their amazing features.

I think many (most?) users wouldn’t even know where to change those MariaDB settings you refer to. :man_shrugging:

I definitely agree that the internal DB is not setup optimally for higher capability servers/VMs though. There should be an easier way to pick a correct set of defaults/settings based on server capability.

I also like TimescaleDB a lot.
Currently using Influx (V2), but with the end of flux I also think about migrating to something different.

Using something (configured by yourself, or already optimized by the developers) that can handle time series in a good way is always a good idea → meaning that fetching stuff is really optimized for that task.
It will speed up requests (and your graphs in something like Grafana) a lot.

Most important to me is, that it’s independent from HA.
There are so many reports about lost statistics after exchanging devices or only because of configuration changes in HA, that I don’t want to rely on the automatic management of the historical data.

If I save it in my own DB, and the data structure coming from HA changes, I can migrate the old data on my own, but it doesn’t get deleted for unknown reasons.

The reason I use InfluxDB is because there are many visualisations I can create with Grafana that are not possible (or a lot more difficult) with the options available in Home Assistant. e.g. Monitoring shading on my solar panels:

Having a separate visualisation suite to Lovelace is not an issue for me.

That is not due to the databases,but rather related to people using device IDs instead of entity IDs.
Device IDs are unique to the specific device, so another device can not get the same device ID and then history will be lost.

As I always used my own DB for visualization, I’m not sure if I can follow. :smile:
It’s not really clear to me how people are in control of that, can you describe this a little bit further?

E.g. I have some devices automatically created by an integration in HA.
These have entities.

Now let’s say one of these devices has an energy measurement in one of its entites.
Then I select this measurement in the dropdown of the energy dashboard.
It’s not clear to me if this uses a device or a entity now.

And if I replace the device but stick to the same entity names by renaming them to the old ones, will the history of these devices still be visible in the energy dashboard? And if not, should I at least see old and new history statistics using this entity id when manually requesting it (e.g. history actions in dev tools)?

Sorry for this noob questions, but as I use a seperate DB I’m not really in this topic.
So it might be good to correct my statement above in case I was wrong. :wink:

It really doesn’t matter. To maintain continuity of history, you need to ensure that entity id will never change. For years…
It’s doable as long as you don’t change measuring devices, don’t make changes to the house etc. Otherwise, it’s virtually impossible to secure entity id not changing over time. HA alone provides no tools to make it happen.

You might try to dig in sqlite database, but… You need to do that offline. Besides it means HA outage any mistake may cause havoc in HA. Also, some values are cached in json files.

On the other hand, once you have all that data in an additional database, you may do any transformation you need. You can change entity names or data units retrospectively to match the current setup. etc etc. And you do that without risking HA, without outage, at any time. And on top of that, any operation is faster by magnitude

I changed my PV array sensor but I can graph both the old and new entity ids contiguously in Grafana:

SELECT last("value") FROM "kWh" WHERE ("entity_id" = 'self_solar_consumption_daily' OR "entity_id"::tag = 'self_solar_energy') AND $timeFilter GROUP BY time(1d) fill(null) tz('Australia/Hobart')

I agree. As I said you can do anything once you have data in external databases
You cannot achieve that in HA alone (to maintain the history continuous)

You quoted me before I did a sneaky edit, after reading the rest of your post :slight_smile:

Ah, good point, I’d forgotten about the built-in short vs. long-term data handling. I’ve updated the original post to mention this.

I posted a CliffsNotes version because I didn’t want to repeat a lot of existing material (and it’d be a very long post), if you google “home assistant database tuning” you’ll find lots of posts and articles, just choose the one that seems to describe it best. The example I gave in the table was for an ODroid H4 (16GB RAM, 512GB SSD), the tuning really depends on what your hardware is capable of so don’t take that as gospel.

I agree that it should be something the HA install sorts out for you instead of throttling everything back to the least capable hardware that you can run HA on. Judging by the comments in various tuning threads, lots of people are running extremely suboptimal configs for the database because that’s what the default setting is.

That was actually my reason for looking into InfluxDB in the first place, I wanted to play with Grafana. However after looking at what it’d involve I decided the existing MariaDB and RomRider / vingerha’s ApexCharts card does most of what I want except for Grafana’s exotic display capabilities, which would look kinda cool but then I’d have to rework everything in Grafana and I already spend way too much time on HA as it is.