I’m trying to understand where sensor values are stored, I know there is a “recorder” which stores values in “home-assistant_v2.db” but I have also installed InfluxDB so that I can use Grafana to display various graphs. This is a newish installation, previously I ran HA (less InfluxDB and Grafana) for a year with no issues but this time around with InfluxDB and Grafana it has crashed a number of times, but that’s not why I’m here.
My biggest issue at the moment is I’m trying to build up a week’s worth of data to play with but when HA crashes I lose hours of data. For instance, today it crashed at 11:00 am, I was there and immediately checked the history, but all data from 2:00 am onwards was lost!
At first, I thought it was fine as data is being recorded by the recorder integration and by InfluxDB but alas both areas were devoid of any data in that 9 hour period.
Just in case it was anything to do with my sensors (which come from DeConz) I also checked the history of the solar_anglex software sensor, and those values were missing as well.
I’m running an approved supervised version, latest updates etc, on a Linux machine.
My recorder integration is set for
purge_keep_days: 8
commit_interval: 1
My InfluxDB has a retention policy of 8d
I simply can’t understand how a system crash can erase 9 hours of historic data that should already have been committed into the InfluxDB database. Any theories?
As a side note, if I don’t intend to use history cards in Lovelace, instead just use Gafana dashboards, do I actually NEED recorder, can I disable it?
it wouldn’t delete 9 hours of data. You most likely had a recorder issue 9 hours ago and you didn’t notice it. Always check your logs. Also, what database are you using for recorder? If you don’t have anything specified, everything is going into home-assistant_v2.db.
As for influx, it’s an additional DB. You’re storing twice the information essentially. If both are failing at the same place, you have something wrong with your HA.
Lastly, HA is pretty stable, you shouldn’t have crashes. I recommend going through your logs and fixing all errors that you see.
Thanks Petro, I appreciate your input and agree entirely with your last comment, I’m only a week into this build and will be going through the logs when time permits, but it’s been hectic.
Interestingly, just before the system crashed at 11:00 am today, the missing data was definitely showing in both Lovelace and Grafana. I was commenting to a friend that it was the first time the outside temperature had gone below zero this autumn, twice, once at 6:00 am and again at 8:00 am - so that data was in the system somewhere - I’m mystified.
Recorder was using the default DB (home-assistant_v2.db) so now you’ve confirmed I don’t need it, I shall remove it from my config file and delete the home-assistant_v2.db file.
Just before it crashed I had made a slight change to the Grafana dashboard but it wouldn’t allow me to save those changes. It was then that I decided to restart HA but it failed to come back and I had to reboot the Linux host.
More info, I just checked the HA logbook and there are no entries between 02:13 am and 11:27 am. Maybe I should rebuild the Linux box and start again.
Yes but was that on the graph itself or when you were viewing the sensor in the frontend? The graphs are built off the data stored in the database. The frontend still displays the current sensor information with or without information being stored in a database.
That’s an indication that recorder was not recording just as I thought.
You should check the logbook regulary. When it stops recording check your HA logs and see what errors are in the logs.
Use Home Assistant DB for short-term storage, for instance 7 days. You can use alternative DB manager, MariaDB is probably the most common DB alternative, way more powerful than the default HA DB manager
Use InfluxDB for long-term storage
I don’t say it is the best way to handle storage but it’s a very common way, and it’s also mine.
But first, you have to find why your HA crash. It should not. HA is rock solid when used on a correct hardware. I didn’t have any crash for months and having lots of integrations/modules/automotions/…
Juste take care about updates. I think it’s better to disable automatic updates, perform backup before each update, and run updates manually.
Yes, Petro, it was the graph in Lovelace and the Grafana dashboard. I’m sure it was generating those graphs from a database as I played about with the time periods to zoom in on the two coldest temperatures. But after I restarted HA, there was just a flat line.
Thinking back three days I had a similar experience, although everything was still working, things coming on and off etc. But I noticed all the sensor data had stopped changing at 4:00 am, no one was awake and nothing special happens at that time. I just rebooted the system and forgot about it till today.
Thanks, David, as a nube, it’s so useful to know what others have done and it now makes a lot more sense. After my last posting, I realised I can’t disable recorder as it’s needed for the logs etc so I have set it for as this to keep it small:
recorder:
include:
domains:
- sun #### just a token small domain to exclude the rest
purge_keep_days: 3
commit_interval: 1
I have to stop for today, I’ll leave it running and see what happens tomorrow. I suspect the easiest route will be to reformat the Linux SBC and start again, I’ve only invested a week or so on this build so haven’t too much to lose. Yes, the last time I had it running for about a year and it never gave me any issues. Thanks guys, much appreciate your input. I’m quite old and know nothing about Linux, docker, containers etc but I’m willing to learn if I get a few pointers.
Hi guys, I’ve just come across this page. It just happens that the “Entity Controller” is one of the few things I have installed.
If it’s filling up a log and then crashing, that might explain why it happens in the middle of the night:
My log is set as the default so I may not have noticed certain events, I will set it to a more verbose setting.
