PSA: 2024.7 recorder problems

Configuration
3
8 Likes
4

Thanks both of you for the heads-up!

I noticed a trend in the release notes thread which seemed to point at Recorder issues. Nice to have specifics so we know whether or not this impacts us.

5

Aha, that explains why my recorder is borked. I am running places. Weird thing is that there are no meaningful messages in the logs when this happens.

1 Like
6

hi there! care to elaborate on the symptoms? since upgrading, recorder stops working every night right after 4am. haven’t been able to find any meaningful log entry. it resumes after a restart, when i see a log entry saying “The system could not validate that the sqlite3 database at //config/home-assistant_v2.db was shutdown cleanly”. is this behavior consistent with the culprit integrations you mention or should i keep digging further? i am using hass-variables, just updated it to the supposedly fixed version, will see tomorrow morning if this was the issue. thanks!

5 Likes
7

Same here, every night just after four in the morning the recorder stops. I have the same error message about sqlite when restarting HA to get it running again.

I don’t have any of the above mentioned integrations but a few others.

4 Likes
8

as a workaround, you can configure your recorder to not auto_purge.
To do this, edit your configuration.yaml:

recorder:
  auto_purge: false

this should fix the issue for now - BUT: keep in mind to remove this change when a fix has been applied, else, your recorder will grow and that will cause other issues long term.

@silfax - same recommendation for you.

4 Likes
9

Does anyone know where the issue actually stems from? It seems like the recorder is “pausing” overnight, but that seems like a very strange kind of bug.

10

Have u read the first message in this thread?

11

yes, it is known.
in some cases it is related to some custom components.
in other cases it is related to an issue with the database - where an faulty index is causing a full tablescan → therefore, the db is in a locked state.

second issue will be fixed but requries some heavy database actions and therefore, can probably not be rolled out with a patch-release.

12

I have the exact same issue. Not using any of the mentioned integrations either. It’s been like this since upgrade to 2024.7 :confused:

4 Likes
13

Yeah, I also don’t have any of the potentially problematic integrations…and never had any trouble with the Database either…
Somehow the 2024.7 release seems to have more issues than usual. My HA Server seems to get “stuck” at 30% CPU usage regularly since I updated for some reason. RAM useage is also unusually high…

Just checked the logs and there seems to be some serious trouble with the “recorder”. It is running out of memory and CPU time and has now stopped completely. My “Helpers” like utility meters are apparently also affected…this is not good…

3 Likes
14

had the same problem with the EnergyDashboard as many here. For me it was definitely iCloud3. After importing backups a few times, I uninstalled iCloud3. Then the update and the Energy Dashboard worked without any problems.

1 Like
15

iCloud probably triggers it earlier, but I doubt removing it will solve the underlying issue. From what I understand, the recorder “purge” isn’t working and hangs, and that causes anything that relies on the Recorder to break down.

16

Nope, custom integrations like icloud and other integrations are incorrectly applying recorder filters, which is causing recorder to break down.

17

Is that definite? Because I don’t think I have any of the “offending” integrations and my recorder breaks reliably at 04:00 AM no matter what I do? Restarting HA fixes it until the next time 04:00 rolls through and deactivating the scheduled Purge seems to fix things for those affected (can’t verify myself, because I only enabled the workaround today and have to wait for the next 04:00 AM time window).

18

Yes, this is definite. Look in your logs for errors, post them here. The first post mentions that we don’t know all integrations that are causing the problem.

1 Like
19

Logger: homeassistant.components.recorder.util
Quelle: components/recorder/util.py:610
Integration: Recorder (Dokumentation, Probleme)
Erstmals aufgetreten: 14:08:00 (1 Vorkommnisse)
Zuletzt protokolliert: 14:08:00

Ended unfinished session (id=688 from 2024-07-05 18:29:01.716302)

Logger: homeassistant.components.recorder.util
Quelle: components/recorder/util.py:331
Integration: Recorder (Dokumentation, Probleme)
Erstmals aufgetreten: 14:07:59 (1 Vorkommnisse)
Zuletzt protokolliert: 14:07:59

The system could not validate that the sqlite3 database at //config/home-assistant_v2.db was shutdown cleanly

Those are relevant to the Recorder. Before the latest reboot I had more, mostly telling me that the Recorder queue is full and thus nothing new can be added by the Recorder. The issue tracker on GitHub shows pretty much the same and the general advice is to deactivate the “purge” as that works as a stopgap for now while the database/recorder purge issue has been fixed.

20

Can you please post the full stack trace from the text logs, not the ui. The previous log is also stored in home-assistant.log.1

21

I am afraid not. I have restarted and even rebooted several times today while trying to get my energy dashboard and graphs to work again, so my earliest logs are from 14:00 today.
Since the recorder freezes at 04:00 and then works again after a reboot/restart my logs don’t have anything yet.
And since I now disabled the purge temporarily I doubt I will have any issues, but I will report back here tomorrow of the issues show up again.

22

If you want you can force a recorder purge by running:

service: recorder.purge
data:
  repack: false
  apply_filter: false

So you don’t have to wait until tomorrow morning 4:10.

1 Like