PSA: 2024.7 recorder problems

[bdraco (J. Nick Koston)]

2024.7.3 which is currently scheduled for the end of the week will better handle low resource systems.

Make sure you have at least as much free disk space available as the size of your database at all times. This is general advice not specific to the table rebuild.

For the table rebuild it is preferred to have 2.5x the size of your database available in the event your system previously experienced an unclean shutdown (ie power loss, bad power supply, bad sd card, other hardware failure) which corrupted the states table in the past, and the corruption is only encountered during the table rebuild.

[VanFlipsen]

I have same issues regarding the purge at 4;something even running 2024.7.2.

This morning recorder was not functioning, changes like window sensors are not noticed/recorded by home assistant, my rolling shutters where closing against an open window. Now i have to repair 1 of my rolling shutters.

Another issue with my database lies on energy dashboard (LTS?) in 10 september 2023, cannot access data that Day/Week/Q3/Year I dont know how to fix that. maybe that is a issue causing the trouble with purging.

[wizard7]

thanks, I’ve tried that, but it looks like it doesn’t even start to rebuild

[RezzZ (ReneS)]

also here problems with 2024.7+, recorder is stuck.
Running HA for years (unraid server, HA docker, MariaDB v11 latest docker.

Solution 1 to disable autopurge doest prevent recorder to break in my setup and because history is not available any more some automations won’t run.
For now I’ve reverted back (as suggested) to 2024.6.4 and all has been stable for 2 days now.
I’m not running any of the other custom integrations mentioned in issue 2. Happy to provide additional info if required (for the next 2 days till I go on holiday )

My custom integrations from HACS:
Homewizard Climate
Daily Schedule
Switch Manager
Electrolux Care Integration V2 (Not Official)
nordpool
simpleicons
midea air appliances (LAN)
Tuya Local
Neerslag App

edit: hmm reverting back seems to have broken al lot of other pages seems…
my developer tools settings pages now all show “Error while loading page” oops

[wizard7]

I’ve restored a backup from 9 days ago, when I didn’t have this issue.
It was 2024.7.1, I’ve updated to 2024.7.2, after reboot it started to rebuild the DB.

I hope it will solve the issue. I’ve lost 9 days history, but I couldn’t make it rebuild any newer DB I had.

edit: it works now

[bdraco (J. Nick Koston)]

Two of the changes were released today in 2024.7.3

Narrow sqlite database corruption check to ensure disk image is malformed

Log add/remove index complete at the same level as when it starts

The remaining two will be in 2024.8.0

Fix recorder max backlog calculation being too small

Ensure states table rebuild still happens if the event_id index was removed

[sebazzz]

I also have the issue in 2024.7.2:

# sqlite3 home-assistant_v2.db
SQLite version 3.44.2 2023-11-24 11:41:44
Enter ".help" for usage hints.
sqlite> .schema event_data
CREATE TABLE event_data (
        data_id INTEGER NOT NULL,
        hash BIGINT,
        shared_data TEXT,
        PRIMARY KEY (data_id)
);
CREATE INDEX ix_event_data_hash ON event_data (hash);
sqlite> .schema events
CREATE TABLE events (
        event_id INTEGER NOT NULL,
        event_type VARCHAR(64),
        event_data TEXT,
        origin VARCHAR(32),
        time_fired DATETIME,
        created DATETIME,
        context_id VARCHAR(36),
        context_user_id VARCHAR(36),
        context_parent_id VARCHAR(36), data_id INTEGER, origin_idx INTEGER, time_fired_ts FLOAT, context_id_bin BLOB, context_user_id_bin BLOB, context_parent_id_bin BLOB, event_type_id INTEGER,
        PRIMARY KEY (event_id)
);
CREATE INDEX ix_events_data_id ON events (data_id);
CREATE INDEX ix_events_time_fired_ts ON events (time_fired_ts);
CREATE INDEX ix_events_context_id_bin ON events (context_id_bin);
CREATE INDEX ix_events_event_type_id_time_fired_ts ON events (event_type_id, time_fired_ts);
sqlite> .tables
event_data              schema_changes          states_temp_1720627841
event_types             state_attributes        statistics
events                  states                  statistics_meta
migration_changes       states_meta             statistics_runs
recorder_runs           states_temp_1720625711  statistics_short_term

Database looks fine, doesn’t it?

[nickrout]

Have you read the post immediately above yours?

[bkbartk]

2024.7.3 fixes my energy dashboard issues.

now I noticed that sometimes (on iOS) the interface won’t load,
don’t know if this is related to anything

[parautenbach (Pieter Rautenbach)]

@bkbartk

now I noticed that sometimes (on iOS) the interface won’t load,

Try to reset the frontend cache (under the companion app settings). I’m not sure it will solve an intermittent issue though, but worth a try.

[GarryG]

Not sure if this is the same issue, but when upgrading from 2024.6.4 to 2024.7.2 or .3, while the normal entity charts look fine, all the History access is stuck … i.e., I click a chart, the popup with the history charge opens, but the data keeps loading, forever … how can I diagnose the cause of this?

[zigomatic]

May be link to the DB rebuild, this take times. Check your logs

[talormanda]

I was told in discord this was fixed with 2024.7.3 , but I still have this problem. At 4 AM my recorder stops working, and manually calling a ‘purge’ in the developer tools section also recreates the issue. What’s going on?

[GarryG]

OK, for the .3 update, I just checked immediately after the update because for .2, I lost 2+ days of data due to going back to the .6.4 backup, so I didn’t want to risk it … can I somehow initiate the rebuild (or whatever it is that’s stopping the history from working), in order to see if it is indeed fixed with the .3 patch?

[denwald]

If you run HA in a manually managed Docker container, you could spin up a second container to test the migration procedure. You need to make sure though, that you have enough free RAM and disk space. This is what I did.

Edit: Sorry, I misunderstood your question. The next answer after mine is more appropriate I think. You can also check the logs to see if the migration was successful.

[CaptTom (Tom)]

In Developer Tools / Services you could select Recorder: Purge. Select a Days to Keep, then Repack, then be sure to slide the switch next to Repack to “on.” (No, I don’t know why HA requires you to select the same thing twice on the same line.)

[parautenbach (Pieter Rautenbach)]

The one is to include the option and the second is to provide its (new) value. What makes it feel silly is that it has a default, so if you deviate from that, it seems like doing the same thing twice. A better UX would be to enable the option automatically the moment you change the default value.

[CaptTom (Tom)]

@parautenbach

The one is to include the option and the second is to provide its (new) value. What makes it feel silly is that it has a default, so if you deviate from that, it seems like doing the same thing twice.

I’m not sure I understand. From what I can see, there are two binary selections for “Repack”; one check box and one slider. There are no other options. Unless I’m missing something, both of them have to be selected or nothing happens, right?

Obviously either one of those selection methods would be fine, by itself. Or, as you say, have the UI automatically select one when the user selects the other. That would still be silly, but at least it wouldn’t be a trap for the unsuspecting user who might not notice the on/off slider way over on the right.

A minor point perhaps, but I do think it’s only fair to mention it whenever instructing someone to use this option.

[parautenbach (Pieter Rautenbach)]

@CaptTom

I’m not sure I understand. From what I can see, there are two binary selections for “Repack”; one check box and one slider. There are no other options. Unless I’m missing something, both of them have to be selected or nothing happens, right?

Sorry, I thought you were referring to this.

[CaptTom (Tom)]

Yes, exactly. At first I would check the “Repack” box, but forget to go over and turn “on” the slider switch. It seemed to me no repack took place unless I did both. If that’s the case, it seems a less than optimal UI. Or maybe I’m totally missing the point.