History is missing a lot of data points

This is a follow up to my post from here: Energy dashboard stopped tracking energy but sensors are still reporting readings - #4 by benbergman

I recently noticed that the history for my entities is not updating as frequently as it should (ie. entities show updates every couple days instead of the expected every minute updates). This data has been missing for a few weeks. Prior to noticing the issue, I migrated my install from a Raspberry Pi 4 to an x86 mini PC. Within a few hours of the new server coming online my history started reporting correctly, but 24 hours later the history is again not updating.

I’m using MariaDB add-on v2.6.1. I’m not seeing any errors in that add-on’s logs, but there are a couple warnings. I am also seeing high CPU usage (25% on my i5 7500T).

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service mariadb-pre: starting
[11:50:31] INFO: Using existing mariadb
s6-rc: info: service mariadb-pre successfully started
s6-rc: info: service mariadb-core: starting
s6-rc: info: service mariadb-core successfully started
s6-rc: info: service mariadb-post: starting
[11:50:31] INFO: Starting MariaDB
2023-10-12 11:50:31 0 [Note] Starting MariaDB 10.6.12-MariaDB source revision 4c79e15cc3716f69c044d4287ad2160da8101cdc as process 66
2023-10-12 11:50:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-10-12 11:50:31 0 [Note] InnoDB: Number of pools: 1
2023-10-12 11:50:31 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-10-12 11:50:31 0 [Note] mysqld: O_TMPFILE is not supported on /var/tmp (disabling future attempts)
2023-10-12 11:50:31 0 [Note] InnoDB: Using Linux native AIO
2023-10-12 11:50:31 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2023-10-12 11:50:31 0 [Note] InnoDB: Completed initialization of buffer pool
2023-10-12 11:50:31 0 [Note] InnoDB: 128 rollback segments are active.
2023-10-12 11:50:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-10-12 11:50:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-10-12 11:50:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-10-12 11:50:31 0 [Note] InnoDB: 10.6.12 started; log sequence number 621771438348; transaction id 22705724
2023-10-12 11:50:31 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-10-12 11:50:31 0 [Note] InnoDB: Loading buffer pool(s) from /data/databases/ib_buffer_pool
2023-10-12 11:50:31 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-10-12 11:50:31 0 [Note] Server socket created on IP: '::'.
2023-10-12 11:50:31 0 [Note] InnoDB: Buffer pool(s) load completed at 231012 11:50:31
2023-10-12 11:50:31 0 [Note] mysqld: ready for connections.
Version: '10.6.12-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
[11:50:32] INFO: MariaDB started
[11:50:32] INFO: Check data integrity and fix corruptions
mysql.column_stats                                 OK
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.global_priv                                  OK
mysql.gtid_slave_pos                               OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.index_stats                                  OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.roles_mapping                                OK
mysql.servers                                      OK
mysql.table_stats                                  OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.transaction_registry                         OK
homeassistant.event_data                           OK
homeassistant.event_types                          OK
homeassistant.events                               OK
homeassistant.recorder_runs                        OK
homeassistant.schema_changes                       OK
homeassistant.state_attributes                     OK
homeassistant.states                               OK
homeassistant.states_meta                          OK
homeassistant.statistics                           OK
homeassistant.statistics_meta                      OK
homeassistant.statistics_runs                      OK
homeassistant.statistics_short_term                OK
sys.sys_config                                     OK
[11:50:32] INFO: Ensuring internal database upgrades are performed
[11:50:32] INFO: Ensure databases exists
[11:50:32] INFO: Create database homeassistant
[11:50:32] INFO: Ensure users exists and are updated
[11:50:32] INFO: Update user homeassistant
[11:50:32] INFO: Init/Update rights
[11:50:32] INFO: Granting all privileges to homeassistant on homeassistant
[11:50:32] INFO: Sending service information to Home Assistant
s6-rc: info: service mariadb-post successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
2023-10-12 12:28:42 19 [Warning] Aborted connection 19 to db: 'homeassistant' user: 'homeassistant' host: '172.30.32.1' (Got an error reading communication packets)
2023-10-12 12:28:42 3 [Warning] Aborted connection 3 to db: 'homeassistant' user: 'homeassistant' host: '172.30.32.1' (Got an error reading communication packets)

Here is a history screenshot that shows the dead spots in the history. I’ve got one z-wave, one zigbee, and one internal entity, all different measurement types. You can see that the updates that did happen in the dead spots seem to be syncronized between the entities.

Any suggestions to further investigate this issue?

I also happen to have InfluxDB installed and it does not appear to be missing data. From this graph of the CPU usage, you can see that the dead spots in MariaDB correlate with the high CPU usage points in InfluxDB (the first one starting Sept 27 at 1PM on both graphs).

Out of curiosity, I tried uninstalling MariaDB and reinstalling it. That got rid of all my history, so I restored MariaDB only from a recent full backup, which restored my history and got new data recording again. After about a day and a half, recording errors started happening again.