Problem with updating from MariaDB Addon 2.5.2

Ethric · June 1, 2026, 6:29am

as my HA installation "just worked" in the past threeish years, I didn't take much (or any) time for updates. So, I am stuck now with a two year old problem related to the MariaDB addon.

I am coming from HAOS 10.1 and HA Core 2023.something, which I already upgraded to 17.3 and 2026.5.4.

I do fail, however, to upgrade the MariaDB addon from 2.5.2, which seems to be related to this problem of the DB not properly shutting down. As soon as I upgrade (in my case to 3.0.1 which is offered by the addons/apps UI), I get repeatedly the following log output:

[07:50:50] INFO: Starting MariaDB
2026-06-01  7:50:50 0 [Note] Starting MariaDB 11.4.10-MariaDB source revision 054a893f1645b77e52a329a7fc8cf614eebd1fad server_uid Vax+WMZfLOTEqhu0bqDBLuI8SRI= as process 279
2026-06-01  7:50:50 0 [Note] InnoDB: Compressed tables use zlib 1.3.2
2026-06-01  7:50:50 0 [Note] InnoDB: Number of transaction pools: 1
2026-06-01  7:50:50 0 [Note] InnoDB: Using generic crc32 instructions
2026-06-01  7:50:50 0 [Note] InnoDB: Using Linux native AIO
2026-06-01  7:50:50 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m
2026-06-01  7:50:50 0 [Note] InnoDB: Completed initialization of buffer pool
2026-06-01  7:50:50 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2026-06-01  7:50:50 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.8. You must start up and shut down MariaDB 10.7 or earlier.
2026-06-01  7:50:50 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2026-06-01  7:50:50 0 [Note] InnoDB: Starting shutdown...
2026-06-01  7:50:50 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2026-06-01  7:50:50 0 [Note] Plugin 'FEEDBACK' is disabled.
2026-06-01  7:50:50 0 [Note] Plugin 'wsrep-provider' is disabled.
2026-06-01  7:50:50 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2026-06-01  7:50:50 0 [ERROR] Aborting
[07:50:50] INFO: Service mariadb exited with code 1 (by signal 0)

All my attempts to manually gracefully shut down the 2.5.2 addon_core_mariadb docker container failed so far.

Executing docker stop -t 600 addon_core_mariadb (after disabling recorder and watchdog for restart) takes the full 10 minutes and after that docker inspect addon_core_mariadb shows ExitCode 137, which is "forcefully killed with SIGKILL".

I can't find any other logs showing me what might have gone wrong. docker logs addon_core_mariadb just ends with the latest startup results:

[...]
2026-06-01  7:26:19 0 [Note] /usr/bin/mariadbd: ready for connections.
Version: '10.6.10-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
[07:26:20] INFO: Check data integrity and fix corruptions
mysql.column_stats                                 OK
[...]
sys.sys_config                                     OK
[07:26:20] INFO: Ensuring internal database upgrades are performed
[07:26:21] INFO: Ensure databases exists
[07:26:21] INFO: Create database homeassistant
[07:26:21] INFO: Ensure users exists and are updated
[07:26:21] INFO: Update user homeassistant
[07:26:21] INFO: Init/Update rights
[07:26:21] INFO: Granting all privileges to homeassistant on homeassistant
[07:26:22] INFO: Successfully send service information to Home Assistant.

I also tried to jump into the container with docker exec -it addon_core_mariadb sh to shut down mariadb properly, but I not really confident what to do there. Just killing the /usr/bin/mariadbd process results in s6 to just spawn a new one:

2026-06-01  7:57:31 0 [Note] /usr/bin/mariadbd (initiated by: unknown): Normal shutdown
2026-06-01  7:57:31 0 [Note] InnoDB: FTS optimize thread exiting.
2026-06-01  7:57:31 0 [Note] InnoDB: Starting shutdown...
2026-06-01  7:57:31 0 [Note] InnoDB: Dumping buffer pool(s) to /data/databases/ib_buffer_pool
2026-06-01  7:57:31 0 [Note] InnoDB: Restricted to 2028 pages due to innodb_buf_pool_dump_pct=25
2026-06-01  7:57:31 0 [Note] InnoDB: Buffer pool(s) dump completed at 260601  7:57:31
2026-06-01  7:57:31 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2026-06-01  7:57:31 0 [Note] InnoDB: Shutdown completed; log sequence number 196819667167; transaction id 44695680
2026-06-01  7:57:31 0 [Note] /usr/bin/mariadbd: Shutdown complete
260601 07:57:31 mysqld_safe mysqld from pid file /data/databases/core-mariadb.pid ended
[05:57:31] INFO: Service restart after closing
[07:57:31] INFO: Using existing mariadb initial system
[07:57:31] INFO: Starting MariaDB
[...]

When I s6-svc -d /run/s6/legacy-services/mariadb and s6-svc -d /run/s6/legacy-services/mariadb-lock it remains gone:

2026-06-01  7:59:19 0 [Note] /usr/bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
2026-06-01  7:59:19 0 [Note] InnoDB: FTS optimize thread exiting.
2026-06-01  7:59:19 0 [Note] InnoDB: Starting shutdown...
2026-06-01  7:59:19 0 [Note] InnoDB: Dumping buffer pool(s) to /data/databases/ib_buffer_pool
2026-06-01  7:59:19 0 [Note] InnoDB: Restricted to 2028 pages due to innodb_buf_pool_dump_pct=25
2026-06-01  7:59:19 0 [Note] InnoDB: Buffer pool(s) dump completed at 260601  7:59:19
2026-06-01  7:59:20 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2026-06-01  7:59:20 0 [Note] InnoDB: Shutdown completed; log sequence number 196819985010; transaction id 44695739
2026-06-01  7:59:20 0 [Note] /usr/bin/mariadbd: Shutdown complete
260601 07:59:20 mysqld_safe mysqld from pid file /data/databases/core-mariadb.pid ended
[05:59:20] INFO: Service restart after closing

(there is no restart after the final INFO output)

However, stopping the container still results in SIGKILL after the 10 minutes timeout and the "Upgrade after a crash is not supported" problem after upgrading the addon to 3.0.1 remains, even though it seemed that the database properly shut down, but not the container.

I am running out of ideas, so I am trying my luck here. Is there anything more I can try?

Thanks in advance
Stefan

checking12 · June 1, 2026, 6:52am

Afaik (not seen it myself), it could be because of the redo log file. If you shutdown mariadb and it states 'shutdown complete'. Delete the redo file eg. ib_logfile0. Then upgrade will not be halted by that 'crashed' message.

Ethric · June 1, 2026, 7:33am

@checking12 Thanks for the quick reply. I did experiment with manually deleting the redo log file yesterday already, but gave it another try, as I don't recall what exactly I was doing.

Unfortunately, the result is the same as I remember from yesterday, i.e., the DB complaining that the file is not there:

[09:28:04] INFO: Starting MariaDB
2026-06-01  9:28:04 0 [Note] Starting MariaDB 11.4.10-MariaDB source revision 054a893f1645b77e52a329a7fc8cf614eebd1fad server_uid QR/h4Y/doJkB89aIfwhLogNyfVI= as process 564
2026-06-01  9:28:04 0 [Note] InnoDB: Compressed tables use zlib 1.3.2
2026-06-01  9:28:04 0 [Note] InnoDB: Number of transaction pools: 1
2026-06-01  9:28:04 0 [Note] InnoDB: Using generic crc32 instructions
2026-06-01  9:28:04 0 [Note] InnoDB: Using Linux native AIO
2026-06-01  9:28:04 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m
2026-06-01  9:28:04 0 [Note] InnoDB: Completed initialization of buffer pool
2026-06-01  9:28:04 0 [ERROR] InnoDB: File ./ib_logfile0 was not found
2026-06-01  9:28:04 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2026-06-01  9:28:04 0 [Note] InnoDB: Starting shutdown...
2026-06-01  9:28:04 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2026-06-01  9:28:04 0 [Note] Plugin 'FEEDBACK' is disabled.
2026-06-01  9:28:04 0 [Note] Plugin 'wsrep-provider' is disabled.
2026-06-01  9:28:04 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2026-06-01  9:28:04 0 [ERROR] Aborting
[09:28:04] INFO: Service mariadb exited with code 1 (by signal 0)

Here's, for the record, what I did:

run action recorder.disable in Developer Tools
disable watch dog in addon settings
docker exec into container
s6-svc -d /run/s6/legacy-services/mariadb and s6-svc -d /run/s6/legacy-services/mariadb-lock
Check addon logs in HA UI for /usr/bin/mariadbd: Shutdown complete
rm /data/databases/ib_logfile0
exit docker exec
trigger MariaDB addon upgrade to 3.0.1 via UI

checking12 · June 1, 2026, 9:01am

I see, this is interesting, it's supposed to create a new one in the new format.
Btw, i've also seen some force shutdown option flag innodb_fast_shutdown, you have that set to 0?

Ethric · June 1, 2026, 10:11am

I have not changed it, so it's probably at its default value (which is 1?). How would I usually set this value? I tried adding it to /etc/my.cnf.d/mariadb-server.cnf, but this resulted in the same not working behavior.

I also found innodb_force_recovery, which is supposed to just skip recovery and have (potential) data loss. So, I removed the redo log, updated to 3.0.1 and added it to /etc/my.cnf.d/mariadb-server.cnf. I tried all values from 1-6 (with 6 being "skip completely"). 1-5 resulted in the original error message ("Upgrade after a crash is not supported."), but for 6:

[11:56:38] INFO: Starting MariaDB
2026-06-01 11:56:38 0 [Note] Starting MariaDB 11.4.10-MariaDB source revision 054a893f1645b77e52a329a7fc8cf614eebd1fad server_uid q9ipnkTsdLit8vybpOUtW+zAxXA= as process 10971
2026-06-01 11:56:38 0 [Note] InnoDB: !!! innodb_force_recovery is set to 6 !!!
2026-06-01 11:56:38 0 [Note] InnoDB: Started in read only mode
2026-06-01 11:56:38 0 [Note] InnoDB: Compressed tables use zlib 1.3.2
2026-06-01 11:56:39 0 [Note] InnoDB: Number of transaction pools: 1
2026-06-01 11:56:39 0 [Note] InnoDB: Using generic crc32 instructions
2026-06-01 11:56:39 0 [Note] InnoDB: Using Linux native AIO
2026-06-01 11:56:39 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m
2026-06-01 11:56:39 0 [Note] InnoDB: Completed initialization of buffer pool
2026-06-01 11:56:39 0 [Note] InnoDB: innodb_force_recovery=6 skips redo log apply
2026-06-01 11:56:39 0 [Note] InnoDB: innodb_force_recovery=6 skips redo log apply
2026-06-01 11:56:39 0 [ERROR] InnoDB: innodb_read_only=ON prevents an upgrade of the change buffer
2026-06-01 11:56:39 0 [ERROR] InnoDB: Plugin initialization aborted with error Read only transaction
2026-06-01 11:56:39 0 [Note] InnoDB: Starting shutdown...
2026-06-01 11:56:39 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2026-06-01 11:56:39 0 [Note] Plugin 'FEEDBACK' is disabled.
2026-06-01 11:56:39 0 [Note] Plugin 'wsrep-provider' is disabled.
2026-06-01 11:56:39 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2026-06-01 11:56:39 0 [ERROR] Aborting
[11:56:39] INFO: Service mariadb exited with code 1 (by signal 0)

It seems, that innodb_read_only can not be set in config, only as startup parameter. How can I switch it off for the addon? (and why is it on in the first place???)

I also found a quite recent bug report and pull request regarding innodb_force_recovery=6 and innodb_read_only=ON, which I did not fully understand (I am already way "deeper in" than I hoped for). Maybe it's not on, but the problem is something else?

checking12 · June 3, 2026, 5:59am

Thinking about it a bit more, I would suggest to make smaller version steps instead of one go to latest. That would possibly take a technical path that circumvent the blocking issues here

And btw shutdown param is with
SET GLOBAL innodb_fast_shutdown = 0; SHUTDOWN;

Ethric · June 4, 2026, 6:33am

@checking12 Using fast_shutdown 0 actually did the trick! Thank you very much for your support!!

For the record, here's how I did it:

get into docker with docker exec -it addon_core_mariadb bash
connect to db with mariadb
SET GLOBAL innodb_fast_shutdown = 0; exit; (not using SHUTDOWN;, as s6 will immediately restart mariadb)
s6-svc -d /run/s6/legacy-services/mariadb
s6-svc -d /run/s6/legacy-services/mariadb-lock
exit docker container
use UI to update App to 3.0.1

The redo log file needed to stay (otherwise, the new version again complained about it not being there), but it seems with fast_shutdown 0 there was no redo data in it, so the new mariadb version accepted it.

By the way, I originally thought about doing smaller update steps as well, but did not find a way to do this. Using the UI or CLI always installs "latest" by default. Is there a proposed way?

checking12 · June 4, 2026, 10:53am

With cli you should always have option to specify a version. Just like ha updates
ha core update --version 202x.x.x

Should also work for addonds

Ethric · June 4, 2026, 4:06pm

Unfortunately, there seems to be no such option. I'll look into this again later; and maybe open a dedicated thread, as the system already complains that I still answer here even though it's already marked as solved.

Thanks again for your help!