Yellow: Corrupt file system

Viking · October 11, 2024, 6:22pm

Home Assistant 2024.10.1 running on a Yellow with 1TB NVMe SSD

Today I cannot access any HA Add-On’s. All I get is a white, blank page.
If I try to check the configuration I just get file configuration.yaml not found
The System -->> Backups also just gives a white, blank page so restore is also impossible.
HA still runs, and all devices and entities seems alive and “well” but the log is flooded with file system errors. Here’s one:

Logger: homeassistant.components.recorder.core
Source: components/recorder/core.py:891
integration: Recorder (documentation, issues)
First occurred: 09:06:08 (1 occurrences)
Last logged: 09:06:08

Unrecoverable sqlite3 database corruption detected: (sqlite3.DatabaseError) database disk image is malformed [SQL: SELECT states_meta.metadata_id, states_meta.entity_id FROM states_meta WHERE states_meta.entity_id IN (?)] [parameters: ('sensor.ikea_inspelning_smart_plug2_power',)] (Background on this error at: https://sqlalche.me/e/20/4xp6)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
sqlite3.DatabaseError: database disk image is malformed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 891, in _process_one_task_or_event_or_recover
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 1029, in _process_one_event
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 1124, in _process_state_changed_event_into_session
  File "/usr/src/homeassistant/homeassistant/components/recorder/table_managers/states_meta.py", line 63, in get
  File "/usr/src/homeassistant/homeassistant/components/recorder/table_managers/states_meta.py", line 113, in get_many
  File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 195, in execute_stmt_lambda_element
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2236, in _execute_internal
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/sql/lambdas.py", line 603, in _execute_on_connection
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2353, in _handle_dbapi_exception
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed
[SQL: SELECT states_meta.metadata_id, states_meta.entity_id 
FROM states_meta 
WHERE states_meta.entity_id IN (?)]
[parameters: ('sensor.ikea_inspelning_smart_plug2_power',)]
(Background on this error at: https://sqlalche.me/e/20/4xp6)

All attempts ro restart, reboot, power off fails.
All I can do seems to be to power cycle the Yellow box, but I’m afraid to do that as I feel pretty sure it will never boot again.
Can I ask for some advice before progressing?

tom_l · October 11, 2024, 11:28pm

SSH into your Yellow (or use a SAMBA share) and delete config/home-assistant_v2.db then restart Home Assistant (not host hardware reboot).

You will lose all your history but all your config will be left unchanged.

If you have a backup from yesterday when your Yellow was working you could replace that file with the one from your backup and you would only lose a day’s history.

Otherwise you could search the forum for “database repair” and try that from the SSH command line.

Viking · October 12, 2024, 7:21am

Thanks for your reply, but how do I SSH into my Yellow?

renespit · October 12, 2024, 7:31am

If you working on a windows machine then you have to use a program like: putty.exe

If you working on a Mac then cmd+space. Search for terminal.

After that you have to type: ssh [email protected]
It will ask for your password.

And then you are in your Home Assistant OS.

tom_l · October 12, 2024, 7:49am

Assuming you set it up in the first place.

If you did not then you will need to connect a monitor and keyboard to your Yellow.

nickrout · October 12, 2024, 8:15am

First job after install is advanced ssh & terminal addon

Viking · October 12, 2024, 10:27am

Thanks everyone for all your support.
First of all - i have the SSH&Terminal addon installed, but it is no longer accessible from the WEB UI.
Putty does connect, but when I type my HA username, it immediately disconnects.
Isn’t my HA username supposed to be valid for logon?

petro · October 12, 2024, 2:02pm

Just plug in a keyboard and monitor.

Viking · October 12, 2024, 6:15pm

If I only could, but my Yellow is in Spain and I’m in Norway

nickrout · October 12, 2024, 6:16pm

When you installed ssh & Terminal addon you set up a separate username and password. Use it.

Viking · October 12, 2024, 6:21pm

I’m not sure I did (unless the installation procedure required it as mandatory), besides I cannot remember having done so, nor can I remember any username/password combination I might have chosen

nickrout · October 12, 2024, 6:34pm

Tuck that monitor and keyboard under your arm and get walking then

Viking · October 16, 2024, 7:54am

And where do i plug in a monitor on the Yellow?

nickrout · October 17, 2024, 8:38am

That is a good point. Best you remember your ssh u/pw.

petro · October 17, 2024, 3:24pm

does it not have a monitor connection?

I see a usb c connection on the side, can’t be used for a monitor?

EDIT: Confirmed, not usable.

Viking · October 21, 2024, 6:16pm

Well, as days goes by, the problems (and the errors reported in the log) just escalates, and finally I decided to remotely power cycle my Yellow. There was little else to do, and guess what?! It came back on line and all was well again. It’s now been almost a week with no further problems! (Knock-on-wood)
Both colleagues and friends were amazed that the Yellow managed to get back on its feet again after a completely corrupted file system as well as a corrupt DB.

I have no Idea what caused my problem in the first place, nor how bad the damage truly was (but it appeared completely damaged). So I must say that I’m either incredibly lucky or totally impressed (or both).