500 Internal Server Error

Tockers · March 2, 2019, 5:27am

Running HA inside docker on a pi3 raspbian. Everything has been running great for months.

My pi did a hard power down due to an outage, everything seemed ok after a reboot, etc. Before I restarted, I stopped HA and removed the DB file which was getting huge before restarting (done many times without issue). Worked the first time however I noticed in lovelave notifications that logbook and hostory had not loaded. I also have an energy monitor (OEM emonPi) which was, and still is, working and publishing to web just fine. I rebooted and now I am getting:

Server got itself in trouble

I have tried an update/reinstall to 88.2 from 88.1 just in case, but still no change. Anyone able to decipher the logs?

Logs from HA:

2019-03-03 09:17:30 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/real_ip.py", line 34, in real_ip_middleware
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/ban.py", line 67, in ban_middleware
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/auth.py", line 91, in auth_middleware
    users = await app['hass'].auth.async_get_users()
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/__init__.py", line 110, in async_get_users
    return await self._store.async_get_users()
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/auth_store.py", line 61, in async_get_users
    await self._async_load()
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/auth_store.py", line 307, in _async_load
    for group_dict in data.get('groups', []):
AttributeError: 'list' object has no attribute 'get'

Tockers · March 2, 2019, 10:11am

Seems like it all falls out of:

2019-03-02 17:15:10 ERROR (MainThread) [aiohttp.server] Error handling request

Tockers · March 2, 2019, 10:48pm

Have removed all HTTP config (trusted networks, API pass, base URL) and cleared caches on browser and now being presented with login screen for local instance. Enter local instance user/pass and page just sits with nothing happening. If I enter incorrect credentials, it rejects with bad user/pass as expected.

Logs still reporting issues with aiohttp:

2019-03-03 09:45:20 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/real_ip.py", line 34, in real_ip_middleware
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/ban.py", line 67, in ban_middleware
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/auth.py", line 99, in auth_middleware
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/view.py", line 118, in handle
    result = await result
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/data_validator.py", line 47, in wrapper
    result = await method(view, request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/http/ban.py", line 77, in handle_req
    resp = await func(view, request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/auth/login_flow.py", line 220, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/data_entry_flow.py", line 80, in async_configure
    flow, step_id, user_input)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/data_entry_flow.py", line 110, in _async_handle_step
    result = await self._async_finish_flow(flow, dict(result))
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/__init__.py", line 421, in _async_finish_login_flow
    result['data'])
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/providers/homeassistant.py", line 241, in async_get_or_create_credentials
    for credential in await self.async_credentials():
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/providers/__init__.py", line 71, in async_credentials
    users = await self.store.async_get_users()
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/auth_store.py", line 61, in async_get_users
    await self._async_load()
  File "/usr/local/lib/python3.7/site-packages/homeassistant/auth/auth_store.py", line 307, in _async_load
    for group_dict in data.get('groups', []):
AttributeError: 'list' object has no attribute 'get'

Tockers · March 3, 2019, 12:09am

After reading up on some auth reset steps (remove auth, etc. from /.storage) I found a corrupted file. Unable to remove despite attempts to umount and fsck so renamed the directory, recreate /.storage and moved the files back over. The corrupted file is sitting on its own in a renamed directory for now… no idea what to do with it but HA back online.

pi@emonpi:~/data/homeassistant/.storage $ ls -al
ls: cannot access 'auth': Structure needs cleaning
total 92
drwxr-xr-x 2 root root  1024 Mar  3 09:57 .
drwxr-xr-x 6 root root  1024 Mar  3 09:57 ..
-????????? ? ?    ?        ?            ? auth
-rw------- 1 root root   290 Dec 17 19:05 auth_provider.homeassistant
-rw-r--r-- 1 root root   188 Feb 16 12:44 cloud
-rw-r--r-- 1 root root   844 Feb 12 19:50 core.config_entries
-rw-r--r-- 1 root root  1342 Dec 17 20:08 core.device_registry
-rw-r--r-- 1 root root  5276 Jan 31 17:48 core.entity_registry
-rw-r--r-- 1 root root 46178 Mar  3 09:57 core.restore_state
-rw-r--r-- 1 root root 29765 Mar  2 13:56 lovelace
-rw------- 1 root root   113 Dec 17 19:05 onboarding

anon34565116 · March 3, 2019, 12:15am

Try running fsck on the Linux filesystem?

Tockers · March 3, 2019, 1:12am

Tried but reporting busy despite stopping services.

pi@emonpi:/var $ sudo fsck /home/pi/data
fsck from util-linux 2.29.2
e2fsck 1.43.4 (31-Jan-2017)
/dev/mmcblk0p3 is in use.
e2fsck: Cannot continue, aborting.

anon34565116 · March 3, 2019, 1:11pm

OK then it gets more complicated. If you type the “mount” command does “/home” show up as a separate entry? If so you may be ale to stop HomeAssistant and unmount /home to fsck it.
The Linux OS has that filesystem busy. It is pronanly best to just leave is. If you rename it to something beginning with “.” like “.deleteme” it will be hidden from some commands.