2021.4: For our advanced users ❤️

mirekmal · April 10, 2021, 9:42am

It seems to me that in case of DB migration being required process should look like:

First HA should try to estimate how long it could take to migrate DB (not sure how, but perhaps performing few sample record changes on temporary created ‘test’ tables?) and evaluating number of records in DB. This should give at least some order of magnitude view on how long the process might take for particular setup.
Then we should be given options:
a) Migrate to new DB, discarding all of historical data, but process would be instant.
b) Migrate with DB being converted in background, X hours of data will be lost during conversion. HA starts instantly and no service disruption. Data starts to recorded once DB update is completed.
c) Perform full migration, no data lost but HA not available for X hours.

Strangely difference between option b) and c) is that while we anyhow loose X hours of data (either data not being recorded during conversion or HA not working at all), but we have or not HA providing us with some core services… so my preference anyhow would be to start HA and agree to some data loss
Another question to DB gurus… would below process work?

at start of conversion HA renames old tables to not use them for data recording.
creates new tables, with new structure in the same DB instance, with old names
HA starts recording new info instantly to new tables
background process reads data from old tables, converts as required and writes to new tables
old tables are deleted after completion

This way we have no data loss and HA services available during conversion. I’m not sure about consistency of data for some queries, if only recent data is available, but historical might be incomplete.

And another side remark. during my ~2 years journey with HA it is only second time that I recall when DB structure was updated. First time it was ~30 minutes for me (but then my setup was way simpler). So with this frequency of DB changes I’m not really sure if this situation shouldn’t be just accpted. If it would be repeated every HA update, this would be another story.
It would obviously help to change wording from ‘could take few minutes’ to ‘could take several hours’, just to set proper expectations At least this could easy stress related to not knowing what is going on… if this is still conversion or something failed…

DavidFW1960 · April 10, 2021, 9:44am

Seriously? I can remember more than that in the last few months… it gets migrated a fair bit.

Mariusthvdb · April 10, 2021, 9:45am

heck yes, it was twice already during the last beta

mirekmal · April 10, 2021, 9:49am

Then it was even more transparrent! Seriously I recall only case from the past, when DB schemas were changed and migration was causing some delay in startup. If there were any more it was fully transparrent to me. Migration to 2021.4 was the first case that really scared me…

Well, I’m not participating in beta! This might explain why I did not noticed any more changes!

DavidFW1960 · April 10, 2021, 9:51am

No there have been more, not just in beta either.

ludeeus · April 13, 2021, 11:32am

32 posts were split to a new topic: MQTT 5.11 issue

kajmaj · April 10, 2021, 9:53am

I would say, at least in my case - after looong time (almost 24 hrs) - it seems that conversion was performed (in db apparently startded to appear new writes), SAMBA works - can connect to config, addons, backup etc., MQTT (Mosquitto) works - ongoing traffic visible in mqtt explorer, but there is no connection to HA. Anybody else met the same?

Maco65 · April 10, 2021, 10:22am

This feature is GENIUS !!!

Thank you so MUCH @emontnemery @thomasloven @balloob @bramkragten !

djelo · April 10, 2021, 10:54am

yeah, I’m having similar issue…
I used cli to upgrae to 4.2…but still have this:

guixop · April 10, 2021, 11:16am

Hello,
after the update everything works correctly but the xiaomi_miio integration is problematic, the logs are full of errors

Logger: homeassistant.components.xiaomi_miio
Source: components/xiaomi_miio/__init__.py:98
Integration: Xiaomi Miio (documentation, issues)
First occurred: 11:39:53 (1105 occurrences)
Last logged: 13:11:54

Got exception while fetching the state: Got an exception while sending command get_bright

2021-04-10 13:12:37 ERROR (SyncWorker_61) [homeassistant.components.xiaomi_miio] Got exception while fetching the state: Got an exception while sending command get_bright
2021-04-10 13:12:44 ERROR (SyncWorker_63) [homeassistant.components.xiaomi_miio] Got exception while fetching the state: Got an exception while sending command get_bright
2021-04-10 13:12:47 ERROR (SyncWorker_31) [homeassistant.components.xiaomi_miio] Got exception while fetching the state: Got an exception while sending command get_bright
2021-04-10 13:12:54 ERROR (SyncWorker_42) [homeassistant.components.xiaomi_miio] Got exception while fetching the state: Got an exception while sending command get_bright
2021-04-10 13:12:57 ERROR (SyncWorker_62) [homeassistant.components.xiaomi_miio] Got exception while fetching the state: Got an exception while sending command get_bright
2021-04-10 13:13:04 ERROR (SyncWorker_16) [homeassistant.components.xiaomi_miio] Got exception while fetching the state: Got an exception while sending command get_bright

The mistakes don’t stop …

nutrabaits · April 10, 2021, 11:34am

After upgrade 5h Db not working .every restart is long 3-4 hours.rpi3 + ssd.going back on old version.

Rhoxus · April 10, 2021, 11:52am

Anybody else experiencing issues with the Automations tab not being visible or automations not running since the last update?

Looking into the logs I see this:


Logger: homeassistant.setup
Source: components/device_automation/__init__.py:86 
First occurred: 13:41:37 (1 occurrences) 
Last logged: 13:41:37

Error during setup of component automation
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/device_automation/__init__.py", line 83, in async_get_device_automation_platform
    integration = await async_get_integration_with_requirements(hass, domain)
  File "/usr/src/homeassistant/homeassistant/requirements.py", line 52, in async_get_integration_with_requirements
    integration = await async_get_integration(hass, domain)
  File "/usr/src/homeassistant/homeassistant/loader.py", line 553, in async_get_integration
    raise IntegrationNotFound(domain)
homeassistant.loader.IntegrationNotFound: Integration '' not found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 240, in _async_setup_component
    result = await task
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 219, in async_setup
    if not await _async_process_config(hass, config, component):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 623, in _async_process_config
    await async_validate_config_item(hass, raw_config),
  File "/usr/src/homeassistant/homeassistant/components/automation/config.py", line 81, in async_validate_config_item
    config[CONF_ACTION] = await script.async_validate_actions_config(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 232, in async_validate_actions_config
    return await asyncio.gather(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 276, in async_validate_action_config
    choose_conf[CONF_SEQUENCE] = await async_validate_actions_config(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 232, in async_validate_actions_config
    return await asyncio.gather(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 247, in async_validate_action_config
    platform = await device_automation.async_get_device_automation_platform(
  File "/usr/src/homeassistant/homeassistant/components/device_automation/__init__.py", line 86, in async_get_device_automation_platform
    raise InvalidDeviceAutomationConfig(
homeassistant.components.device_automation.exceptions.InvalidDeviceAutomationConfig: Integration '' not found

Dave_Bryant · April 10, 2021, 11:56am

Re: Error loading /config/configuration.yaml: Secrets not supported in this YAML file

I have used the Check Configuration (against 2014.4.3) and get the same result. I’m not using lovelave_gen or dwains dashboard. I have checked all the YAML files in /Config for a reference to Secret. Could the problem be with existing addon integrations?

makai · April 10, 2021, 12:00pm

I’ve opened an issue. You can add logs if you want to help.

Pirol62 · April 10, 2021, 12:04pm

The log shows, that an integration wasn’t found.
Did you check all your integrations. Are they all available?

Pirol62 · April 10, 2021, 12:15pm

Maybe it should be mentioned, that with 2021.3.4 I manually have to reload MET.no after every HA restart. So the issue might be older. Running HA OS on an raspb3+

Mariusthvdb · April 10, 2021, 12:35pm

almost slipped by, but I see we now can filter log messages https://www.home-assistant.io/integrations/logger/#log-filters

this might be the only option on the terribly annoying NMBS integration errors that are not to be avoided (issue in Github has been there for ages, and wont get resolved any time soon I fear)

this is in the logs, in many variants daily, mostly during the night, but also randomly at daytime

2021-04-10 09:27:10 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.brussel_noord_essen fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 292, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 498, in async_device_update
    await task
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/nmbs/sensor.py", line 272, in update
    if connections is None or not connections["connection"]:
KeyError: 'connection'
2021-04-10 09:34:40 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.essen_brussel_noord fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 292, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 498, in async_device_update
    await task
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/nmbs/sensor.py", line 272, in update
    if connections is None or not connections["connection"]:
KeyError: 'connection'

How could I filter that given the fact that the main Error is on homeassistant.helpers.entity but the significant bit is at the end:

File "/usr/src/homeassistant/homeassistant/components/nmbs/sensor.py", line 272, in update
    if connections is None or not connections["connection"]:
KeyError: 'connection'

and the KeyError is not always identical ? eg

  File "/usr/src/homeassistant/homeassistant/components/nmbs/sensor.py", line 155, in update
    if liveboard is None or not liveboard["departures"]:
KeyError: 'departures'

thanks for having a look, hope this can be done.

lolongan · April 10, 2021, 1:01pm

Same here. I have upgraded yesterday to 2021.4.2 using Supervisor. Everything seemed to be OK. No update pending message.
Then this morning, I get this:

Should I click on “Update” ?
I have this only on one of my HA. The other HA still shows that the Newest Version is 2021.4.2 (while indeed 2021.4.3 has been released) ???

GaryK · April 10, 2021, 1:05pm

I had to replace all secrets in my configuration files with their actual values before I could pass a config check to 2021.4.x.

makai · April 10, 2021, 1:13pm

No do not downgrade, not needed. Just wait for this bug to be solved