Solving core.entity_registry issue which dropped a chunk of my devices

Stewface · December 22, 2019, 11:03pm

Hi everyone,

I’ve run into a serious issue which seems to be caused (but not the root of) problems as soon as I activated notifications on iOS app 2.0.

All was working well with the old app, I tested the new app never got it working properly (due to lack of time to experiment and get everything going) but alas I got a new iphone and my hand was forced to get the new app (thanks apple…) as i’ve lost notifications on my phone currently and needed to get them back.

Everything was working pretty seemlessly with HASS but as soon as I add the few extra lines of code I need for notifications to work everything went up in smoke.

notify:
  - name: iOSApps
    platform: ios
    target: 'd9gIxbnZNUQ:APA91bHywcwTljAl2eKPZacgPRPHwtgMg0DZ1e2io97uqekrev0mobvSm8pOTXa0HDv0jV2A6R3glhMq79QpnQS5DtGUnwRNF38HmG80tdDiuwf6HW0D651jIa5AUP__DtMwXXXXXXXX'``

My wifes phone is setup in there exactly the same using the old ios app with notifications working perfectly (and still works even after these new lines of code break everything)

Strangely as soon as this line is added and I reboot, all my xiaomi devices drop off, 8 temp sensors, 2 gateways, 5 motion sensors, 10 door/window (says it can’t discover the gateway), that’s the main issue, but also my intergrations for broadlink RF stop and as well as some issues with my solar inverter (fronius)

More frustratingly is the fact that I can’t seem to backtrack and remove the code and get them to be discovered, soon as I do that the problem remains, (and then I lose notifications too) sometimes even after restoring a old backup over the top with the last backup with xiaomi sensors working it doesn’t solve it.

I can get them back with a fresh install (on a blank SD) and update then restoring a backup (strangely not when doing a restore over the top of it after its broken) then as soon as I add the notifty IOS line it all breaks and I need to wipe the card, download and install a fresh hass version.

Things i’ve attempted to debug (with my limited knowledge)

Slowly removing intergrations & lines of the configuration on the broken install (without success) getting back to an almost fresh install (that still doesn’t work)
Upgrading to latest version
Installing the App + notifications & Xiaomi devices on a fresh install (which does work!) but with dozens of devices and customisations it would take days to install everything piece by piece.

P.S. @robbiet480 I’m sure this has nothing to do with your coding of the incredible app you have put together and more so some sloppy integration work ive done at some point in time that has for some reason come to light now.

Anyone heard of anything like this, here is a summary of what i’ve got in hass:

bwalarm
duckdns
hass cloud
alexa
mqtt

8x sonos
broadlink rf
xiaomi:
8x temp
12x door/window
5x motion
1x curtain
iotawatt power sensor
fronius solar inverter
2x sony bravia tvs
6x hikvision cameras
1x magichome LED light (Via tasmota)
10x sonoff via tasmota lights
2x diakin climate
dozens of automations

Stewface · February 13, 2020, 12:07am

Just an update here - i’ve been living with this problem for months now, my best course of action seems to be a new SD card, fresh install then install from backup once whatever sets this off occurs.

After slowly rebuilding my whole configuration, addons, lovelace etc, I have found that the culprit lies with the core.entity_registry file - removing that seems to be the only thing that will help restore my Xiaomi devices.

Seems to be the only devices that are affected, once I got them all up and running with the new ios 2.0 app I thought I was home free, but when I added my wife’s phone I lost them again.

Not sure if it is to do with renaming the devices to friendly names. e.g.

"entity_id": "sensor.humidity_deck"

instead of

"entity_id": "sensor.humidity_158d000349bf87"

I’m at odds with what to do next I have 30+ xiaomi devices for temp, motion, door, window & curtains its the core of my home automation system!

All my log shows is that is can’t find the Xiaomi hub, but as soon as I delete my core.entity_registry I can usually get the devices back witha bit of jigging (sometimes deleting DB etc.)

2020-02-12 23:49:01 WARNING (MainThread) [homeassistant.setup] Setup of xiaomi_aqara is taking over 10 seconds.
2020-02-12 23:48:53 INFO (SyncWorker_11) [homeassistant.components.xiaomi_aqara] Discovering Xiaomi Gateways (Try 1)
~ 
2020-02-12 23:53:49 INFO (SyncWorker_11) [homeassistant.components.xiaomi_aqara] Discovering Xiaomi Gateways (Try 60)
2020-02-12 23:53:54 INFO (SyncWorker_11) [xiaomi_gateway] Gateway discovery finished in 5 seconds
2020-02-12 23:53:54 ERROR (SyncWorker_11) [homeassistant.components.xiaomi_aqara] No gateway discovered

Any help would be so very much appreciated! I spent a solid 8 hours trying to resolve last week only for it to come back

Stewface · February 13, 2020, 4:01am

Interesting update…

Looks like when I strip my core.entity_registry file right back to only the xiaomi entities (removing everything else) to get back my customisations it seems to have worked - but it’s concerning that another device can throw off all my xiaomi devices.

A few hours on from this discovery and further customisations and fixes and all of a sudden all the xiaomi devices dropped off again, so i’m replacing the core.entity_registry file again with the devices that I have customised.

Seems this time what threw it was playing around with my hikvision cameras, so anything seems to be able to set it off.

Stewface · February 13, 2020, 11:06pm

Further work trying to restore my xiaomi devices and it seems that deleting the core.entity_registry file didn’t work this time.

So I ended up having to go back to an existing backup and restore + wipe along with removing that core.entity_registry file.

ARGH!!! This is driving me nuts!

How many people have migrated away from xiaomi hubs towards https://www.zigbee2mqtt.io/ Maybe that is my only option to rectify this problem?

What is the formal way to add a bug report and what is the best way of trying to show more logs / console to figure out what is causing this (It always just seems to happen on a reboot though) and you can see the log above of what is shown to me just that it can’t detect gateway.

Stewface · February 23, 2020, 11:05pm

Looks like the problem has re-appeared.

I’m really lost for what to do next beside having to do a wipe and restore every 2-3 weeks

sjee · February 24, 2020, 8:50am

Tried to understand your issue but I failed Let’s start by which version of HA you’re running. If I understand correctly you are trying to use both the old and new version of the IOS app, right? Did you read the docs on the companion app, the IOS integration is setup in the GUI, no need to add anything manually.

Stewface · February 24, 2020, 11:47pm

Thanks @sjee for chiming in.

It’s hard for me to understand too!!

Home Assistant 0.105.5 HassOS 3.11 Supervisor 201

Not a conflict with the new vs old app, seems that the iOS app (or all the entities it created) was the initial trigger for knocking all my Xiaomi devices offline. But now it just happens randomly every 2-3 weeks even without editing / changing any config or adding new devices. I’ve now got both my phone and my wifes on the new iOS app.

I suspect its entity related as that has been a method of fixing it, (firstly I tried restoring the current install from backup wipe + restore but that failed) but this most recent time what I needed to do to get them back was:

Load up a fresh blank SD card
Install latest version + add Samba
Get a dropbox backup from 2-3 days ago and restore backup
(Still didn’t work at this point! When it usually is resolved)
Update to the latest version, leave it running
12 hrs later when I checked back the devices had come back online.

I really need a long term solution for this that doesn’t involve losing a chunk of my devices (and more importantly the wife approval factor to keep building my HA system).

Can I turn on any specific logging that might pickup what is the trigger for this to happen?

sjee · February 26, 2020, 9:22am

So basically the “only” issue you’re having is the Xiaomi gateway becomes unavailable randomly. There are many topics on this.