Core.entity_registry prevents home-assistant from starting

giovanni · January 17, 2023, 3:54pm

Hi all,
I have just upgraded home-assistant and it won’t start unless I delete core.entity_registry before as I get the following error:

2023-01-17 15:52:33.738 ERROR (SyncWorker_3) [homeassistant.util.json] Could not parse JSON content: /home/homeassistant/.homeassistant/.storage/core.entity_registry
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.9/site-packages/homeassistant/util/json.py", line 39, in load_json
    return orjson.loads(fdesc.read())  # type: ignore[no-any-return]
  File "/usr/local/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 86216: invalid start byte

What can I do? Thanks.

Olivier1974 · January 17, 2023, 4:30pm

Did you change the .storage/core.entity_registry file?
This is not a good idea if you did but it might also come from a corruption of the SD Card.
Your file is malformed and HA can’t read it.
There is a invalid character at position 86216 of the file (a 0xb0) (I believe it is a ° but not sure)
Can you check with a file editor?
With VSCode, you can use Ctrl+K CTRL+G after installing this addon

Based on average, it should be around line 3200

giovanni · January 17, 2023, 4:37pm

I haven’t modified the file, I deleted it and it gets re-created every time. The system doesn’t use a sd card but a ssd which is healthy.

Olivier1974 · January 17, 2023, 4:41pm

Can you have a look at that character in the file? And replace it maybe? Before putting it back from the UI?

giovanni · January 17, 2023, 4:53pm

I tried using sublime text and reaching character 86216 i see a letter “a”. I higlighted it with **a**

      {
        "aliases": [],
        "area_id": null,
        "capabilities": null,
        "config_entry_id": null,
        "device_cl**a**ss": null,
        "device_id": null,
        "disabled_by": null,
        "entity_category": null,
        "entity_id": "automation.laundryroom_ventilation_timer",
        "hidden_by": null,
        "icon": null,
        "id": "787f67d30d76466bf6a80ebdde706915",
        "has_entity_name": false,
        "name": null,
        "options": {},
        "original_device_class": null,
        "original_icon": null,
        "original_name": "Laundryroom ventilation timer",
        "platform": "automation",
        "supported_features": 0,
        "translation_key": null,
        "unique_id": "cf063a3a5d984cb7a3bc85d225b0a5b1",
        "unit_of_measurement": null
      },

Olivier1974 · January 17, 2023, 4:57pm

That’s probably not the right place, as device_class is a valid tag.
That will be really annoying to debug.
Maybe an online json validator, can you copy paste your whole file there?

And press “Validate JSON”

giovanni · January 17, 2023, 5:12pm

It’s a valid JSON according to that validator.

Olivier1974 · January 17, 2023, 5:27pm

Try the provided solution on that file (.storage/core.entity_registry) and let us know.

iconv -f iso-8859-1 -t utf-8 .storage/core.entity_registry > .storage/core.entity_registry.converted

and compare the 2 files if you can (or just replace one with the other)

giovanni · January 17, 2023, 5:37pm

Thank you so much! This solved my problem

EDIT: nope, on the second restart the problem reappears

Olivier1974 · January 17, 2023, 5:46pm

Damn, you should do a comparison then, it means that something from your config is recreating the error.
It will guide you to the integration/entity that is culprit

giovanni · January 17, 2023, 5:56pm

So the question is why home-assistant is creating a version of the file with the wrong encoding?

giovanni · January 17, 2023, 6:00pm

Upon pening the file with kedit, I receive the following error:

The file you opened has some invalid characters. If you continue editing this file, you could corrupt this document.
You can also choose another character encoding and try again.

Olivier1974 · January 17, 2023, 6:04pm

The content of the file core.entity_registry and core.device_registry is coming from the configuration.yaml + all the integrations + all customizations + …

If you put an iso-8859-1 encoded character (by saving with a windows application for exemple) in one of the files used by HA to recreate the core files, when the file will be re-created, it will contain that character and the whole process will fail.

I know that because I did a modification in HA that included a french character that HA did not like. I knew which one it was and replaced it with another more “English friendly” one.

EDIT: Doing a diff between the files will pinpoint the character that HA doesn’t like in that file and will most probably help you modify another file such as at the next restart, the core file will not include it again.

giovanni · January 17, 2023, 6:16pm

I found the culprit! Look at the last line, there is a question mark instead of °.
This is that this is a zwave sensor, how can I debug the problem from now? It never created this issue in older versions of HA.

      {
        "aliases": [],
        "area_id": null,
        "capabilities": {
          "state_class": "measurement"
        },
        "config_entry_id": "645f5c225c839ffac1d4fa6d81c15fd5",
        "device_class": null,
        "device_id": "0b6af9320e8b4a1aaf8fc5a447fb4c2f",
        "disabled_by": null,
        "entity_category": null,
        "entity_id": "sensor.living_room_temperature",
        "hidden_by": null,
        "icon": null,
        "id": "9e3e59afe0ed0dd28805cdd796af3639",
        "has_entity_name": false,
        "name": null,
        "options": {},
        "original_device_class": "temperature",
        "original_icon": null,
        "original_name": "Living Room Temperature",
        "platform": "esphome",
        "supported_features": 0,
        "translation_key": null,
        "unique_id": "esp01sensorliving_room_temperature",
        "unit_of_measurement": "?C"
      },

tom_l · January 17, 2023, 6:19pm

I was about to mention that this

Is the degree symbol.

Olivier1974 · January 17, 2023, 6:19pm

Have a look at the configuration of that sensor in ESPHome.

Olivier1974 · January 17, 2023, 6:20pm

Said it in reply #1

giovanni · January 17, 2023, 6:28pm

I don’t understand where is the problem…

Olivier1974 · January 17, 2023, 6:35pm

You should have a yaml file describing your sensor in ESPHome, lsomething that look like this:

esphome:
  name: DaikinAP36230
  platform: ESP32
  board: mhetesp32devkit

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

web_server:
  port: 80
  auth:
    username: admin
    password: !secret web_server_password

logger:

api:

ota:

esp32_ble_tracker:

tom_l · January 17, 2023, 6:58pm

Also in that ESPHome file it is this specific sensor: Living Room Temperature