HA stops working after some hours

PatpaC · March 2, 2024, 10:40pm

Hello
I hope someone can help me with that, because I am at the end of my knowledge.

First up:
My Home Assistant Instance runs on a Raspberry Pi4 4GB via USB to Sata SSD.
It runs now since 1 1/2 years.

So here is my problem:

The Observer URL is down.
All Home Assistant addons are not available anymore. Z2M, uptime-kuma, file editor etc…
Some parts of settings and other parts of the UI are not available anymore. - shows an error
Some integrations stop working completely.
I cant reboot the SYSTEM anymore the button to do that is no longer shown.
When i hook the raspberry hdmi to a monitor it is not sending a video signal anymore.

Here is what I tried to solve it:

Cut the power and start HA again. – this works again for a couple of hours
Reboot every day at 4 am – was not rly helping
Looked into the Logs from HA Core and Supervisor for last boot. – could not find anything that allways shows up when the error appears
Repaired the supervisor “ha supervisor repair” – nothing changed
looked into the journalctl – cannot find anything suspicious
Disabled the bluetooth adapter via bluetoothctl → power of – nothing changed

My main problem is that I do not even have a single reference point and I do not know where to look anymore.

I hope someone can help me…
Thanks in advance!!

boheme61 · March 2, 2024, 11:01pm

you could try to run “ha os update , ha core update, ha supervisor update” , and show “the error”
And make sure your power-supply is sufficient for the peripherals such as the SSD

What about prior to the “errors”, And is there ANY Errors in the logs ( which you don’t find relevant to show/tell ) ?

stevemann · March 2, 2024, 11:25pm

I noticed that you haven’t tried the most common reason for your problem. Get a bigger PSU.

PatpaC · March 3, 2024, 2:47pm

Everything is up to date.

I also thought it could be the PSU but it worked now for 1 1/2 years without any issues. The CPU Usage, RAM Usage did not increase by any means.
Here is my CPU Usage and temp while it stops working.

The Errors I get are usually allways the same.
This is what happend right befor it became unresponsive.

2024-03-03 03:30:08.352 ERROR (MainThread) [homeassistant.util.logging] Exception in state_received when handling msg on 'awox-ble-mesh-hub/group-32768/state': 'ON'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 43, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 483, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/light/schema_json.py", line 346, in state_received
    values = json_loads_object(msg.payload)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 61, in json_loads_object
    value: JsonValueType = json_loads(__obj)
                           ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 47, in json_loads
    return orjson.loads(__obj)  # type:ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

2024-03-03 03:35:07.374 ERROR (MainThread) [homeassistant.util.logging] Exception in state_received when handling msg on 'awox-ble-mesh-hub/group-32768/state': 'ON'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 43, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 483, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/light/schema_json.py", line 346, in state_received
    values = json_loads_object(msg.payload)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 61, in json_loads_object
    value: JsonValueType = json_loads(__obj)
                           ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 47, in json_loads
    return orjson.loads(__obj)  # type:ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

2024-03-03 03:40:07.699 ERROR (MainThread) [homeassistant.util.logging] Exception in state_received when handling msg on 'awox-ble-mesh-hub/group-32768/state': 'ON'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 43, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 483, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/light/schema_json.py", line 346, in state_received
    values = json_loads_object(msg.payload)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 61, in json_loads_object
    value: JsonValueType = json_loads(__obj)
                           ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 47, in json_loads
    return orjson.loads(__obj)  # type:ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

2024-03-03 03:45:07.533 ERROR (MainThread) [homeassistant.util.logging] Exception in state_received when handling msg on 'awox-ble-mesh-hub/group-32768/state': 'ON'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 43, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 483, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/light/schema_json.py", line 346, in state_received
    values = json_loads_object(msg.payload)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 61, in json_loads_object
    value: JsonValueType = json_loads(__obj)
                           ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 47, in json_loads
    return orjson.loads(__obj)  # type:ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

2024-03-03 03:50:07.458 ERROR (MainThread) [homeassistant.util.logging] Exception in state_received when handling msg on 'awox-ble-mesh-hub/group-32768/state': 'ON'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 43, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 483, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/light/schema_json.py", line 346, in state_received
    values = json_loads_object(msg.payload)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 61, in json_loads_object
    value: JsonValueType = json_loads(__obj)
                           ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 47, in json_loads
    return orjson.loads(__obj)  # type:ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

2024-03-03 03:55:08.111 ERROR (MainThread) [homeassistant.util.logging] Exception in state_received when handling msg on 'awox-ble-mesh-hub/group-32768/state': 'ON'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 43, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 483, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/light/schema_json.py", line 346, in state_received
    values = json_loads_object(msg.payload)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 61, in json_loads_object
    value: JsonValueType = json_loads(__obj)
                           ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 47, in json_loads
    return orjson.loads(__obj)  # type:ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

2024-03-03 04:00:08.288 ERROR (MainThread) [homeassistant.util.logging] Exception in state_received when handling msg on 'awox-ble-mesh-hub/group-32768/state': 'ON'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 43, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 483, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/light/schema_json.py", line 346, in state_received
    values = json_loads_object(msg.payload)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 61, in json_loads_object
    value: JsonValueType = json_loads(__obj)
                           ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/json.py", line 47, in json_loads
    return orjson.loads(__obj)  # type:ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

For context this is from my ESP32 that connects via bluetooth to my AWOX lights and sends them to HA via MQTT.

PatpaC · March 3, 2024, 2:52pm

Okay i will try. Can you tell me what specs the PSU should have? I think more Amps? but how much?
I have never thought of this because it ran now for 1 1/2 years without a problem. Also the load did not increase in the last few weeks. But maybe it broke over the time.

francisp · March 3, 2024, 2:52pm

They do wear out.

iamjosh · March 3, 2024, 2:56pm

You can use the Raspberry Pi Power Supply Checker integration to check your power supply.

stevemann · March 3, 2024, 5:16pm

All power supplies lose efficiency over time. The Wall Wart USP supplies are barely adequate to power a Raspberry Pi, so yes after a couple of years it’s available power could drop below the functional needs of the Raspberry.

What size should you get? You can never have too many amps. I am running a server on a Raspberry Pi3 with a 5 Amp PSU. It’s been running for a few years.

PatpaC · March 3, 2024, 5:20pm

Good point. I allready have that since i set up my RPI because it was detected by HA itself. But it only shows OK since then.

PatpaC · March 3, 2024, 5:24pm

Oh ok did not know that they degrade that fast. I will buy one with more Amps and report back.
Thanks!

stevemann · March 3, 2024, 5:42pm

Please do. Others watching this thread would benefit.

WallyR · March 3, 2024, 9:04pm

You need a 3A 5V PSU for the Raspberry Pi 4.
The Raspberry can not draw more amperes than the 3A, but a PSU is more unstable the closer you get to its max output.
That means a good brand PSU can be one rated to 3A, but a less brand PSU rated to 5A might be just as stable at 3A as the good brand one.

PatpaC · April 24, 2024, 9:24am

Nearly forgot to report back…

HAOS gave me finally an output after it crashed. Not just a black screen.
There was a lot of journal and ext4-fs Errors.

I was close to take a backup and restore it onto a new hard drive(SSD).

But for some reason the problem is gone since it showed me those errors.
I have not swapped the PSU also have not swapped any hardware and there was no supervisor update or any update at that time.
It must be something that fixed it for me, but I can’t tell what.

Conclusion: I do not know what that was. I also do not know how I fixed it.
But it is solved…

zfed · June 28, 2024, 10:24pm

Hello!

I am fighting with same issue. I use Sensecap M1 case, fan (constantly on to 5V pin) and it worked fine for year or so… and then 2 weeks ago it started to freeze, all docker containers are down, but I could ping Raspbeery but not possible to access thru SSH.
Tasks ran on device:

TTN gateway;
HA;
mqtt broker;
influxDB cli version;
Grafana;
node red but not used.

+SSD with USB3

I tried to take out of case, maybe cooling?! but HA add-on about Raspberry status showed max 45C for CPU without fan and then I added it to 5V constanlty ON that leaded to 31…22C for CPU.

tried to fix eith:

PSU form Ali, stated 5A for Rpi5 - same;
other 3A PSU for other Sensecap M1 kit - same;
last to try was maybe SD card… ok, late evening, not possible to buy new… then I tool other Sensecap M1 that was just as TTN gateway only and it worked fine for weeks, I installed docker there, moved all above to this unit and same, after 4…5h it goes down;
no ideas how to even troubleshoot, shell I connect to display and try to catch log or screen behaviour… mistery.

PatpaC · July 3, 2024, 9:42pm

Hi,

You lack a lot of info here.
For Example:
Rpi Model.
What OS you run on.
Versions… etc…

I just assume you run HAOS, because that here is about HAOS.
Kinda hard to tell, because you said you had to install docker?

I have 2 things in my mind that you haven’t tried yet.(Or at least not listed here)

Make a full HA backup and restore it on a fresh installed SSD.
Use the 3.3V Pin on your Raspberry to power the fan.
BTW: 45°C is not even close to make any problem.

grafik