Why is it crashing every morning?

nicktheguitar · November 27, 2020, 9:55am

Since about the 20th November my RasPi3+ HASSOS instance has been crashing every morning some time between 0730 and 0830.

I’ve got no automations running at those times, the only thing it would be doing is reporting some light and media player states for various chromecasts the kids are watching/listening to.

An example is this plot for processor use on the Pi, note the spikes and then flat lines until I pull the plug and restart it.

Does anyone have any ideas of what it could be or how I could diagnose it? Obviously the logs are blank when I restart it but there’s nothing in there of note for the rest of the day, just the odd warning for a slow responding API or two.

tom_l · November 27, 2020, 11:47am

What’s your memory use like?

nicktheguitar · November 27, 2020, 5:37pm

Seems to be doing some heavy lifting just before the crashes. This is the ‘Memory Free’ sensor.

tom_l · November 27, 2020, 5:50pm

Yeah that’s the reason for the crash (out of memory). The cause though…

You could start by disabling custom components one by one (if you have any). It’s a guess but they do tend to be a major cause of issues.

Or if you can monitor your memory use from 7:30 and notice it start to climb run Profler and or Py-spy and report your reults in this topic for analysis: Python3 high CPU Usage

finity · November 27, 2020, 6:36pm

I also have noticed a slow rise ion my memory usage lately. I’m not sure when it started tho but it goes up at about 2-3% per day.

I killed each of my containers one by one (except HA) and watched for a reduction in memory use but none of those things changed it.

I’m going to update HA just to see if that will help and then I’ll dig into HA itself to see if I can find the source of the leak.

I know it’s not the same symptoms as what you’re seeing but I figured I’d throw it out here as something related.

tom_l · November 27, 2020, 7:01pm

That’s the thing about such a complex extensible system my memory is rock steady at 14% but Increased CPU usage

nicktheguitar · November 29, 2020, 9:26am

I had a guess from looking at the times of the processor use chart that it was something to do with the Sun component, as those times were roughly to do with sunrise and set, but it was inconsistent and the sun is very consistent! I have two CCTV cameras streaming to the system and they’d change from monochrome to colour at around those times too, and the times would change depending on light conditions. I killed the camera components at about 1530 yesterday and no crash this morning. this is the chart so far: