Home Assistant crashing after clocks changed?

My last entry in the logbook was 2:59:39 AM and then HA seemed to have frozen.
A shutdown/start solved it.

I had the same problem.
Automatic winter time change has caused HA failuer. After restart it runs ok.

Probable reason was python 3 crush when my system run out of memory because of the recorder reaching 30000 queue.

2021-10-31 02:00:55 ERROR (MainThread) [homeassistant.components.recorder] The recorder queue reached the maximum size of 30000; Events are no longer being recorded

Almost certainly, something in the core HA is working on local time rather than UTC and ended up with a negative value of time.

Really tricky to test for!

You could test for it to set the UTC time zone to one ahead and see if it happens again?
But why a reboot would fix this i dont know, as the time difference would be the same afterwards.

Did this happen last year?

Not really. It is the fact that, in local time, you get 2 lots activity of 1am times for the same date.

No, but so much has changed since. It also seems it might be the latest release. I’ve edited my response for the release I am running - UK is not the first DST change this autumn.

1 Like

I’m sorry for you, but this is pretty funny :smile:

It also shows how far Home Assistant is still away from being the mature version 1.0 that was touted for so long before changing versioning schemes… FWIW, my HA core did not experience an issue, even though I (still) use a time pattern trigger. Then again, I haven’t updated HA for over 6 months.

Tricky ? It’s literally one single command on the CLI to artificially turn the clock back an hour and observe what happens. There’s no excuse for not testing this.

1 Like

Except you do also need to stop the system auto resetting the clock. Also if you do that, you are actually changing the UTC, not the local time (which will change in step). To replicate DST change, you possibly need to change the locale (to change the local time without changing UTC) but even then, there is no guarantee you are replicating the situation where UTC marches on, but the local time repeats for an hour.

DST is a real bugger from a programming perspective! You must do all calculations in UTC under the hood.

1 Like

Ive had this and ive rebooted and its fine - but my HD usage has gone up 10%. Presumably this is log files and DB entries. Will this all just eventually clear itself and reduce?

1 Like

Noticed the same problem this morning, after the winter time automatic change.
Also had a couple of high cpu usage notifications, showing that the system had additional load than expected.

One side question: how is data in this “repeated” hour saved normally on historic systems? I mean, you have data from 2:00 to 2:59, then you have new data from 2:00 to 2:59 then the system continues logging data at 3:00
Is HA is able to display this data?

considering recorder stopped logging any data after dst change until reboot, likely there is no data to display at all :wink:

But returning to your question: afaik HA stores local time, with all consequences (which doesn’s justify the tonight’s behavior)

1 Like

@ baz123, oh please. DST is a solved problem. The way Linux handles the clock and locale is somewhat convoluted, but documented and perfectly replicatable. In the absolute worst case, you can set up a fake NTP server to reproduce as many DST changes you want, the whole day long.

I’ve done it myself, multiple times. With license management on commercial software. Means that if something goes wrong, customers will either be able to use a 40k a seat license for free or sue you because all their licenses went down. It’s perfectly doable, there’s nothing magic about it.

The HA devs simply forgot about it. It’s as simple as that. It’s not part of their testing protocol. Which shows a lack of product maturity. Anyway, this is probably OT.

6 Likes

Same here, around 3AM crashed due winter time trigger. It’s was a suprise this morning for me. I just have to reboot and everything is back fine luckely.

Similar issue this morning.

Got woken up 4:45 by my son saying his light switch doesn’t work for his Lego’ing (Shelly in detached mode, controlling hue colour candle bulb).
I told him to go to bed for another hour and then it would work:)

Anyway, HA’s webpage appeared non existent.
Rebooted VM Guest and all appeared fine.
Not had this before (i’m quite up to date core-2021.10.4 / supervisor-2021.10.6)

Hit by the DST bug as well :frowning: Luckily there is a year time to fix the bug, I believe there’s no issue when changing time in the opposite direction, or is there?

1 Like

Yes, a year for those of us who have just gone through it, but the US will go through this in 7 days, so it’s going to need fixing pretty fast.

3 Likes

Wow. Brings back memories of the early days of mainframes. We had one system we used to have to shut down for that hour during the time change. I might have to do that for the US time change if the fix isn’t out by then. Glad they moved the date we change the clocks a few years ago!

1 Like

It also seems funny to me now, but at three o’clock in the morning it was like a carousel :smiley:

It will likely kill some SD card systems though. Writing 50 ish lines every second to the log, will be pretty tough on the SD card, my log grew to 127mb during this period. Thankfully I am not on an SD card system.

After DST change, the first automation triggered by a time pattern got in loop and triggered 16 times per second, that is the amount of log entries I found. From that time on all my sensors (MQTT and RfxTrx) flatlined. The only remedy was to restart HA. I can imagine that this amount of info written to the logs can be the cause of many different errors.

Lucky my defaults for crashing are ‘light off’. I do not make use of time pattern triggers, but I do have several time sensors that seemed to have run out of sync at 2:00 in the morning.

I run two instances of HassOS headless, on a Pi3b and the other on a Pi4. Both crashed past night. Burned a couple of SSD’s in the past by plugging the power off/on whenever an HA instance crashed. Since I am stable since a couple of months on new SSD’s, I now resolved the issue by:

  • Connecting a USB keyboard to the Pi.
  • Type root and hit enter.
  • Type core start and hit enter.
  • Wait 2-3 minutes and try to connect.

Since our alarm clock is set in Home Assistant for radio and wake-up lights, this problem could have resulted in us not waking up on time. Guess having set a backup alarm because we suspected this crash was not a bad idea. :crazy_face: