Timer got out of sync. Resetting - is there a solution?

do you use an FFMPEG camera?

Not sure if this helps but when booting up HA service, watching the terminal window i spotted a bit of a loop loading the FFMPEG camera.

HA would continue to load, but keep looping back to try and load the camera. It may not be your issue, but certainly helped me track down the culprit (as the timer got out of sync message didnt include any traceback in the log)

Experiencing the same thing, can’t seem to figure out what is wrong.

The error is even cause my automations based on platform time to malfunction. What I have noticed is that python process of hass is spiking when the error occurs.

@jaburges - I do not use FFMPEG, just some generic cameras. That’s a good thought though.

So randomly I bounced Home Assistant this morning to see what would happen. I ran clean up until a few minutes ago where I got a single out of sync:

2018-03-06 08:53:08 ERROR (MainThread) [homeassistant.core] Timer got out of sync. Resetting

What’s particularly interesting about this one, is that similar to what @phdelodder said it DID correspond with a slight resource usage spike. I also noticed that the DB size dropped so guessing it was the purge process running:

What all that means I haven’t a clue…

What I have learned so far that every minute the system must update the timer, if a process is running longer than a minute the error occures.

For my setup I have increased the logging level to INFO hoping to get a closer look at what might cause the a long running job/process/component.

@danodemano can you share you grafana setup?

Good info, I just set my logging to info also see if I can see anything odd.

What are you looking for on the Grafana? I haven’t done a ton with it, this one was specifically setup to monitor the resource usage with Home Assistant. I have a couple others ones I semi-use. Energy usage and Internet health (speedtests and pihole). Drop me a PM if you want to talk more, no need to clog up this discussion. :slight_smile:

It seems to mean that your database clean up process is taking a lot of system resources that are starving HA of the resources it needs.

4.5G is a very large database. Have you considered reducing the amount of data in it by reducing the number of days of data that it keeps?

lol, if that’s very large I would love to know what a 30 day database looks like. I only keep 2 days:

purge_interval: 1
purge_keep_days: 2
db_url: !secret recorder_db_url

I can’t keep any shorter than that and maintain the history for the various sensors.

I keep my database for three days, and it is only 37M, so there is something strange going on. Can you limit the sensor data that is going in to it by using the include/exclude directives?

I’m already doing this as best as I can: https://github.com/danodemano/Home-AssistantConfig/blob/master/recorder.yaml

I think you need to find out what is saving so much data. Storing 2GB/day must be more than just temperature recordings.

Do some sensors save pictures or video that is getting put in the DB?

If I was to take a wild guess, I would say it’s my z-wave energy meter. I do have some temp sensors, door/window sensors, presence and location, and some other misc stuff like bitcoin prices and speedtest runs.

As far as I know there isn’t any video. I use the generic camera but I don’t think that’s logged to the database.

What would be the best way to figure out what’s eating up all the space?

You can probably do it by querying the DB with SQL commands, but my knowledge of SQL is rudimentary.

I just did a quick look, and there doesn’t seem to be anything in the log file when the database is written to, so tracking it that way doesn’t look likely.

So I am out of ideas. Maybe you could create a new thread with that specific question to get some more input.

So this is interesting @phdelodder - since enabling info logging and restarting Home Assistant this morning I only have one out of sync error and it was right after startup so I think that was a non-issue.

Maybe related…maybe not when I posted that screenshot above and the DB sized dropped from ~4.5 to ~1.5 GB during the purge.

This gives some potential to @gpbenton 's theory on DB size. Which by the way appears to be the energy monitors. My info log in almost 100MB since this morning and most of it is state changes for my server rack monitor. Going to see if I can adjust it’s reporting frequency.