Mentioned back here, but there is a lot of info in this thread now.
It is a very simple 2 host cluster;
Hypervisor:VMware ESXi, 7.0.3, 19482537
Model:NUC11PAHi7
No shared storage currently. Identical hardware and esx versions.
Migrations between hosts are always compute and datastore, so it’s either all on one or the other host.
No difference in network configuration.
Home Assistant Core 2022.5.5
Home Assistant Supervisor 2022.05.2
Home Assistant OS 8.0
Kernel version 5.15.38
Agent version 1.2.1
I came here because I’m also having trouble with my automations triggering at odd times.
Here’s a dead simple 6pm automation, running 2 minutes late: It does indeed appear to be running 2 minutes later than 6pm, when I confirm against my iPhone time and wall clocks.
When you run in a VM you will by default not have access to a high precision realtime clock (RTC), due to the hardware abstraction layer the hypervisor have inserted between the VM and the actual hardware.
Some hypervisors have an integration package that can be installed on common guest OSes to provide the RTC feature.
Other solutions is to get an USB RTC and connect that directly to the VM.
I too am seeing the same behavior and I am not using a VM. In my case, I am using virtualenv installation of HA core (2022.11.5) on Xubuntu 22.04 on an intel celeron N3350 machine.
Automations using a time based trigger are triggering approx 3 minutes late.
Unsure how long this issue has been occurring. I only became aware of it after having recently enabled my holiday lighting routines and noticing they were triggering just a little late.
I confirmed that the home assistant machine local time is correct by verifying with {{now()}} in template console and via ‘date’ in terminal.
HA log shows no unusual lines associated with the automation triggers.
Going to experiment by adding a few seconds to the trigger time (e.g. at: ‘06:30:01’ ) to see if the behavior changes.
I have seen several of these issues posted on the forum and they are all related to hypervisor software.
My guess is that without the proper hypervisor guest integration install in the guest OS the time will slide a bit.
This makes sense. I think that the gazilion of docker containers drift apart in time, but this should not happen. Should be as simple as requesting NTP time to the host machine to sync.
On a bare bone system the hardware keeps a pretty precise clock frequency, but on a hypervisor that clock is emulated and the resources to keep it updated is shared with the host OS.
Specific times of day or do the automations use certain resources?
If the time is specific, then it can be related to when the ntp pulls are done, because these will correct any time skew at that point in time.
If it is a resource, then it can be delays in access to these that cause it.
I will add it is not just the automations set to an exact time- it is also the sunset/sunrise time based triggers that have the same problem
Has anyone found the source of the problem? It is so odd that the times are seemingly random and not off by a set value each day.
I had a telegram alert sent for each time the automation is triggered and the correct time was reported. Basically a copy of what was noted above on template editor
Maybe it is actually running on the correct time, but the logging is delayed in the database.
HA does not log the time the event happened, but the time it was written to the database.
It does not trigger the automation at the correct time. I don’t think the logging is the issue. My lights at specific time in AM and the PM lights that are based on sunset time are both off by 120 +/ 20ish minutes - similar to everyone above
edit: actually the time difference is always 2 hours, 23 minutes +/- 2 seconds - this is the same for various automations i have that go off at 10pm, 11pm, and a few hours earlier based on sunset time
I have found a reliable workaround that does not require to change any existing automations!
By coincidence I found out that if you add a entity that changes every second, it somehow makes ALL automations to trigger ON TIME. This can be a dummy sensor.
Just add following and you should be golden:
# Add a dummy entity that updates every second
template:
- trigger:
- platform: time_pattern
seconds: "/1"
sensor:
- name: timestamp
state: "{{ now().timestamp() }}"
# Exclude above entity from recorder
# to avoid filling up the database with useless info
recorder:
exclude:
entities:
- sensor.timestamp
