Specific times of day or do the automations use certain resources?
If the time is specific, then it can be related to when the ntp pulls are done, because these will correct any time skew at that point in time.
If it is a resource, then it can be delays in access to these that cause it.
I have the exact same problem. I have checked what was suggested above.
My setup is a docker VM on an ESXI host.
I logged into each and all times match – ESXI host, ubuntu VM, and the docker container all match. I have setup manual NTP syncs.
Also tried mounting the ubuntu VM’s time into the container:
- "/etc/timezone:/etc/timezone:ro"
- "/etc/localtime:/etc/localtime:ro"
I will add it is not just the automations set to an exact time- it is also the sunset/sunrise time based triggers that have the same problem
Has anyone found the source of the problem? It is so odd that the times are seemingly random and not off by a set value each day.
I had a telegram alert sent for each time the automation is triggered and the correct time was reported. Basically a copy of what was noted above on template editor
Maybe it is actually running on the correct time, but the logging is delayed in the database.
HA does not log the time the event happened, but the time it was written to the database.
It does not trigger the automation at the correct time. I don’t think the logging is the issue. My lights at specific time in AM and the PM lights that are based on sunset time are both off by 120 +/ 20ish minutes - similar to everyone above
edit: actually the time difference is always 2 hours, 23 minutes +/- 2 seconds - this is the same for various automations i have that go off at 10pm, 11pm, and a few hours earlier based on sunset time
I have found a reliable workaround that does not require to change any existing automations!
By coincidence I found out that if you add a entity that changes every second, it somehow makes ALL automations to trigger ON TIME. This can be a dummy sensor.
Just add following and you should be golden:
# Add a dummy entity that updates every second
template:
- trigger:
- platform: time_pattern
seconds: "/1"
sensor:
- name: timestamp
state: "{{ now().timestamp() }}"
# Exclude above entity from recorder
# to avoid filling up the database with useless info
recorder:
exclude:
entities:
- sensor.timestamp
2 Likes
If you are running Home Assistant in a VM and experiencing clock-drift (automations fail to execute at the correct time) you may wish to consider adjusting the value of the VM’s ‘Paravirtualization’ parameter.
There’s an in-depth discussion in the following topic:
Have the same problem using hass os on esxi. Just applied Valentino’s workaround. Hoping tomorrow my air purifier stops on time
Happy to report the workaround resolved the issue for me. My air purifier stopped at 8am on the dot. Yesterday i also enabled the time service on both my esxi nodes. Will remove the workaround and report back whether that solved the underlying issue.
Ok removed the workaround and checked execution history and the automations are still on time. I guess setting the time service on esxi solved the issue for me.
This didn’t do it for me. I tried Valentino’s entity and I can see when the entity updates every second, but all of my timed trigger (and sun trigger) automations still trigger one hour to three hours late.
I have a custom esphome pool controller that relies on the Hime Assistant NTP to start/stop the pump that runs exactly on time too. It’s something with HAOS.
I had this problem for a long time, yesterday I was looking into the ESXi-7 and found it has an issue with the time, even when you go to the portal it will say the correct time but it isn’t, you have to start the daemon via ssh and change the time in the ESXi host manually (without NTP and via CLI), my solution was upgrading to ESXI-7U3 and make sure the time was set via SSH, since yesterday all my automation are running on perfect time
Hi im also having same issue with my automations.
My Setup:
Laptop Acer V5-573G
Proxmox v8.1.3
VM with HAOS (latest, always doing updates)
Whenever i create an automation which should trigger lets say now + 10 min it works.
But when the time when the automation should trigger is far away, like ill create it at 11:00 am, to trigger at 23:30 (11:30 pm) it will fail to stay on time, mostly around 2-3 hours.
Chrony addon from HACS did not help at all.
The suggested solution with Paravirtualization - i couldnt find anything to setup on my machine for that - anyone knows how to?
Please help me out with this, with this issue my Home Automation is just trash. Cause if this doesnt has a solution i can dump this whole project lol
Hello to all,
this is my first post in the community, even if I use HA since 2019 (or before).
I always used it on a VM (VMWare 16 before and 17.5 now) running on a mini-itx equipped with a Intel Celeron CPU J3455 1.50GHz with 8Gb RAM and 60Gb SSD.
The VM was created following the official instructions and setted on lower hardware requested (2cpus, 2Gb, 32Gb HD). HA was updated to the latest release during the last 2 months without any significant change in this behaviour.
What I noted in my system is the following:
- if the automations should start just after the midniight, they will delay;
- if I schediule an automation around noon, it will start almost perfectly in-time;
- all the automations scheduled in the same “time frame” will delay exactly the same amount of time ;
- the more I increase the active time-based automations (no matter whether the time frame is or is not the same), the more will increase the delay ;
- the recorder does not show anything’s special going on at the same time of the automations that are delaying.
I do not have such a lot of added components or integrations to be worried about the system performance… but could this delay depend on available system resources?
EDIT: for troubleshooting purposes, I duplicated one of my time-based automations and changed the trigger from time to the state of a time sensor (but keeping it at the same time, since the event will be just a new row in the history): I will let you know the result of this test as soon as possible.
my best,
1 Like
Hi again, all,
here after, some corrections / updates to my post, after some testing:
now I can confirm that is not depending on the timeframe. Strange thing is that how far (in the time) I schedule an action does affect how much delay it will start with. I.e. if I activate now an automation that has to be started at a certain time that will occur in a few minutes (les of one hour) it will start almost in time; if it will occur in 3 to 24 hours, the delay will increase.
this is still true.
this seems to be confirmed but I have just clues, not any evidence: I cloned the VM and launched it on a different hardware. Leaving just one time-based automation it started perfectly on-time.
Now, the very important news is that, the automation triggered by the status of a time-based sensor started almost in time!
here is the code:
alias: Delay_check_SENSOR-BASED_1240
description: the same as the time-triggered one but based on a sensor
trigger:
- platform: state
entity_id:
- sensor.time
from: "12:39"
to: "12:40"
condition: []
action:
- service: logbook.log
data:
name: Automation punctuality notification
message: >-
Delay_check_SENSOR-BASED activated at {{
now().strftime('%H:%M:%S') }}
mode: restart
I created 8 of these automations, at different times during the 24h and, until now, they started just 1 or 2 seconds late, while the time-triggered ones took from 2" to 16’59"
Take a look to this log
| date | automation | scheduled time | actual time of execution | delay | delay delta |
|---|---|---|---|---|---|
| 22/01/2024 | delay check time based | 17:40 | 17:44:26 | 04:26 | 00:00 |
| 22/01/2024 | delay check time based | 18:40 | 18:47:18 | 07:18 | 02:52 |
| 22/01/2024 | delay check time based | 19:40 | 19:50:22 | 10:22 | 03:04 |
| 22/01/2024 | delay check time based | 20:40 | 20:40:57 | 00:57 | -09:25 |
| 22/01/2024 | delay check time based | 21:40 | 21:44:17 | 04:17 | 03:20 |
| 22/01/2024 | delay check time based | 22:40 | 22:47:47 | 07:47 | 03:30 |
| 22/01/2024 | delay check time based | 23:40 | 23:48:20 | 08:20 | 00:33 |
| 22/01/2024 | delay check time based | 00:00 | 00:08:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check time based | 00:35 | 00:43:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check time based | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check SENSOR BASED | 00:40 | 00:40:00 | 00:00 | |
| 22/01/2024 | host shutdown | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 23/01/2024 | host reboot | 00:58 | 00:58:39 | 00:00 | n/a |
| 23/01/2024 | delay check time based | 01:00 | 01:00:01 | 00:01 | |
| 23/01/2024 | delay check time based | 01:40 | 01:40:02 | 00:02 | 00:01 |
| 23/01/2024 | delay check time based | 02:40 | 02:40:44 | 00:44 | 00:42 |
| 23/01/2024 | delay check time based | 03:40 | 03:41:43 | 01:43 | 00:59 |
| 23/01/2024 | delay check time based | 04:40 | 04:42:50 | 02:50 | 01:07 |
| 23/01/2024 | delay check SENSOR BASED | 04:40 | 04:40:01 | 00:01 | |
| 23/01/2024 | delay check time based | 05:40 | 05:44:06 | 04:06 | 01:16 |
| 23/01/2024 | delay check time based | 06:40 | 06:45:30 | 05:30 | 01:24 |
| 23/01/2024 | delay check time based | 07:40 | 07:47:03 | 07:03 | 01:33 |
| 23/01/2024 | delay check time based | 08:40 | 08:48:45 | 08:45 | 01:42 |
| 23/01/2024 | delay check SENSOR BASED | 08:40 | 08:40:01 | 00:01 | |
| 23/01/2024 | delay check time based | 09:40 | 09:50:35 | 10:35 | 01:50 |
| 23/01/2024 | delay check time based | 10:40 | 10:52:34 | 12:34 | 01:59 |
| 23/01/2024 | delay check time based | 11:40 | 11:54:43 | 14:43 | 02:09 |
| 23/01/2024 | delay check time based | 12:40 | 12:56:59 | 16:59 | 02:16 |
| 23/01/2024 | delay check SENSOR BASED | 12:40 | 12:40:02 | 00:02 | |
| 23/01/2024 | delay check time based | 13:40 | 13:59:25 | 19:25 | 02:26 |
starting from the reboot task scheduled (externally) at 00:58, the delay is increasing as the time is passing by, but the sensor-based automations are delaying just a few.
This means that it cannnot be a time drift (I refer to ‘paravirtualization’) due to the VM… or am I wrong?
The test is still going on: I will update you all later.
Good morning, all
let’s share some updates
Confirmed but it seems that the delay increases as long as time passes by the last reboot, which resets the delay
Confirmed
Still confirmed without evidence
CONFIRMED!!!
new log (covering 24+hs)
| date | automation | scheduled time | actual time of execution | delay | delay delta |
|---|---|---|---|---|---|
| 22/01/2024 | delay check time based | 17:40 | 17:44:26 | 04:26 | |
| 22/01/2024 | delay check time based | 18:40 | 18:47:18 | 07:18 | 02:52 |
| 22/01/2024 | delay check time based | 19:40 | 19:50:22 | 10:22 | 03:04 |
| 22/01/2024 | delay check time based | 20:40 | 20:40:57 | 00:57 | -09:25 |
| 22/01/2024 | delay check time based | 21:40 | 21:44:17 | 04:17 | 03:20 |
| 22/01/2024 | delay check time based | 22:40 | 22:47:47 | 07:47 | 03:30 |
| 22/01/2024 | delay check time based | 23:40 | 23:48:20 | 08:20 | 00:33 |
| 22/01/2024 | delay check time based | 00:00 | 00:08:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check SENSOR BASED | 00:40 | 00:40:00 | 00:00 | |
| 22/01/2024 | delay check time based | 00:35 | 00:43:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check time based | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 22/01/2024 | host shutdown | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 23/01/2024 | host reboot | 00:58 | 00:58:39 | n/a | n/a |
| 23/01/2024 | delay check time based | 01:00 | 01:00:01 | 00:01 | -08:19 |
| 23/01/2024 | delay check time based | 01:40 | 01:40:02 | 00:02 | 00:01 |
| 23/01/2024 | delay check time based | 02:40 | 02:40:44 | 00:44 | 00:42 |
| 23/01/2024 | delay check time based | 03:40 | 03:41:43 | 01:43 | 00:59 |
| 23/01/2024 | delay check SENSOR BASED | 04:40 | 04:40:01 | 00:01 | 00:01 |
| 23/01/2024 | delay check time based | 04:40 | 04:42:50 | 02:50 | 01:07 |
| 23/01/2024 | delay check time based | 05:40 | 05:44:06 | 04:06 | 01:16 |
| 23/01/2024 | delay check time based | 06:40 | 06:45:30 | 05:30 | 01:24 |
| 23/01/2024 | delay check time based | 07:40 | 07:47:03 | 07:03 | 01:33 |
| 23/01/2024 | delay check SENSOR BASED | 08:40 | 08:40:01 | 00:01 | 00:00 |
| 23/01/2024 | delay check time based | 08:40 | 08:48:45 | 08:45 | 01:42 |
| 23/01/2024 | delay check time based | 09:40 | 09:50:35 | 10:35 | 01:50 |
| 23/01/2024 | delay check time based | 10:40 | 10:52:34 | 12:34 | 01:59 |
| 23/01/2024 | delay check time based | 11:40 | 11:54:43 | 14:43 | 02:09 |
| 23/01/2024 | delay check SENSOR BASED | 12:40 | 12:40:02 | 00:02 | 00:01 |
| 23/01/2024 | delay check time based | 12:40 | 12:56:59 | 16:59 | 02:16 |
| 23/01/2024 | delay check time based | 13:40 | 13:59:25 | 19:25 | 02:26 |
| 23/01/2024 | delay check time based | 14:40 | 15:02:00 | 22:00 | 02:35 |
| 23/01/2024 | delay check time based | 15:40 | 16:04:45 | 24:45 | 02:45 |
| 23/01/2024 | delay check SENSOR BASED | 16:40 | 16:40:02 | 00:02 | 00:00 |
| 23/01/2024 | delay check time based | 16:40 | 17:07:39 | 27:39 | 02:54 |
| 23/01/2024 | delay check time based | 17:40 | 18:10:43 | 30:43 | 03:04 |
| 23/01/2024 | delay check time based | 18:40 | 19:13:53 | 33:53 | 03:10 |
| 23/01/2024 | delay check time based | 19:40 | 20:17:17 | 37:17 | 03:24 |
| 23/01/2024 | delay check SENSOR BASED | 20:40 | 20:40:04 | 00:04 | 00:02 |
| 23/01/2024 | delay check time based | 20:40 | 21:20:49 | 40:49 | 03:32 |
| 23/01/2024 | delay check time based | 21:40 | 22:24:27 | 44:27 | 03:38 |
| 23/01/2024 | delay check time based | 22:40 | 23:26:27 | 46:27 | 02:00 |
| 23/01/2024 | delay check time based | 23:40 | 00:26:27 | 46:27 | 00:00 |
| 24/01/2024 | delay check SENSOR BASED | 00:40 | 00:40:00 | 00:00 | -00:04 |
| 24/01/2024 | delay check time based | 00:00 | 00:46:27 | 46:27 | 00:00 |
| 24/01/2024 | host reboot | 00:58 | 00:58:33 | n/a | n/a |
| 24/01/2024 | delay check time based | 01:00 | 01:00:00 | 00:00 | -46:27 |
| 24/01/2024 | delay check time based | 01:40 | 01:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 02:40 | 02:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 03:40 | 03:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 04:40 | 04:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check SENSOR BASED | 04:40 | 04:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 05:40 | 05:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 06:40 | 06:40:45 | 00:45 | 00:45 |
| 24/01/2024 | delay check time based | 07:40 | 07:41:44 | 01:44 | 00:59 |
| 24/01/2024 | delay check SENSOR BASED | 08:40 | 08:40:01 | 00:01 | 00:01 |
| 24/01/2024 | delay check time based | 08:40 | 08:42:51 | 02:51 | 01:07 |
| 24/01/2024 | delay check time based | 09:40 | 09:44:07 | 04:07 | 01:16 |
Confirmed - weird thing: today the first delay occurred 5hs later than yesterday 
Yesterday I copied out all these automations into a cloned VM running on different phisical hardware (mini-itx equipped with a Intel Core i5 CPU 4590T 2.00GHz with 8Gb RAM and 250Gb SSD) and network and the result is terrifying: until now, all occurred perfectly on time. 
[2024-01-25T18:59:00Z] UPDATE
Dear friends
(even if it seems that I am talking to none 
)
today I tried:
-
to switch off all the application running on the phisical machine but the VM that contains HA → no result
-
to deselect the "Virtualize Intel VT-x/EPT or AMD-v/RVI option in VMware → no result
-
to find any other clue about what’s going on in HA or VM while the automation should be starting → found a clue!!!
lots of these messages in the Host Log:
Looking for these in google, unfortunately we come back to Paravirtualization VirtualBox problem with HA:
Unfortunately, there is no “Paravirtualization Interface” setting in VMware to be switched off. 
…but I am not convinced at all about the connection between this Para-thing and the automations triggering late. I mean:
- if there is a time drift, why should it affect just the automation trigger and not the clock itself?
- why a clone of the VM, played on different hardware, is not affected by the same defect?
…here is why now I am trying deselecting an option that, in my mind, should obviously be selected
all my finger are crossed.
[2024-01-26T08:44:00Z] VERY LAST UPDATE
I think we did it 
deselcting the VMware option about synchronizing guest time with host did the job and, since that, until now, all the automations triggered perfectly in time.
even if several details have to be clarified, if this solution is confirmed by other users, should they think adding this detail to the HA on VMWare installation guide?
[2024-02-15T14:02:00Z]
still monitoring → still working perfectly in time 
1 Like
After reading the above, the only conclusion I can come to is that HA maintains two different “clocks”: One that it shows to users, and one that it uses to trigger automations.
If this is correct, I fail to see how it is a good idea.
No, HA only have the time provided by the hypervisor running on the host OS.
The problem is that this time is disconnected from the hardware time on the host OS.
The RTC HA sees is a virtual RTC and this one will time skew when the Host OS gets bogged down with other tasks, like installing Windows updates.
The integration packages usually include a SNTP or better yet a NTP client to set the time.
HA looks like it has the integration package for the QEMU hypervisor installed, so on those systems the time skew should be minimal, but for other hypervisors, there seems to be no integrations package included (which might be due to license/copyright limitations).
HA is using a SNTP client, which is required on RPis, because they have no RTC onboard, but RPis usually run HA directly or as a container, which makes the time skew very small.
The SNTP will also save some situations on a hypervisor, but the time skew on a hypervisor can be big and fluctuating, due to tasks on the host OS. A NTP client could solve it, but require multiple connections to time servers on the internet.
When talking about time in relation to VMs articles seems to differentiate between drift and time skew.
Drift is a near constant factor that makes the clock go too slow or too fast. Time skew on the other hand can go both ways and vary in degree from event to event.
In the article below there on the difference between NTP and SNTP there is an explaination of how SNTP works, which makes it clear that your bug hunting can be blurred by the fact that a SNTP client have updated the time since your problematic event and thereby removed the evidence of the time skew.
Except that what you’re saying doesn’t make any sense. These are several posts above about how HA shows the correct time, and yet the automations fail to execute at the correct time. The only way this is possible is if HA maintains two distinct clocks. Otherwise, when the clock strikes 12, X should happen - and it’s not.
Further, at least one of the prior posts references using vSphere as the host hypervisor, not Windows. vSphere is an enterprise-class product, and unless your %CPUREADY values are OBSCENE (in excess of 25-30%), you are not going to notice even a TINY bit of clock issue - vmware tools or not. VMware tools don’t make a BIT of difference in terms of clock synchronization, since vSphere is able to properly and correctly emulate that hardware and abstract it from the guest operating system. We aren’t talking about virtualbox on a Windows 10 host here - at least not for all of the issues that were reported above. If you have the skills to configure shared storage, and vMotion, and vSphere, and vCenter, and the ability to buy all that stuff, AND you had %CPUREADY issues even approaching anything that high, there is no possible chance that performance issues in every single one of the other VMs on that host would go unnoticed.