If you are running Home Assistant in a VM and experiencing clock-drift (automations fail to execute at the correct time) you may wish to consider adjusting the value of the VM’s ‘Paravirtualization’ parameter.
There’s an in-depth discussion in the following topic:
Have the same problem using hass os on esxi. Just applied Valentino’s workaround. Hoping tomorrow my air purifier stops on time
Happy to report the workaround resolved the issue for me. My air purifier stopped at 8am on the dot. Yesterday i also enabled the time service on both my esxi nodes. Will remove the workaround and report back whether that solved the underlying issue.
Ok removed the workaround and checked execution history and the automations are still on time. I guess setting the time service on esxi solved the issue for me.
This didn’t do it for me. I tried Valentino’s entity and I can see when the entity updates every second, but all of my timed trigger (and sun trigger) automations still trigger one hour to three hours late.
I have a custom esphome pool controller that relies on the Hime Assistant NTP to start/stop the pump that runs exactly on time too. It’s something with HAOS.
I had this problem for a long time, yesterday I was looking into the ESXi-7 and found it has an issue with the time, even when you go to the portal it will say the correct time but it isn’t, you have to start the daemon via ssh and change the time in the ESXi host manually (without NTP and via CLI), my solution was upgrading to ESXI-7U3 and make sure the time was set via SSH, since yesterday all my automation are running on perfect time
Hi im also having same issue with my automations.
My Setup:
Laptop Acer V5-573G
Proxmox v8.1.3
VM with HAOS (latest, always doing updates)
Whenever i create an automation which should trigger lets say now + 10 min it works.
But when the time when the automation should trigger is far away, like ill create it at 11:00 am, to trigger at 23:30 (11:30 pm) it will fail to stay on time, mostly around 2-3 hours.
Chrony addon from HACS did not help at all.
The suggested solution with Paravirtualization - i couldnt find anything to setup on my machine for that - anyone knows how to?
Please help me out with this, with this issue my Home Automation is just trash. Cause if this doesnt has a solution i can dump this whole project lol
Hello to all,
this is my first post in the community, even if I use HA since 2019 (or before).
I always used it on a VM (VMWare 16 before and 17.5 now) running on a mini-itx equipped with a Intel Celeron CPU J3455 1.50GHz with 8Gb RAM and 60Gb SSD.
The VM was created following the official instructions and setted on lower hardware requested (2cpus, 2Gb, 32Gb HD). HA was updated to the latest release during the last 2 months without any significant change in this behaviour.
What I noted in my system is the following:
I do not have such a lot of added components or integrations to be worried about the system performance… but could this delay depend on available system resources?
EDIT: for troubleshooting purposes, I duplicated one of my time-based automations and changed the trigger from time to the state of a time sensor (but keeping it at the same time, since the event will be just a new row in the history): I will let you know the result of this test as soon as possible.
my best,
Hi again, all,
here after, some corrections / updates to my post, after some testing:
now I can confirm that is not depending on the timeframe. Strange thing is that how far (in the time) I schedule an action does affect how much delay it will start with. I.e. if I activate now an automation that has to be started at a certain time that will occur in a few minutes (les of one hour) it will start almost in time; if it will occur in 3 to 24 hours, the delay will increase.
this is still true.
this seems to be confirmed but I have just clues, not any evidence: I cloned the VM and launched it on a different hardware. Leaving just one time-based automation it started perfectly on-time.
Now, the very important news is that, the automation triggered by the status of a time-based sensor started almost in time!
here is the code:
alias: Delay_check_SENSOR-BASED_1240
description: the same as the time-triggered one but based on a sensor
trigger:
- platform: state
entity_id:
- sensor.time
from: "12:39"
to: "12:40"
condition: []
action:
- service: logbook.log
data:
name: Automation punctuality notification
message: >-
Delay_check_SENSOR-BASED activated at {{
now().strftime('%H:%M:%S') }}
mode: restart
I created 8 of these automations, at different times during the 24h and, until now, they started just 1 or 2 seconds late, while the time-triggered ones took from 2" to 16’59"
| date | automation | scheduled time | actual time of execution | delay | delay delta |
|---|---|---|---|---|---|
| 22/01/2024 | delay check time based | 17:40 | 17:44:26 | 04:26 | 00:00 |
| 22/01/2024 | delay check time based | 18:40 | 18:47:18 | 07:18 | 02:52 |
| 22/01/2024 | delay check time based | 19:40 | 19:50:22 | 10:22 | 03:04 |
| 22/01/2024 | delay check time based | 20:40 | 20:40:57 | 00:57 | -09:25 |
| 22/01/2024 | delay check time based | 21:40 | 21:44:17 | 04:17 | 03:20 |
| 22/01/2024 | delay check time based | 22:40 | 22:47:47 | 07:47 | 03:30 |
| 22/01/2024 | delay check time based | 23:40 | 23:48:20 | 08:20 | 00:33 |
| 22/01/2024 | delay check time based | 00:00 | 00:08:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check time based | 00:35 | 00:43:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check time based | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check SENSOR BASED | 00:40 | 00:40:00 | 00:00 | |
| 22/01/2024 | host shutdown | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 23/01/2024 | host reboot | 00:58 | 00:58:39 | 00:00 | n/a |
| 23/01/2024 | delay check time based | 01:00 | 01:00:01 | 00:01 | |
| 23/01/2024 | delay check time based | 01:40 | 01:40:02 | 00:02 | 00:01 |
| 23/01/2024 | delay check time based | 02:40 | 02:40:44 | 00:44 | 00:42 |
| 23/01/2024 | delay check time based | 03:40 | 03:41:43 | 01:43 | 00:59 |
| 23/01/2024 | delay check time based | 04:40 | 04:42:50 | 02:50 | 01:07 |
| 23/01/2024 | delay check SENSOR BASED | 04:40 | 04:40:01 | 00:01 | |
| 23/01/2024 | delay check time based | 05:40 | 05:44:06 | 04:06 | 01:16 |
| 23/01/2024 | delay check time based | 06:40 | 06:45:30 | 05:30 | 01:24 |
| 23/01/2024 | delay check time based | 07:40 | 07:47:03 | 07:03 | 01:33 |
| 23/01/2024 | delay check time based | 08:40 | 08:48:45 | 08:45 | 01:42 |
| 23/01/2024 | delay check SENSOR BASED | 08:40 | 08:40:01 | 00:01 | |
| 23/01/2024 | delay check time based | 09:40 | 09:50:35 | 10:35 | 01:50 |
| 23/01/2024 | delay check time based | 10:40 | 10:52:34 | 12:34 | 01:59 |
| 23/01/2024 | delay check time based | 11:40 | 11:54:43 | 14:43 | 02:09 |
| 23/01/2024 | delay check time based | 12:40 | 12:56:59 | 16:59 | 02:16 |
| 23/01/2024 | delay check SENSOR BASED | 12:40 | 12:40:02 | 00:02 | |
| 23/01/2024 | delay check time based | 13:40 | 13:59:25 | 19:25 | 02:26 |
starting from the reboot task scheduled (externally) at 00:58, the delay is increasing as the time is passing by, but the sensor-based automations are delaying just a few.
This means that it cannnot be a time drift (I refer to ‘paravirtualization’) due to the VM… or am I wrong?
The test is still going on: I will update you all later.
Good morning, all
let’s share some updates
Confirmed but it seems that the delay increases as long as time passes by the last reboot, which resets the delay
Confirmed
Still confirmed without evidence
CONFIRMED!!!
| date | automation | scheduled time | actual time of execution | delay | delay delta |
|---|---|---|---|---|---|
| 22/01/2024 | delay check time based | 17:40 | 17:44:26 | 04:26 | |
| 22/01/2024 | delay check time based | 18:40 | 18:47:18 | 07:18 | 02:52 |
| 22/01/2024 | delay check time based | 19:40 | 19:50:22 | 10:22 | 03:04 |
| 22/01/2024 | delay check time based | 20:40 | 20:40:57 | 00:57 | -09:25 |
| 22/01/2024 | delay check time based | 21:40 | 21:44:17 | 04:17 | 03:20 |
| 22/01/2024 | delay check time based | 22:40 | 22:47:47 | 07:47 | 03:30 |
| 22/01/2024 | delay check time based | 23:40 | 23:48:20 | 08:20 | 00:33 |
| 22/01/2024 | delay check time based | 00:00 | 00:08:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check SENSOR BASED | 00:40 | 00:40:00 | 00:00 | |
| 22/01/2024 | delay check time based | 00:35 | 00:43:20 | 08:20 | 00:00 |
| 22/01/2024 | delay check time based | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 22/01/2024 | host shutdown | 00:40 | 00:48:20 | 08:20 | 00:00 |
| 23/01/2024 | host reboot | 00:58 | 00:58:39 | n/a | n/a |
| 23/01/2024 | delay check time based | 01:00 | 01:00:01 | 00:01 | -08:19 |
| 23/01/2024 | delay check time based | 01:40 | 01:40:02 | 00:02 | 00:01 |
| 23/01/2024 | delay check time based | 02:40 | 02:40:44 | 00:44 | 00:42 |
| 23/01/2024 | delay check time based | 03:40 | 03:41:43 | 01:43 | 00:59 |
| 23/01/2024 | delay check SENSOR BASED | 04:40 | 04:40:01 | 00:01 | 00:01 |
| 23/01/2024 | delay check time based | 04:40 | 04:42:50 | 02:50 | 01:07 |
| 23/01/2024 | delay check time based | 05:40 | 05:44:06 | 04:06 | 01:16 |
| 23/01/2024 | delay check time based | 06:40 | 06:45:30 | 05:30 | 01:24 |
| 23/01/2024 | delay check time based | 07:40 | 07:47:03 | 07:03 | 01:33 |
| 23/01/2024 | delay check SENSOR BASED | 08:40 | 08:40:01 | 00:01 | 00:00 |
| 23/01/2024 | delay check time based | 08:40 | 08:48:45 | 08:45 | 01:42 |
| 23/01/2024 | delay check time based | 09:40 | 09:50:35 | 10:35 | 01:50 |
| 23/01/2024 | delay check time based | 10:40 | 10:52:34 | 12:34 | 01:59 |
| 23/01/2024 | delay check time based | 11:40 | 11:54:43 | 14:43 | 02:09 |
| 23/01/2024 | delay check SENSOR BASED | 12:40 | 12:40:02 | 00:02 | 00:01 |
| 23/01/2024 | delay check time based | 12:40 | 12:56:59 | 16:59 | 02:16 |
| 23/01/2024 | delay check time based | 13:40 | 13:59:25 | 19:25 | 02:26 |
| 23/01/2024 | delay check time based | 14:40 | 15:02:00 | 22:00 | 02:35 |
| 23/01/2024 | delay check time based | 15:40 | 16:04:45 | 24:45 | 02:45 |
| 23/01/2024 | delay check SENSOR BASED | 16:40 | 16:40:02 | 00:02 | 00:00 |
| 23/01/2024 | delay check time based | 16:40 | 17:07:39 | 27:39 | 02:54 |
| 23/01/2024 | delay check time based | 17:40 | 18:10:43 | 30:43 | 03:04 |
| 23/01/2024 | delay check time based | 18:40 | 19:13:53 | 33:53 | 03:10 |
| 23/01/2024 | delay check time based | 19:40 | 20:17:17 | 37:17 | 03:24 |
| 23/01/2024 | delay check SENSOR BASED | 20:40 | 20:40:04 | 00:04 | 00:02 |
| 23/01/2024 | delay check time based | 20:40 | 21:20:49 | 40:49 | 03:32 |
| 23/01/2024 | delay check time based | 21:40 | 22:24:27 | 44:27 | 03:38 |
| 23/01/2024 | delay check time based | 22:40 | 23:26:27 | 46:27 | 02:00 |
| 23/01/2024 | delay check time based | 23:40 | 00:26:27 | 46:27 | 00:00 |
| 24/01/2024 | delay check SENSOR BASED | 00:40 | 00:40:00 | 00:00 | -00:04 |
| 24/01/2024 | delay check time based | 00:00 | 00:46:27 | 46:27 | 00:00 |
| 24/01/2024 | host reboot | 00:58 | 00:58:33 | n/a | n/a |
| 24/01/2024 | delay check time based | 01:00 | 01:00:00 | 00:00 | -46:27 |
| 24/01/2024 | delay check time based | 01:40 | 01:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 02:40 | 02:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 03:40 | 03:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 04:40 | 04:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check SENSOR BASED | 04:40 | 04:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 05:40 | 05:40:00 | 00:00 | 00:00 |
| 24/01/2024 | delay check time based | 06:40 | 06:40:45 | 00:45 | 00:45 |
| 24/01/2024 | delay check time based | 07:40 | 07:41:44 | 01:44 | 00:59 |
| 24/01/2024 | delay check SENSOR BASED | 08:40 | 08:40:01 | 00:01 | 00:01 |
| 24/01/2024 | delay check time based | 08:40 | 08:42:51 | 02:51 | 01:07 |
| 24/01/2024 | delay check time based | 09:40 | 09:44:07 | 04:07 | 01:16 |
Confirmed - weird thing: today the first delay occurred 5hs later than yesterday 
Yesterday I copied out all these automations into a cloned VM running on different phisical hardware (mini-itx equipped with a Intel Core i5 CPU 4590T 2.00GHz with 8Gb RAM and 250Gb SSD) and network and the result is terrifying: until now, all occurred perfectly on time. 
[2024-01-25T18:59:00Z] UPDATE
Dear friends
(even if it seems that I am talking to none 
)
today I tried:
to switch off all the application running on the phisical machine but the VM that contains HA → no result
to deselect the "Virtualize Intel VT-x/EPT or AMD-v/RVI option in VMware → no result
to find any other clue about what’s going on in HA or VM while the automation should be starting → found a clue!!!
lots of these messages in the Host Log:
Looking for these in google, unfortunately we come back to Paravirtualization VirtualBox problem with HA:
Unfortunately, there is no “Paravirtualization Interface” setting in VMware to be switched off. 
…but I am not convinced at all about the connection between this Para-thing and the automations triggering late. I mean:
…here is why now I am trying deselecting an option that, in my mind, should obviously be selected
all my finger are crossed.
[2024-01-26T08:44:00Z] VERY LAST UPDATE
I think we did it 
deselcting the VMware option about synchronizing guest time with host did the job and, since that, until now, all the automations triggered perfectly in time.
even if several details have to be clarified, if this solution is confirmed by other users, should they think adding this detail to the HA on VMWare installation guide?
[2024-02-15T14:02:00Z]
still monitoring → still working perfectly in time 
After reading the above, the only conclusion I can come to is that HA maintains two different “clocks”: One that it shows to users, and one that it uses to trigger automations.
If this is correct, I fail to see how it is a good idea.
No, HA only have the time provided by the hypervisor running on the host OS.
The problem is that this time is disconnected from the hardware time on the host OS.
The RTC HA sees is a virtual RTC and this one will time skew when the Host OS gets bogged down with other tasks, like installing Windows updates.
The integration packages usually include a SNTP or better yet a NTP client to set the time.
HA looks like it has the integration package for the QEMU hypervisor installed, so on those systems the time skew should be minimal, but for other hypervisors, there seems to be no integrations package included (which might be due to license/copyright limitations).
HA is using a SNTP client, which is required on RPis, because they have no RTC onboard, but RPis usually run HA directly or as a container, which makes the time skew very small.
The SNTP will also save some situations on a hypervisor, but the time skew on a hypervisor can be big and fluctuating, due to tasks on the host OS. A NTP client could solve it, but require multiple connections to time servers on the internet.
When talking about time in relation to VMs articles seems to differentiate between drift and time skew.
Drift is a near constant factor that makes the clock go too slow or too fast. Time skew on the other hand can go both ways and vary in degree from event to event.
In the article below there on the difference between NTP and SNTP there is an explaination of how SNTP works, which makes it clear that your bug hunting can be blurred by the fact that a SNTP client have updated the time since your problematic event and thereby removed the evidence of the time skew.
Except that what you’re saying doesn’t make any sense. These are several posts above about how HA shows the correct time, and yet the automations fail to execute at the correct time. The only way this is possible is if HA maintains two distinct clocks. Otherwise, when the clock strikes 12, X should happen - and it’s not.
Further, at least one of the prior posts references using vSphere as the host hypervisor, not Windows. vSphere is an enterprise-class product, and unless your %CPUREADY values are OBSCENE (in excess of 25-30%), you are not going to notice even a TINY bit of clock issue - vmware tools or not. VMware tools don’t make a BIT of difference in terms of clock synchronization, since vSphere is able to properly and correctly emulate that hardware and abstract it from the guest operating system. We aren’t talking about virtualbox on a Windows 10 host here - at least not for all of the issues that were reported above. If you have the skills to configure shared storage, and vMotion, and vSphere, and vCenter, and the ability to buy all that stuff, AND you had %CPUREADY issues even approaching anything that high, there is no possible chance that performance issues in every single one of the other VMs on that host would go unnoticed.
Hi, there,
I am not a geek (and please excuse me for that, reading the following) but I think that you both are a half right and wrong.
if the clock of the VM ran too slow or too fast, I expected the “delay delta” coulmn in my reports to be always almost the same - which it is not.
I actually do not know whether HA has two clocks but it is weird that the clock that I can show in the frontend AND the one taken as a reference for the history recorder are perfectly on time, while just the automation is starting late (I would have expected a history which shows events always perfectly in time but the actual events starting late versus the time that I can see on my watch, outside the VM). I think that the “triggering” is happening at the right time (or it does not make any sense in my mind that a computer is triggering at 22:47:27 intead of 21:40:00) but what is going on during the delay time? …and why changing the triggering method (the value of time instead the time itself) lowers to 4 seconds a delay already grown to 41 minutes?
we can see something’s wrong in the Host Log: it seems that VMWare is correcting the time of the hosted system with a frequency of 34" (?!?), causing, at the very end, a non-linear delay…but why? even if this correction is made on a puctual clock, it would not cause any problem but a small waste of resources… wouldn’t it?
Do you have a point you are attempting to make, or are you simply googling support docs?
Do you have hands-on experience with vSphere?
I haven’t got a ton of experience with VMware workstation with this type of workload, as this isn’t really what it’s designed for. I’ve used it on company machines for years as a way to have a clean environment to connect to customer systems using their VPN client of choice, but I only ever run one VM at a time, and when I’m finished with it, I shut it down, so… I don’t have much in the way of direct advice re: your particular installation - aside from to say that I scrolled back up, and I think your host system specs may be a bit on the anemic side. How many VMs are you running, and what else is the host doing aside from running VMware workstation?
I have little experience with vSphere, but I have experience with other hypervisors and the issue is a general one.
Even if I had experience with vSphere it would just be my word against yours, so I found it better to point you to the VMWare article regarding the issue on their products, which explains it pretty well.
Ok, we’re done here. LOL
