My automations with set time triggers always trigger later than requested

INkEtIcU · May 9, 2023, 7:14am

@networked Have you found a solution? I have the exact same issue and setup.

ludorl82 · May 30, 2023, 12:55pm

Have the same problem using hass os on esxi. Just applied Valentino’s workaround. Hoping tomorrow my air purifier stops on time

ludorl82 · May 31, 2023, 12:04pm

Happy to report the workaround resolved the issue for me. My air purifier stopped at 8am on the dot. Yesterday i also enabled the time service on both my esxi nodes. Will remove the workaround and report back whether that solved the underlying issue.

ludorl82 · June 3, 2023, 1:35am

Ok removed the workaround and checked execution history and the automations are still on time. I guess setting the time service on esxi solved the issue for me.

FTWMaker · June 23, 2023, 2:38am

This didn’t do it for me. I tried Valentino’s entity and I can see when the entity updates every second, but all of my timed trigger (and sun trigger) automations still trigger one hour to three hours late.

I have a custom esphome pool controller that relies on the Hime Assistant NTP to start/stop the pump that runs exactly on time too. It’s something with HAOS.

cfpandrade · July 22, 2023, 12:46pm

I had this problem for a long time, yesterday I was looking into the ESXi-7 and found it has an issue with the time, even when you go to the portal it will say the correct time but it isn’t, you have to start the daemon via ssh and change the time in the ESXi host manually (without NTP and via CLI), my solution was upgrading to ESXI-7U3 and make sure the time was set via SSH, since yesterday all my automation are running on perfect time

JericoPablo · January 21, 2024, 4:17pm

Hi im also having same issue with my automations.

My Setup:
Laptop Acer V5-573G
Proxmox v8.1.3
VM with HAOS (latest, always doing updates)

Whenever i create an automation which should trigger lets say now + 10 min it works.
But when the time when the automation should trigger is far away, like ill create it at 11:00 am, to trigger at 23:30 (11:30 pm) it will fail to stay on time, mostly around 2-3 hours.

Chrony addon from HACS did not help at all.

The suggested solution with Paravirtualization - i couldnt find anything to setup on my machine for that - anyone knows how to?

Please help me out with this, with this issue my Home Automation is just trash. Cause if this doesnt has a solution i can dump this whole project lol

Deimos76 · January 22, 2024, 3:51pm

Hello to all,

this is my first post in the community, even if I use HA since 2019 (or before).

I always used it on a VM (VMWare 16 before and 17.5 now) running on a mini-itx equipped with a Intel Celeron CPU J3455 1.50GHz with 8Gb RAM and 60Gb SSD.
The VM was created following the official instructions and setted on lower hardware requested (2cpus, 2Gb, 32Gb HD). HA was updated to the latest release during the last 2 months without any significant change in this behaviour.

What I noted in my system is the following:

if the automations should start just after the midniight, they will delay;
if I schediule an automation around noon, it will start almost perfectly in-time;
all the automations scheduled in the same “time frame” will delay exactly the same amount of time ;
the more I increase the active time-based automations (no matter whether the time frame is or is not the same), the more will increase the delay ;
the recorder does not show anything’s special going on at the same time of the automations that are delaying.

I do not have such a lot of added components or integrations to be worried about the system performance… but could this delay depend on available system resources?

EDIT: for troubleshooting purposes, I duplicated one of my time-based automations and changed the trigger from time to the state of a time sensor (but keeping it at the same time, since the event will be just a new row in the history): I will let you know the result of this test as soon as possible.

my best,

Deimos76 · January 23, 2024, 1:39pm

Hi again, all,

here after, some corrections / updates to my post, after some testing:

now I can confirm that is not depending on the timeframe. Strange thing is that how far (in the time) I schedule an action does affect how much delay it will start with. I.e. if I activate now an automation that has to be started at a certain time that will occur in a few minutes (les of one hour) it will start almost in time; if it will occur in 3 to 24 hours, the delay will increase.

this is still true.

this seems to be confirmed but I have just clues, not any evidence: I cloned the VM and launched it on a different hardware. Leaving just one time-based automation it started perfectly on-time.

Now, the very important news is that, the automation triggered by the status of a time-based sensor started almost in time!

here is the code:

alias: Delay_check_SENSOR-BASED_1240
description: the same as the time-triggered one but based on a sensor
trigger:
  - platform: state
    entity_id:
      - sensor.time
    from: "12:39"
    to: "12:40"
condition: []
action:
  - service: logbook.log
    data:
      name: Automation punctuality notification
      message: >-
        Delay_check_SENSOR-BASED activated at {{
        now().strftime('%H:%M:%S') }}
mode: restart

I created 8 of these automations, at different times during the 24h and, until now, they started just 1 or 2 seconds late, while the time-triggered ones took from 2" to 16’59"

Take a look to this log

date	automation	scheduled time	actual time of execution	delay	delay delta
22/01/2024	delay check time based	17:40	17:44:26	04:26	00:00
22/01/2024	delay check time based	18:40	18:47:18	07:18	02:52
22/01/2024	delay check time based	19:40	19:50:22	10:22	03:04
22/01/2024	delay check time based	20:40	20:40:57	00:57	-09:25
22/01/2024	delay check time based	21:40	21:44:17	04:17	03:20
22/01/2024	delay check time based	22:40	22:47:47	07:47	03:30
22/01/2024	delay check time based	23:40	23:48:20	08:20	00:33
22/01/2024	delay check time based	00:00	00:08:20	08:20	00:00
22/01/2024	delay check time based	00:35	00:43:20	08:20	00:00
22/01/2024	delay check time based	00:40	00:48:20	08:20	00:00
22/01/2024	delay check SENSOR BASED	00:40	00:40:00	00:00
22/01/2024	host shutdown	00:40	00:48:20	08:20	00:00
23/01/2024	host reboot	00:58	00:58:39	00:00	n/a
23/01/2024	delay check time based	01:00	01:00:01	00:01
23/01/2024	delay check time based	01:40	01:40:02	00:02	00:01
23/01/2024	delay check time based	02:40	02:40:44	00:44	00:42
23/01/2024	delay check time based	03:40	03:41:43	01:43	00:59
23/01/2024	delay check time based	04:40	04:42:50	02:50	01:07
23/01/2024	delay check SENSOR BASED	04:40	04:40:01	00:01
23/01/2024	delay check time based	05:40	05:44:06	04:06	01:16
23/01/2024	delay check time based	06:40	06:45:30	05:30	01:24
23/01/2024	delay check time based	07:40	07:47:03	07:03	01:33
23/01/2024	delay check time based	08:40	08:48:45	08:45	01:42
23/01/2024	delay check SENSOR BASED	08:40	08:40:01	00:01
23/01/2024	delay check time based	09:40	09:50:35	10:35	01:50
23/01/2024	delay check time based	10:40	10:52:34	12:34	01:59
23/01/2024	delay check time based	11:40	11:54:43	14:43	02:09
23/01/2024	delay check time based	12:40	12:56:59	16:59	02:16
23/01/2024	delay check SENSOR BASED	12:40	12:40:02	00:02
23/01/2024	delay check time based	13:40	13:59:25	19:25	02:26

starting from the reboot task scheduled (externally) at 00:58, the delay is increasing as the time is passing by, but the sensor-based automations are delaying just a few.

This means that it cannnot be a time drift (I refer to ‘paravirtualization’) due to the VM… or am I wrong?

The test is still going on: I will update you all later.

Deimos76 · January 24, 2024, 9:45am

Good morning, all

let’s share some updates

Confirmed but it seems that the delay increases as long as time passes by the last reboot, which resets the delay

Confirmed

Still confirmed without evidence

CONFIRMED!!!

new log (covering 24+hs)

date	automation	scheduled time	actual time of execution	delay	delay delta
22/01/2024	delay check time based	17:40	17:44:26	04:26
22/01/2024	delay check time based	18:40	18:47:18	07:18	02:52
22/01/2024	delay check time based	19:40	19:50:22	10:22	03:04
22/01/2024	delay check time based	20:40	20:40:57	00:57	-09:25
22/01/2024	delay check time based	21:40	21:44:17	04:17	03:20
22/01/2024	delay check time based	22:40	22:47:47	07:47	03:30
22/01/2024	delay check time based	23:40	23:48:20	08:20	00:33
22/01/2024	delay check time based	00:00	00:08:20	08:20	00:00
22/01/2024	delay check SENSOR BASED	00:40	00:40:00	00:00
22/01/2024	delay check time based	00:35	00:43:20	08:20	00:00
22/01/2024	delay check time based	00:40	00:48:20	08:20	00:00
22/01/2024	host shutdown	00:40	00:48:20	08:20	00:00
23/01/2024	host reboot	00:58	00:58:39	n/a	n/a
23/01/2024	delay check time based	01:00	01:00:01	00:01	-08:19
23/01/2024	delay check time based	01:40	01:40:02	00:02	00:01
23/01/2024	delay check time based	02:40	02:40:44	00:44	00:42
23/01/2024	delay check time based	03:40	03:41:43	01:43	00:59
23/01/2024	delay check SENSOR BASED	04:40	04:40:01	00:01	00:01
23/01/2024	delay check time based	04:40	04:42:50	02:50	01:07
23/01/2024	delay check time based	05:40	05:44:06	04:06	01:16
23/01/2024	delay check time based	06:40	06:45:30	05:30	01:24
23/01/2024	delay check time based	07:40	07:47:03	07:03	01:33
23/01/2024	delay check SENSOR BASED	08:40	08:40:01	00:01	00:00
23/01/2024	delay check time based	08:40	08:48:45	08:45	01:42
23/01/2024	delay check time based	09:40	09:50:35	10:35	01:50
23/01/2024	delay check time based	10:40	10:52:34	12:34	01:59
23/01/2024	delay check time based	11:40	11:54:43	14:43	02:09
23/01/2024	delay check SENSOR BASED	12:40	12:40:02	00:02	00:01
23/01/2024	delay check time based	12:40	12:56:59	16:59	02:16
23/01/2024	delay check time based	13:40	13:59:25	19:25	02:26
23/01/2024	delay check time based	14:40	15:02:00	22:00	02:35
23/01/2024	delay check time based	15:40	16:04:45	24:45	02:45
23/01/2024	delay check SENSOR BASED	16:40	16:40:02	00:02	00:00
23/01/2024	delay check time based	16:40	17:07:39	27:39	02:54
23/01/2024	delay check time based	17:40	18:10:43	30:43	03:04
23/01/2024	delay check time based	18:40	19:13:53	33:53	03:10
23/01/2024	delay check time based	19:40	20:17:17	37:17	03:24
23/01/2024	delay check SENSOR BASED	20:40	20:40:04	00:04	00:02
23/01/2024	delay check time based	20:40	21:20:49	40:49	03:32
23/01/2024	delay check time based	21:40	22:24:27	44:27	03:38
23/01/2024	delay check time based	22:40	23:26:27	46:27	02:00
23/01/2024	delay check time based	23:40	00:26:27	46:27	00:00
24/01/2024	delay check SENSOR BASED	00:40	00:40:00	00:00	-00:04
24/01/2024	delay check time based	00:00	00:46:27	46:27	00:00
24/01/2024	host reboot	00:58	00:58:33	n/a	n/a
24/01/2024	delay check time based	01:00	01:00:00	00:00	-46:27
24/01/2024	delay check time based	01:40	01:40:00	00:00	00:00
24/01/2024	delay check time based	02:40	02:40:00	00:00	00:00
24/01/2024	delay check time based	03:40	03:40:00	00:00	00:00
24/01/2024	delay check time based	04:40	04:40:00	00:00	00:00
24/01/2024	delay check SENSOR BASED	04:40	04:40:00	00:00	00:00
24/01/2024	delay check time based	05:40	05:40:00	00:00	00:00
24/01/2024	delay check time based	06:40	06:40:45	00:45	00:45
24/01/2024	delay check time based	07:40	07:41:44	01:44	00:59
24/01/2024	delay check SENSOR BASED	08:40	08:40:01	00:01	00:01
24/01/2024	delay check time based	08:40	08:42:51	02:51	01:07
24/01/2024	delay check time based	09:40	09:44:07	04:07	01:16

Confirmed - weird thing: today the first delay occurred 5hs later than yesterday

Yesterday I copied out all these automations into a cloned VM running on different phisical hardware (mini-itx equipped with a Intel Core i5 CPU 4590T 2.00GHz with 8Gb RAM and 250Gb SSD) and network and the result is terrifying: until now, all occurred perfectly on time.

[2024-01-25T18:59:00Z] UPDATE

Dear friends
(even if it seems that I am talking to none )

today I tried:

to switch off all the application running on the phisical machine but the VM that contains HA → no result
to deselect the "Virtualize Intel VT-x/EPT or AMD-v/RVI option in VMware → no result
to find any other clue about what’s going on in HA or VM while the automation should be starting → found a clue!!!
lots of these messages in the Host Log:

Looking for these in google, unfortunately we come back to Paravirtualization VirtualBox problem with HA:

Time is fubar, any suggestions (not Time Zone or Location)

Unfortunately, there is no “Paravirtualization Interface” setting in VMware to be switched off.

…but I am not convinced at all about the connection between this Para-thing and the automations triggering late. I mean:

if there is a time drift, why should it affect just the automation trigger and not the clock itself?
why a clone of the VM, played on different hardware, is not affected by the same defect?

…here is why now I am trying deselecting an option that, in my mind, should obviously be selected

all my finger are crossed.

[2024-01-26T08:44:00Z] VERY LAST UPDATE

I think we did it
deselcting the VMware option about synchronizing guest time with host did the job and, since that, until now, all the automations triggered perfectly in time.

even if several details have to be clarified, if this solution is confirmed by other users, should they think adding this detail to the HA on VMWare installation guide?

[2024-02-15T14:02:00Z]
still monitoring → still working perfectly in time

exx · February 15, 2024, 2:10pm

After reading the above, the only conclusion I can come to is that HA maintains two different “clocks”: One that it shows to users, and one that it uses to trigger automations.

If this is correct, I fail to see how it is a good idea.

WallyR · February 15, 2024, 4:09pm

No, HA only have the time provided by the hypervisor running on the host OS.
The problem is that this time is disconnected from the hardware time on the host OS.
The RTC HA sees is a virtual RTC and this one will time skew when the Host OS gets bogged down with other tasks, like installing Windows updates.

The integration packages usually include a SNTP or better yet a NTP client to set the time.
HA looks like it has the integration package for the QEMU hypervisor installed, so on those systems the time skew should be minimal, but for other hypervisors, there seems to be no integrations package included (which might be due to license/copyright limitations).

HA is using a SNTP client, which is required on RPis, because they have no RTC onboard, but RPis usually run HA directly or as a container, which makes the time skew very small.
The SNTP will also save some situations on a hypervisor, but the time skew on a hypervisor can be big and fluctuating, due to tasks on the host OS. A NTP client could solve it, but require multiple connections to time servers on the internet.

When talking about time in relation to VMs articles seems to differentiate between drift and time skew.
Drift is a near constant factor that makes the clock go too slow or too fast. Time skew on the other hand can go both ways and vary in degree from event to event.

In the article below there on the difference between NTP and SNTP there is an explaination of how SNTP works, which makes it clear that your bug hunting can be blurred by the fact that a SNTP client have updated the time since your problematic event and thereby removed the evidence of the time skew.

exx · February 15, 2024, 10:05pm

Except that what you’re saying doesn’t make any sense. These are several posts above about how HA shows the correct time, and yet the automations fail to execute at the correct time. The only way this is possible is if HA maintains two distinct clocks. Otherwise, when the clock strikes 12, X should happen - and it’s not.

Further, at least one of the prior posts references using vSphere as the host hypervisor, not Windows. vSphere is an enterprise-class product, and unless your %CPUREADY values are OBSCENE (in excess of 25-30%), you are not going to notice even a TINY bit of clock issue - vmware tools or not. VMware tools don’t make a BIT of difference in terms of clock synchronization, since vSphere is able to properly and correctly emulate that hardware and abstract it from the guest operating system. We aren’t talking about virtualbox on a Windows 10 host here - at least not for all of the issues that were reported above. If you have the skills to configure shared storage, and vMotion, and vSphere, and vCenter, and the ability to buy all that stuff, AND you had %CPUREADY issues even approaching anything that high, there is no possible chance that performance issues in every single one of the other VMs on that host would go unnoticed.

WallyR · February 16, 2024, 7:20am

https://kb.vmware.com/s/article/1006427

Deimos76 · February 16, 2024, 1:55pm

Hi, there,

I am not a geek (and please excuse me for that, reading the following) but I think that you both are a half right and wrong.

if the clock of the VM ran too slow or too fast, I expected the “delay delta” coulmn in my reports to be always almost the same - which it is not.
I actually do not know whether HA has two clocks but it is weird that the clock that I can show in the frontend AND the one taken as a reference for the history recorder are perfectly on time, while just the automation is starting late (I would have expected a history which shows events always perfectly in time but the actual events starting late versus the time that I can see on my watch, outside the VM). I think that the “triggering” is happening at the right time (or it does not make any sense in my mind that a computer is triggering at 22:47:27 intead of 21:40:00) but what is going on during the delay time? …and why changing the triggering method (the value of time instead the time itself) lowers to 4 seconds a delay already grown to 41 minutes?
we can see something’s wrong in the Host Log: it seems that VMWare is correcting the time of the hosted system with a frequency of 34" (?!?), causing, at the very end, a non-linear delay…but why? even if this correction is made on a puctual clock, it would not cause any problem but a small waste of resources… wouldn’t it?

exx · February 17, 2024, 2:14am

Do you have a point you are attempting to make, or are you simply googling support docs?

Do you have hands-on experience with vSphere?

exx · February 17, 2024, 2:20am

I haven’t got a ton of experience with VMware workstation with this type of workload, as this isn’t really what it’s designed for. I’ve used it on company machines for years as a way to have a clean environment to connect to customer systems using their VPN client of choice, but I only ever run one VM at a time, and when I’m finished with it, I shut it down, so… I don’t have much in the way of direct advice re: your particular installation - aside from to say that I scrolled back up, and I think your host system specs may be a bit on the anemic side. How many VMs are you running, and what else is the host doing aside from running VMware workstation?

WallyR · February 17, 2024, 10:18am

I have little experience with vSphere, but I have experience with other hypervisors and the issue is a general one.
Even if I had experience with vSphere it would just be my word against yours, so I found it better to point you to the VMWare article regarding the issue on their products, which explains it pretty well.

exx · February 17, 2024, 2:40pm

Ok, we’re done here. LOL

Deimos76 · February 26, 2024, 5:12pm

Hi exx,

I am running one VM per physical machine; this means that I own two physical PC and each of them is running just one VM.

one is running a P2P client and a http file server, besides the HA VM. The main activity is HA because the two others are are not working almost at all.
the other phisical machine has just the HA VM and not any other software running. The VM running here is a clone of the first one

did you take a look to the logs I posted right here a few days ago?