The issue is that HA as standard use a SNTP client, which means it updates with intervals of some hours and between those points the time can drift, with the biggest difference occurring just before an update happens.
Having your own NTP server and setting it up in HA will not change this.
The way to prevent it is using a NTP client instead.
If you are using HAOS or HA supervised, then you can install the Chrony addon.
If you are using another installation type, then Chrony might be available too and probably also other NTP clients.
A NTP client will run all the time and gather time data from several NTP servers, which it is using to get a more precise clock and it then adjust the system clock. This means the system clock will be adjusted all the time with small micro adjustment, instead of the SNTP clients fewer bigger adjustments. The drawback is that the NTP client use some extra RAM, CPU and network traffic on it.
Especially VMs can need a NTP client, because the hardware clock is not available in the VM and a VM can be paused and slowed down by the supervisor, so the clock can easily drift.