Struggling with SNTP

I have this issue with SNTP component with esphome.

For some reason, it never attempt a hourly time sync to fix time drift. So far this sntp stuff only works at boot. That’s it, it never sync after that leading my matrix clock months of uptime with more than 30 seconds behind the actual time.

My NTP Server is running over pfsense NTP service, I set it up in a way that it uses a GPS with PPS as its time reference. Everything is working great on the server side of things.

Funnily enough as I debug things, I set the ntp server to pool.ntp.org and whoala it now syncs the time just fine, what gives? If I set it back to my local NTP server, it wont sync in hourly basis, but it only sync the time at boot and thats it. As a workaround, I placed a NAT firewall rule to intercept pool.ntp.org traffic back to my local ntp server and that worked.

I tried everything, setting static IP on that clock on the yaml code, I used a local IP address instead of a hostname of my ntp server, bang my head on it. I dont know. Why esphome hates my local ntp server. Everything else in my network use that ntp server just fine, tasmota had no problems using it (heck I even go as far as disable KoD because how needy those device asking what time is it).