I too had a similar problem, but my problems seems to go way/hidden if I restart HA,
all I do now is schedule a restart of Hass every 2 days … just don’t know what the probelm is.
Did you manage to fix the issue. I am having the same exact issue. Restarting HA is the only way but it is totally annoying.
Any progress?