Modbus issue

I have an issue where if the modbus server (the device im talking to) goes offline for longer than the scan period, HA never recovers. I actually had to reboot, restarting HA did not resolve.