Nabu Casa Slow when phone is on Local Network

Hi all,
I sometimes have an issue where HA on phones (both iPhone and Android) is running really slow when the phone is on the local network (It can take several seconds to load a dashboard). It runs much faster when a phone is remote and using Nabu Casa to connect remotely.
I tried temporarily disabling Nabu Casa (By turning off Settings → HA Cloud → “Remote Control”). The result was that fixed it (No more lag). So perhaps for some reason HA is still going out to Nabu Casa servers and then back to HA instead of direct to HA (even though the phone is on the local network)?
Any ideas on where I could go from here to troubleshoot further?