Apple Watch not responding with Homekit

I added request logging for HomeKit by adding this to my config.

logger:
  default: warning
  logs:
    homeassistant.components.homekit: debug
    pyhap: debug

Interestingly, my Apple Watch seems to only communicate with Home Assistant’s HomeKit via the AppleTV (my home hub) I think that’s probably a clue. Looks like a others were reporting issues with the Apple Watch and AppleTV starting last September:

If I disable my Apple TV as the home hub then HomeKit changes made on my watch come from my phone’s IP even though the watch is on WiFi. Not sure if this is normal or not. If I disconnect my phone from WiFi, HomeKit no longer works on the watch. Again not sure if that’s normal or not.