With OTBR and IOT devices in the same broadcast domain - it may work as directed traffic between HA and OTBR over a layer 3 boundary shouldn’t be a problem - Although I don’t fully understand the connection distribution between HA and the Matter devices as yet - early days for me on this.
If there is a layer boundary between OTBR and the IoT device - No it won’t work, due to the below.
A few more details:
mDNS do not like or work very well with layer 3 gateways i.e. routers between VLAN’s.
Enabling Bonjour, mDNS forwarding or reflectors, will not solve the VLAN isolation problems for mDNS, it needs to be work 100% of the time for 100% of traffic.
For Thread to really work all the IoT (HA, Lights, Sensors etc.) needs to be in the same VLAN / Broadcast domain.
Layer 2 and layer 3 routing should not really matter, because mDNS is not routable with normal IP routing anyway.
Rwflectors should be able to make it work, but some of them may lack functions or features or the user do not know how to set them up properly.
Unless you are a network master and understand both the protocols, the configuration of the network devices and the end devices requirement fully, then VLANs are not advisable.
Your suggestion to gather both devices, HA and Matter in the same one is the only viably VLAn setup for the vast majority.
As Wally mentioned, you’ll need to add Matter Server to the IOT VLAN as the Matter Server is also using mDNS. One other thing to figure out is that the OTBR, AFAIK, can’t be setup for static IPv6. It will send out IPv6 Router Advertisements that include both a prefix for the IOT VLAN to use and a route to reach the Thread devices, so the system running the Matter Server and will need to be setup to accept and process these RAs appropriately.