Two DIRIGERAS as TBRs Problems

I was using a single DIRIGERA D1 as a TBR with a HA Matter server as controller. I also have some devices joined to the Matter controller on D1 in addition to HA. This works fine, more stable than the HA OTBR.

As I plan to add a lot of devices, I decided to add a 2nd DIRIGERA D2 in a different corner of the home and joined it to the same Thread network (via the standard Home Smart IKEA way). The idea being that they should be good at handling Thread 1.4 and TREL. No controller functionality is used on D2, I only did it for the TBR.

Now, if both TBRs are on, I only see around half of my devices in the HA server. I do see all the ones I orginally connected in other controller apps (Home Smart, Nanoleaf), even when not seeing them in HA anymore. As soon as I switch off D2 again, the nodes appear in HA again gradually after a minute or so.

Does anyone has mutliple DIRIGERAs as TBRs working with HA Matter server? Or any other ideas?