Since I've started the local OTBR it's working stable. So it has something to do with communication to the Apple BR.
Unfortunately with local OTBR it's no good solution because OTBR and Apple BR leads to really bad performance when running both in parallel.
I know this might be a non-answer, but I've noticed a HUGE difference in performance with my 102 devices since moving to the Matter.js beta. I know it's a beta, but it's been a gamechanger in how reliable it is compared to the python Matter server.
I have 6 Apple TBRs and 102 Matter/Thread devices and I would have outages of devices all the time. After moving to the beta, I have no device drops for days or weeks. And when I have drops, it's for maybe 1 or 2 seconds and that's it.
It's worth trying the beta if you are still on the Python server. There have been many iterations of the matter.js server so far, so it's not like you are running something brand new.
I already use the Matter Beta. But the problem happens with beta and with the official version.
The problem is that Home Assistant can‘t reach the ipV6 network subsegment of Thread anymore. And it started with any update. So it seems to be a bug.
But I also seen that the beta runs in principle better.
Just some thoughts ... Its really up to the Apple HomePod Mini to issue Router Advertisements containing the routes to the Thread network. These routes will have certain lifetimes to them. Similarly, the Mini has to issue new routes if the Thread mesh routeable prefixes change, etc. So perhaps check out the route lifetimes seen by HA to see if they are the cause.
The only thing that comes to mind here, is that HA may be issuing a Router Solicitation after it restarts or update command is issued. This would request the Mini to send out a Router Advertisement which would contain the routes to the mesh.
I may have found another solution that helped in my setup.
I had almost identical symptoms (Apple Home always worked, Home Assistant reported address is unreachable for the Thread IPv6 addresses). My setup has Home Assistant on one VLAN and my Apple TV Thread Border Router on another.
What fixed it for me was adding a second network interface to Home Assistant on the Apple TV VLAN with IPv4 disabled and IPv6 set to Automatic. My primary NIC stayed on my server VLAN. As soon as HA had a directly attached IPv6 interface on the TBR’s L2 network, Matter commissioning and communication became stable without affecting my IPv4 routing.
It may not be the same root cause in your case, but given the “no IPv6 route to Thread subnet” symptoms, it might be worth trying.