I’m seeing this too. All my matter devices connected to my OTBR go offline every 6 hours too. The system is an HA Yellow.
This is exactly what a classic memory leak looks like.
And it’s way more precise than a memory leak. It’s goes down precisely every 6 hours.
Though sometimes recovery onto the network takes extra time for some devices.
I checked and I don’t even have the watchdog turned on and the memory is barely in use. With CPU addon usage at 1.3% and Memory at 0.1%
My system has about 90 nodes and the behavior is specific to the 75 or so nodes that are on the HA Yellow Open Border Router.
I have ~15 nodes on a HomePod in a detached structure which is outside of radio range and they don’t go offline with this cadence. And as I’ve read about different thread routers colliding I tried unplugging the homepod for a night taking it and all the devices connected to it offline and there were still outages on schedule while OTBR was the only active thread network.
In the thread border router logs I have found that the event starts with two failures to send then thousands of lines of dropped packets
21d.23:04:52.980 [N] MeshForwarder-: Failed to send IPv6 UDP msg, len:244, chksum:b3b8, ecn:no, to:0x4000, sec:yes, error:NoAck, prio:low, radio:15.4
21d.23:04:52.980 [N] MeshForwarder-: src:[xxxx:xxxx:xxxx:x:xxxx:xxxx:xxxx:xxxx]:33004
21d.23:04:52.980 [N] MeshForwarder-: dst:[yyyy:yyyy:yyyy:y:yyyy:yyyy:yyyy:yyyy]:5540
21d.23:04:53.172 [N] MeshForwarder-: Failed to send IPv6 UDP msg, len:90, chksum:c351, ecn:no, to:0xd800, sec:yes, error:NoAck, prio:low, radio:15.4
21d.23:04:52.980 [N] MeshForwarder-: src:[xxxx:xxxx:xxxx:x:xxxx:xxxx:xxxx:xxxx]:33004
21d.23:04:52.980 [N] MeshForwarder-: dst:[zzzz:zzzz:zzzz:z:zzzz:zzzz:zzzz:zzzz]:5540
21d.23:04:53.194 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0xd90b, dst:0x8c00, sec:yes, tag:63326, offset:1080, dglen:1245
21d.23:04:53.258 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:77, src:0xd90b, dst:0x8c00, sec:yes, tag:63326, offset:1168, dglen:1245
21d.23:04:53.465 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0xd90b, dst:0x8c00, sec:yes, tag:63327, offset:112, dglen:1249
21d.23:04:53.472 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0xd90b, dst:0x8c00, sec:yes, tag:63327, offset:200, dglen:1249
21d.23:04:53.517 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0xe841, dst:0x8c00, sec:yes, tag:42307, offset:112, dglen:1245
21d.23:04:53.593 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0xe841, dst:0x8c00, sec:yes, tag:42307, offset:200, dglen:1245
It goes on for a few minutes and then just stops
21d.23:06:41.815 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0x2842, dst:0x8c00, sec:yes, tag:17996, offset:288, dglen:685
21d.23:06:41.979 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0x2842, dst:0x8c00, sec:yes, tag:17996, offset:464, dglen:685
21d.23:06:41.988 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:21, src:0x2843, dst:0x8c00, sec:yes, tag:51903, offset:112, dglen:133
21d.23:06:42.009 [N] MeshForwarder-: Dropping rx frag frame, error:Drop, len:88, src:0x2842, dst:0x8c00, sec:yes, tag:17996, offset:552, dglen:685
There are a few other errors mixed in. It appears that approximately once a second there’s this one
21d.23:06:40.720 [N] MeshForwarder-: Dropping (reassembly queue) IPv6 UDP msg, len:1251, chksum:2724, ecn:no, sec:yes, error:ReassemblyTimeout, prio:normal, rss:-85.5, radio:15.4
There are more Failed to Send errors interspersed the high rates of dropping
And I found
21d.23:06:22.907 [W] P-RadioSpinel-: Handle transmit done failed: ChannelAccessFailure near some of the repeated failures to send.
And occasionally
21d.23:05:44.092 [N] MeshForwarder-: Dropping IPv6 UDP msg, len:282, chksum:a0db, ecn:no, sec:yes, error:Drop, prio:low, radio:all
There was another thread here: Channel Access Failure - #23 by agners which suggests that the channel access failure implies radio interference.
I"ve included the first and last errors and it takes about 2 minutes of many errors and then drops back to nominal operations.
With my large network of integrated switches I’m very hesitant to jump channels if it might not transition everything. And that doesn’t get to anything that makes sense as to an underlying issue that was resolved.