Routes break on reboot - internet connection lost (HomeAssistant supervised, raspberry pi 4B, Debian 12 bookworm)

I have strange routing behaviour, which started when I upgraded to Debian 12.
The upgrade showed no errors at all, but on rebooting, I found I had lost internet connection.

I’m hoping those with better networking skills than me can give me some ideas where to look for the problem.

After a few hours of debugging I eventually noticed that the routing table looked strange.
One of the routes showed a gateway configuration (net 0.0.0.0 gw 0.0.0.0 netmask 0.0.0.0) set up through a bridge connection to one of the containers (I’m guessing these are what the vethHHHHH links are)…

Not only that, I also noticed that if I waited a few minutes and checked the routes again, that problematic gateway route was pointing to a different container bridge link.

Once it settled though, I tested that the internet was not accessible (via ping) and that HomeAssistant had started properly (though anything that required internet access was down).

I then manually removed the questionable route… and the connection to the internet was reestablished. I restarted HomeAssistant (within HomeAssistant) and it had full connectivity.

To test that this is repeatable, I rebooted the host, and sure enough, the same thing happened and I had to manually find and delete the problematic route to restore the internet connection.

What I don’t understand is why this is happening (what is causing it) and how to stop it from happening (otherwise any unexpected black out and it will be off line until I find out and manage to manually fix it).

… any ideas are very welcome.

Regards, Laz

Please see below logs to show what I mean:

pi@homeassistant:~ $ sudo shutdown -r now

Broadcast message from root@homeassistant on pts/1 (Mon 2023-08-07 20:32:09 ACST):

The system will reboot now!

pi@homeassistant:~ $ Connection to homeassistant.local closed by remote host.
Connection to homeassistant.local closed.

laz@HPVictus:~$ ssh homeassistant

Enter passphrase for key '/home/laz/.ssh/id_rsa':
Linux homeassistant 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Aug  7 19:25:30 2023 from 192.168.65.179

Wi-Fi is currently blocked by rfkill.
Use raspi-config to set the country before use.

pi@homeassistant:~ $ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethd48f6c4
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethb17aadd
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth259e857
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethac2ace6
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth6152b53
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethf14e8f4
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethae49af6
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethc06161c
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth2e12f5d
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethf5e4770
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth2f227a3
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 vethd48f6c4
0.0.0.0         192.168.65.50   0.0.0.0         UG    0      0        0 eth0
0.0.0.0         192.168.65.50   0.0.0.0         UG    100    0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth2f227a3
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethf5e4770
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth2e12f5d
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethc06161c
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethae49af6
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethf14e8f4
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth6152b53
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethac2ace6
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth259e857
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethb17aadd
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethd48f6c4
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.30.32.0     0.0.0.0         255.255.254.0   U     0      0        0 hassio
192.168.65.0    0.0.0.0         255.255.255.0   U     100    0        0 eth0

pi@homeassistant:~ $ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethd3baa60
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth703df21
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethbf9a2b9
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethbbd6d11
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethd48f6c4
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethb17aadd
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth259e857
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethac2ace6
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth6152b53
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethf14e8f4
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethae49af6
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethc06161c
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth2e12f5d
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethf5e4770
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth2f227a3
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 vethd3baa60
0.0.0.0         192.168.65.50   0.0.0.0         UG    0      0        0 eth0
0.0.0.0         192.168.65.50   0.0.0.0         UG    100    0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth2f227a3
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethf5e4770
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth2e12f5d
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethc06161c
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethae49af6
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethf14e8f4
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth6152b53
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethac2ace6
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth259e857
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethb17aadd
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethd48f6c4
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethbbd6d11
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethbf9a2b9
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth703df21
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethd3baa60
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.30.32.0     0.0.0.0         255.255.254.0   U     0      0        0 hassio
192.168.65.0    0.0.0.0         255.255.255.0   U     100    0        0 eth0

pi@homeassistant:~ $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 169.254.165.57 icmp_seq=1 Destination Host Unreachable
From 169.254.165.57 icmp_seq=2 Destination Host Unreachable
From 169.254.165.57 icmp_seq=3 Destination Host Unreachable
From 169.254.165.57 icmp_seq=4 Destination Host Unreachable
ping: sendmsg: No route to host
From 169.254.165.57 icmp_seq=5 Destination Host Unreachable
From 169.254.165.57 icmp_seq=6 Destination Host Unreachable
^C
--- 8.8.8.8 ping statistics ---
10 packets transmitted, 0 received, +6 errors, 100% packet loss, time 9202ms
pipe 4

pi@homeassistant:~ $ sudo route del -net 0.0.0.0 gw 0.0.0.0 netmask 0.0.0.0 dev vethd3baa60

pi@homeassistant:~ $ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethd3baa60
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth703df21
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethbf9a2b9
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethbbd6d11
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethd48f6c4
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethb17aadd
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth259e857
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethac2ace6
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth6152b53
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethf14e8f4
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethae49af6
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethc06161c
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth2e12f5d
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 vethf5e4770
0.0.0.0         0.0.0.0         255.255.255.255 UH    0      0        0 veth2f227a3
0.0.0.0         192.168.65.50   0.0.0.0         UG    0      0        0 eth0
0.0.0.0         192.168.65.50   0.0.0.0         UG    100    0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth2f227a3
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethf5e4770
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth2e12f5d
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethc06161c
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethae49af6
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethf14e8f4
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth6152b53
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethac2ace6
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth259e857
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethb17aadd
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethd48f6c4
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethbbd6d11
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethbf9a2b9
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 veth703df21
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vethd3baa60
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.30.32.0     0.0.0.0         255.255.254.0   U     0      0        0 hassio
192.168.65.0    0.0.0.0         255.255.255.0   U     100    0        0 eth0

pi@homeassistant:~ $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=18.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=19.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=19.0 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=58 time=19.6 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4005ms
rtt min/avg/max/mdev = 18.826/19.267/19.636/0.359 ms