I have strange routing behaviour, which started when I upgraded to Debian 12.
The upgrade showed no errors at all, but on rebooting, I found I had lost internet connection.
I’m hoping those with better networking skills than me can give me some ideas where to look for the problem.
After a few hours of debugging I eventually noticed that the routing table looked strange.
One of the routes showed a gateway configuration (net 0.0.0.0 gw 0.0.0.0 netmask 0.0.0.0) set up through a bridge connection to one of the containers (I’m guessing these are what the vethHHHHH links are)…
Not only that, I also noticed that if I waited a few minutes and checked the routes again, that problematic gateway route was pointing to a different container bridge link.
Once it settled though, I tested that the internet was not accessible (via ping) and that HomeAssistant had started properly (though anything that required internet access was down).
I then manually removed the questionable route… and the connection to the internet was reestablished. I restarted HomeAssistant (within HomeAssistant) and it had full connectivity.
To test that this is repeatable, I rebooted the host, and sure enough, the same thing happened and I had to manually find and delete the problematic route to restore the internet connection.
What I don’t understand is why this is happening (what is causing it) and how to stop it from happening (otherwise any unexpected black out and it will be off line until I find out and manage to manually fix it).
… any ideas are very welcome.
Regards, Laz
Please see below logs to show what I mean:
pi@homeassistant:~ $ sudo shutdown -r now
Broadcast message from root@homeassistant on pts/1 (Mon 2023-08-07 20:32:09 ACST):
The system will reboot now!
pi@homeassistant:~ $ Connection to homeassistant.local closed by remote host.
Connection to homeassistant.local closed.
laz@HPVictus:~$ ssh homeassistant
Enter passphrase for key '/home/laz/.ssh/id_rsa':
Linux homeassistant 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Aug 7 19:25:30 2023 from 192.168.65.179
Wi-Fi is currently blocked by rfkill.
Use raspi-config to set the country before use.
pi@homeassistant:~ $ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethd48f6c4
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethb17aadd
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth259e857
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethac2ace6
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth6152b53
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethf14e8f4
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethae49af6
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethc06161c
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth2e12f5d
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethf5e4770
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth2f227a3
0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 vethd48f6c4
0.0.0.0 192.168.65.50 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.65.50 0.0.0.0 UG 100 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth2f227a3
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethf5e4770
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth2e12f5d
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethc06161c
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethae49af6
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethf14e8f4
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth6152b53
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethac2ace6
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth259e857
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethb17aadd
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethd48f6c4
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.32.0 0.0.0.0 255.255.254.0 U 0 0 0 hassio
192.168.65.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
pi@homeassistant:~ $ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethd3baa60
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth703df21
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethbf9a2b9
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethbbd6d11
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethd48f6c4
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethb17aadd
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth259e857
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethac2ace6
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth6152b53
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethf14e8f4
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethae49af6
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethc06161c
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth2e12f5d
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethf5e4770
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth2f227a3
0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 vethd3baa60
0.0.0.0 192.168.65.50 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.65.50 0.0.0.0 UG 100 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth2f227a3
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethf5e4770
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth2e12f5d
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethc06161c
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethae49af6
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethf14e8f4
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth6152b53
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethac2ace6
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth259e857
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethb17aadd
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethd48f6c4
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethbbd6d11
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethbf9a2b9
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth703df21
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethd3baa60
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.32.0 0.0.0.0 255.255.254.0 U 0 0 0 hassio
192.168.65.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
pi@homeassistant:~ $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 169.254.165.57 icmp_seq=1 Destination Host Unreachable
From 169.254.165.57 icmp_seq=2 Destination Host Unreachable
From 169.254.165.57 icmp_seq=3 Destination Host Unreachable
From 169.254.165.57 icmp_seq=4 Destination Host Unreachable
ping: sendmsg: No route to host
From 169.254.165.57 icmp_seq=5 Destination Host Unreachable
From 169.254.165.57 icmp_seq=6 Destination Host Unreachable
^C
--- 8.8.8.8 ping statistics ---
10 packets transmitted, 0 received, +6 errors, 100% packet loss, time 9202ms
pipe 4
pi@homeassistant:~ $ sudo route del -net 0.0.0.0 gw 0.0.0.0 netmask 0.0.0.0 dev vethd3baa60
pi@homeassistant:~ $ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethd3baa60
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth703df21
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethbf9a2b9
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethbbd6d11
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethd48f6c4
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethb17aadd
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth259e857
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethac2ace6
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth6152b53
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethf14e8f4
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethae49af6
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethc06161c
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth2e12f5d
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 vethf5e4770
0.0.0.0 0.0.0.0 255.255.255.255 UH 0 0 0 veth2f227a3
0.0.0.0 192.168.65.50 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.65.50 0.0.0.0 UG 100 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth2f227a3
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethf5e4770
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth2e12f5d
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethc06161c
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethae49af6
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethf14e8f4
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth6152b53
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethac2ace6
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth259e857
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethb17aadd
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethd48f6c4
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethbbd6d11
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethbf9a2b9
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 veth703df21
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vethd3baa60
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.32.0 0.0.0.0 255.255.254.0 U 0 0 0 hassio
192.168.65.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
pi@homeassistant:~ $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=18.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=19.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=19.0 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=58 time=19.6 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4005ms
rtt min/avg/max/mdev = 18.826/19.267/19.636/0.359 ms