Startup failure after 2024.03.0 upgrade ... dns issues?

[plord (Peter Lord)]

I’ve just upgrade to 2024.03.0 and found although home assistant starts, after a short time cpu usage goes to 100% and then reboots.

I’m running -

# ha info
arch: aarch64
channel: stable
docker: 24.0.7
features:
- reboot
- shutdown
- services
- network
- hostname
- timedate
- os_agent
- haos
- resolved
- journal
- disk
- mount
hassos: "12.0"
homeassistant: 2024.3.0
hostname: arm1
logging: info
machine: odroid-c2
operating_system: Home Assistant OS 12.0
state: startup
supervisor: 2024.03.0
supported: true
supported_arch:
- aarch64
- armv7
- armhf
timezone: Europe/London

Now one thing I noticed (just before the server rebooted itself) is very high cpu usage for dns and adguard containers -

# docker stats
CONTAINER ID   NAME                                        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
c46026c31e53   hassio_multicast                            0.13%     1.492MiB / 1.822GiB   0.08%     0B / 0B           6.57MB / 295kB    8
161b406ce295   hassio_audio                                0.00%     12.62MiB / 1.822GiB   0.68%     238kB / 0B        25.3MB / 389kB    12
05df21d2f21c   hassio_dns                                  103.10%   470.2MiB / 1.822GiB   25.20%    7.65MB / 7.11MB   31.4MB / 311kB    24
890f15f4f911   hassio_cli                                  0.00%     1.75MiB / 1.822GiB    0.09%     247kB / 8.54kB    12.9MB / 299kB    9
fb5b9bd2cacc   hassio_supervisor                           10.11%    79.68MiB / 1.822GiB   4.27%     5.38MB / 5.85MB   113MB / 1.12MB    26
e69a4ee73a04   homeassistant                               9.48%     389.1MiB / 1.822GiB   20.85%    0B / 0B           355MB / 19.6MB    36
7f3cba1db195   hassio_observer                             0.00%     1.188MiB / 1.822GiB   0.06%     239kB / 0B        6.92MB / 4.1kB    5
ec75cfe9fcf2   addon_a0d7b954_chrony                       0.02%     3.027MiB / 1.822GiB   0.16%     244kB / 8.06kB    13.2MB / 446kB    8
62f9f4b614b8   addon_core_mosquitto                        0.06%     20.73MiB / 1.822GiB   1.11%     282kB / 53.9kB    58.1MB / 348kB    21
85a2d8f52022   addon_core_ssh                              0.00%     1.945MiB / 1.822GiB   0.10%     229kB / 1.55kB    5.39MB / 434kB    11
f3242bb21b93   addon_1315902c_signal_messenger             0.00%     13.71MiB / 1.822GiB   0.74%     223kB / 0B        60.6MB / 16.4kB   9
7dc60da19fb8   addon_a0d7b954_adguard                      75.55%    162.3MiB / 1.822GiB   8.70%     0B / 0B           74.3MB / 532kB    25
f82abd6b84f5   addon_5c53de3b_esphome                      0.33%     18.24MiB / 1.822GiB   0.98%     0B / 0B           65.6MB / 729kB    12
f661e37c9782   addon_cebe7a76_hassio_google_drive_backup   1.49%     23.48MiB / 1.822GiB   1.26%     332kB / 63.4kB    33.2MB / 1.06MB   13
26998772fdd2   addon_core_configurator                     0.02%     17.68MiB / 1.822GiB   0.95%     634kB / 6.62kB    35.7MB / 340kB    8
037138e53d18   addon_a0d7b954_wireguard                    0.00%     6.941MiB / 1.822GiB   0.37%     145kB / 3.03kB    6.41MB / 520kB    13
5aa9fcff583d   addon_f4f71350_ewelink_smart_home_slug      7.05%     47.5MiB / 1.822GiB    2.55%     551kB / 131kB     48.4MB / 106kB    25
a3ea669cad96   addon_a0d7b954_sqlite-web                   0.01%     27.32MiB / 1.822GiB   1.46%     138kB / 3.03kB    37.6MB / 483kB    11

I use adguard for both a DHCP and filtering DNS server, it also resolves hosts on the local network. So I configure homeassistant to use the adguard DNS server (so that scripts etc can use hostnames) -

# ha dns info
fallback: true
host: 172.30.32.3
llmnr: true
locals:
- dns://192.168.175.10
mdns: true
servers:
- dns://192.168.175.10
update_available: false
version: 2024.03.0
version_latest: 2024.03.0

If I change the home assistant servers to be a pulic DNS server -

# ha dns option --servers dns://194.168.4.100

Command completed successfully.

Then the high cpu usage and reboot goes away … however this means that many scripts fail since local network hostnames are not resolved.

Looks like a new issue with 2024.03.0 to me.

Any thoughts ? I guess I should also post to plugin-dns github issues.

[francisp (Francis)]

Except the dns container does get get updated together with the HA Container.

Maybe read

Very high CPU usage for CoreDNS

I’ve been running HA in an ESXi virtual machine now for about 8 months without a problem but about two weeks ago I noticed that the server was experiencing higher than usual load and I finally got around to looking into it and found that the culprit is my HA instance which had unusually high CPU usage. So I checked with top to see the processes and which might be using the CPU and it turns out it’s CoreDNS that has CPU usage between 50-100% [CoreDNS] This just randomly seems to have happened…

and

High CPU load hassio_dns Container

Hi @CentralCommand, I’ve been able to root cause the runaway DNS queries as a result of what I call infinite looping configuration in /etc/corefile. The lines of configuration that are contributing to this issue are loop, fallback REFUSED,NXDOMAIN, and max_fails 0. Here’s what happens: CoreDNS starts and executes the loop plugin, which sends a query for <random number>.<random number>.zone that gets forwarded externally and the Root Servers respond with NXDOMAIN. CoreDNS triggers the fallb…

[plord (Peter Lord)]

Thanks.

Some more info here High cpu usage / server reboot with 2024.03.0 · Issue #130 · home-assistant/plugin-dns · GitHub

In this case it seems to be a startup race condition between the stock dns server and adguard dns server.

[plord (Peter Lord)]

@plord

Some more info here High cpu usage / server reboot with 2024.03.0 · Issue #130 · home-assistant/plugin-dns · GitHub

Ah. Stefan found a DNS loop between hassio_dns and adguard … perhaps the recent update was just a trigger.

Fix was to disable “Use private reverse DNS resolvers” in adguard.

[TheMystery (Johnny)]

I had major problems with dns too after the update to supervisor 2024.03.0 yesterday evening and this morning on my second instance. Because the supervisor was the only thing that was updated i found this thread and it’s indeed solved when disabeling “User private reverse DNS resolvers” in adguard.
Thanks

[plord (Peter Lord)]

Also raised to DNS loop between adguard and hassio_dns · Issue #501 · hassio-addons/addon-adguard-home · GitHub if there is any more info to add.

[Tahutipai]

Fix for me was to ssh into home assistant and run:
ha dns options --fallback=false