ESPHome Add-on updates not shown anymore since version 2024.11.0

thusassistint · December 2, 2024, 12:51pm

I just found out that there are two new updates of the ESPHome Add-on since version 2024.11.0 that I have running now.
However, these new updates are not automatically shown in the Home Assistant Settings → Updates interface.
Is this the same for everyone?
As far as I know I did not change anything for the last weeks, so what could have caused this?
In the past I sometimes skipped ESPHome updates, but I am sure I didn’t do this now, and these updates don’t appear when I select “Show skipped updates”.
It might be coincidence, but this is since the name of the ESPHome Add-on is changed from “ESPHome” into “ESPHome Device Compiler”, so could that be related to this problem?

And is there another method to upgrade the ESPHome Add-on manually?

tom_l · December 3, 2024, 7:38am

I’m still seeing the skipped update (I’m on 2024.10.3).

Screenshot 2024-12-03 at 18-36-50 Settings – Home Assistant

What Home Assistant install method are you using?

What are your DNS settings?

What is the add-on repository you are using?

thusassistint · December 3, 2024, 10:12am

Thanks Tom.

I am running two instances of Home Assistant in my local network: the main unit and a testing unit.
The ESPHome Add-on is only installed on the testing unit.
That one is a HAOS install on an Intel NUC with all updates installed:

Core 2024.11.3
Supervisor 2024.11.4
Operating System 13.2
Frontend 20241106.2

I am not sure which settings you are refering to.
In my network I am using two Pi-holes, of which one is defined in the HA IPv4 Network settings. IPv6 is currently disabled in my local network.
In the Pi-hole logs I am not seeing anything related to Home Assistant connection problems, and it looks like all HA DNS queries are resolved.
Which FQDN is used by HA for the Add-on update check?

I think I am using the standard repository, and for instance not HACS, so nothing fancy.
But I don’t remember how to check this, so please can you remind me how to do this?

However, I just checked the Home Assistant Supervisor logs, and am seeing several of these entries:

2024-12-03 10:40:14.671 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-12-03 10:40:14.671 INFO (MainThread) [supervisor.resolution.fixups.store_execute_reset] Reset corrupt Store: d5369777
2024-12-03 10:40:14.672 ERROR (MainThread) [supervisor.utils] Can't remove folder /data/addons/git/d5369777: find: ‘/data/addons/git/d5369777’: No such file or directory
2024-12-03 10:40:14.672 INFO (MainThread) [supervisor.store.git] Cloning add-on https://github.com/music-assistant/home-assistant-addon repository
2024-12-03 10:40:14.696 ERROR (MainThread) [supervisor.store.git] Can't clone https://github.com/music-assistant/home-assistant-addon repository: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v --recursive --depth=1 --shallow-submodules -- https://github.com/music-assistant/home-assistant-addon /data/addons/git/d5369777
  stderr: 'Cloning into '/data/addons/git/d5369777'...
fatal: unable to access 'https://github.com/music-assistant/home-assistant-addon/': Could not resolve host: github.com
'.

This might be related to the problem?

tom_l · December 3, 2024, 10:19am

This is the problem and I bet it is your pi-hole.

Can you turn it of momentarily and then Go to Settings → System → Updates → three dots menu → Check for updates?

Wait a couple of minutes. See if you get an esphome update.

Don’t forget to turn your pi-hole back on afterwards.

thusassistint · December 3, 2024, 10:58am

Thanks Tom.
I temporarily changed the DNS server to my internet providers server, and this indeed did the trick: I am now getting version 2024.11.2 served (didn’t install the update yet).
Note: I had to restart the HA sever to get the new DNS server working.

Still I don’t understand what was going wrong here: I could resolve github.com normally from the IoT VLAN where HA is residing: and get the IP address 140.82.121.3
And I could also ping to both github.com and 140.82.121.3
However, I now also saw this in the Pi-hole logs:

So there clearly is something going wrong here.
Any idea what it could be?

tom_l · December 3, 2024, 11:00am

Unfortunately I have never used pi-hole. Try a Google search for “pihole upstream error blocklist”

thusassistint · December 3, 2024, 11:11am

Thanks, will do that.
In the mean time I reverted the DNS setting back to my Pi-hole, and am getting the same logs in pi-hole again, so it clearly is related to the Pi-hole.
When I find some new info I will report back.

thusassistint · December 3, 2024, 5:51pm

An update on this issue: the problem with github.com not resolving appears to not being caused by the Pi-hole DNS servers, but by the Main DNS server in my network.

To clarify: up to now I have been running two Pi-hole DNS servers in parallel for redundancy, and both of these Pi-holes were forwarding all non-resolvable DNS queries to the main DNS server in my local network.
This main DNS server was forwarding all queries that it could not answer itself to my Internet providers DNS servers.
This main server is an old Windows Server that amongst others is the DHCP, DNS and DFS server for my local network.
This has been working flawlessly for years, but now I suddenly have this problem with HA not being able to access github.com due to these DNS upstream errors.
Strangely enough, up to now I am only seeing this for the github.com FQDN and not for any other DNS queries.
And when I go to github.com through any other means, like a web browser, ping etc, the FQDN is resolved normally.
So the only problem seems to be HA sending a DNS query for github.com to the main DNS server.

To troubleshoot the problem I initially let the Home Assistant server use the Internet providers DNS servers directly (as written in the previous post), and that solved the problem.
But then I let the Home Assistant server use the main local DNS server directly (so skipping the Pi-holes), and the problem was back.
So then I let the Pi-hole DNS servers go out directly to the Internet providers DNS servers (so skipping the main DNS server), and let the Home Assistant servers use the Pi-holes again, and the problem was gone again.
So the conclusion must be that my main DNS server somehow has trouble to forward the github.com FQDN DNS queries from Home assistant to external DNS servers.

A packet capture shows these Format errors in the DNS traffic for the query response to github.com:

Domain Name System (response)
    Transaction ID: 0x2441
    Flags: 0x8181 Standard query response, Format error
        1... .... .... .... = Response: Message is a response
        .000 0... .... .... = Opcode: Standard query (0)
        .... .0.. .... .... = Authoritative: Server is not an authority for domain
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... 1... .... = Recursion available: Server can do recursive queries
        .... .... .0.. .... = Z: reserved (0)
        .... .... ..0. .... = Answer authenticated: Answer/authority portion was not authenticated by the server
        .... .... ...0 .... = Non-authenticated data: Unacceptable
        .... .... .... 0001 = Reply code: Format error (1)

And a packet capture for a normal working DNS query response to another FQDN shows this:

Domain Name System (response)
    Transaction ID: 0xa166
    Flags: 0x8180 Standard query response, No error
        1... .... .... .... = Response: Message is a response
        .000 0... .... .... = Opcode: Standard query (0)
        .... .0.. .... .... = Authoritative: Server is not an authority for domain
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... 1... .... = Recursion available: Server can do recursive queries
        .... .... .0.. .... = Z: reserved (0)
        .... .... ..0. .... = Answer authenticated: Answer/authority portion was not authenticated by the server
        .... .... ...0 .... = Non-authenticated data: Unacceptable
        .... .... .... 0000 = Reply code: No error (0)

So this is identical except for the Format error (1).

I am by no means a network specialist and still don’t understand why this is happening, but when I find some more info I will report back.
If anybody has an idea about this then please let me know.

tom_l · December 3, 2024, 9:39pm

Forward them to 9.9.9.9 instead. Your ISP has a DNS block list.

thusassistint · December 4, 2024, 9:49am

Thanks Tom, but I don’t think the ISP DNS servers are the culprit, because it is working fine when HA or the Pi-holes are forwarding to them directly. The issue only happens when the local main DNS server is in between.
I am still not sure, but it might be that the local main DNS server is too old and not fully supporting specific DNS options like DNSSEC or something else.

tom_l · December 4, 2024, 9:52am

Ah I see. What’s stopping you cutting out the middle DNS server and forwarding pihole direct to your ISP’s DNS servers for unknown queries?

thusassistint · December 4, 2024, 10:05am

Yes, that’s what I am doing now.
This original setup was because initially I only had this one main DNS server.
The idea was to keep DNS traffic local as much as possible to reduce outgoing DNS traffic to only new locally unsolvable queries.
At some time I added the two Pi-holes, but kept the main server in between doing its job.
Now that I am only using those two Pi-holes in parallel it theoretically could mean more outgoing DNS traffic because the two Pi-holes are not synchronized.
But in reality I don’t think it will be much of a performance issue.

Still, purely out of interest I am trying to to understand what exactly the problem is with this old DNS server. If I find the cause I will report back.

tom_l · December 4, 2024, 10:26am

I reckon you might be on the right track with this: