ESPHome over VLAN became unreachable

ESPHome
1

After a year of working perfectly fine, my setup suddenly stopped working with my ESPhome devices not being able to connect anymore. It has been two weeks now and I can’t figure a solution out myself, so help from you guys would be greatly appreciated.

The problem:

Home Assistent, myself and ESPHome can no longer connect to my ESP devices.

In ESPHome all the devices come up as online (green bar), but when I click the log I get the following error, and because of this I now can’t flash them over the air either.

INFO Connecting to 192.168.207.151:6053 (192.168.207.151)
WARNING Initial connection failed. The ESP might not be connected to WiFi yet (Error connecting to 192.168.207.151: timed out). Re-Trying in 2 seconds
  • When trying to contact the device via the browser I get a time-out (ERR_CONNECTION_TIMED_OUT).
  • I can see in the Unifi (Dream Machine) controller that all the devices receive an IP address (statically assigned in ESPhome firmware) that lives for (almost exactly) 15 minutes, then the device reboots and the uptime resets before it starts another 15 minutes (this must be the ESPHome default option for reboot_timeout (docs).
  • Other devices in that same VLAN show no trouble at all (WLED / Hyperion) and I can reach them in the browser for example.

Network summary:

LAN 192.168.1.0/24
This is where the Home Assistent instance lives, and all the family devices (laptops, phones, tablets, etc)

IoT 192.168.107.0/24
This is where the cloud services devices live like the washer, robot vacuum, tv.

NoT 192.168.207.0/24
My devices that should not use the cloud and should one day be blocked from the internet completely via the firewall (these firewall rules are not yet in place!).

Firewall rules:
None, basically. Except for the default ones created by Unifi controller itself. I just want to split the wifi networks (for now), and later start using the firewall rules to actually lock it down.

What I’ve tried/checked:

  • I also can’t ping the ESP devices that are in that network. Other devices in that network can be pinged.
  • I re-flashed (via USB) a ESP device to my main LAN (as 192.168.1.151) and then it all works perfectly fine. I see it come online, I can view the logs and I can flash OTA. Also the web server works so I can reach it in the browser.
    INFO Connecting to 192.168.1.150:6053 (192.168.1.150)
    INFO Successfully connected to 192.168.1.150
    [20:17:33][I][app:105]: ESPHome version 1.20.2 compiled on Jul 31 2021, 20:08:26
    [20:17:33][C][wifi:499]: WiFi:
    [20:17:33][C][wifi:359]:   SSID: [redacted]
    [20:17:33][C][wifi:360]:   IP Address: 192.168.1.150
    [...]
  • It happened to all ESPHome devices at once, not a single device or a subset but all devices.
  • I checked the firewall (LAN in/out) rules but there are no DENY rules. Also tried adding an ALLOW rule from the NoT VLAN to the LAN network for all IP’s and Ports, with no luck. Also this feels redundant because the other services (WLED/Hyperion) work with no problem.
  • I changed the main domain from localdomain to local as @petro mentioned in Securing a vlan & ESPHome Devices - #12 by petro
  • Tried looking in the router logs for anything on those IP addresses, but I can’t find anything of importance.
  • The router was set to auto-update software but I can’t find anything in the changelog’s that would explain my problem.

Help…?

Yes please!
I don’t know networking and the Unifi controller good enough to find the source of this problem…
I would really like to keep my ESP devices in a dedicated LAN and it would be a real annoying/hard job to manually flash all the devices via a cable (they’re in ceilings, behind kitchen appliances, etc). If it would just go back to how it worked for over a year, that would make my day week! Can’t find anything to roll back a back-up to make it magically come back to life I’m afraid.

2

A few more ideas:

  • Can you connect a PC to your NoT VLAN and try to access (ping, web UI) your ESPHome devices?
  • Can you connect via USB/serial to one of your ESPHome devices in the NoT VLAN and check if there are maybe error messages, like being unable to properly connect to WiFi?
  • If the ESPHome device has a statically assigned IP address, then the fact that this IP address is seen by the UDM does not mean it is definitely connected to the network. The UDM shows the IP address reported by the device irrespective of whether it’s valid or not.
  • A firewall rule specifically allowing access may help determining if there is actually any traffic, i.e. you can “Enable Logging” on the rule and then observe the log file to confirm that traffic is logged for this rule.
  • I wouldn’t necessarily trust that the UDM firmware changelog is complete.
3

Yes, I can connect to that WiFi with my laptop, and I’m able to ping the devices in that VLAN that do not respond when I’m in my main LAN.
Even the ESPHome web portal works, showing me the following log:

[D][sensor:117]: 'kitchen_counter_uptime': Sending state 880.61603 s with 0 decimals of accuracye
[D][sensor:117]: 'kitchen_counter_wifi_signal': Sending state -75.00000 dBm with 0 decimals of accuracye
[E][api:084]: No client connected to API. Rebooting...e
[I][app:135]: Forcing a reboot...e
[D][sensor:117]: 'kitchen_counter_uptime': Sending state 95.72900 s with 0 decimals of accuracye

When I switch back to my main LAN WiFi it is unreachable again.
To me that sounds like a routing issue in the Unifi controller. Which is weird because it only happens with ESPHome devices.

I’ll try to set up a ALLOW rule between the two networks with the logging later today, see if that (hopefully) explains some.

4

You said that you configured static IP addresses for your ESPHome devices. Is the correct gateway configured as well?

5

I believe so. The gateway should always be the same IP of the router, right?
In my case 192.168.1.1 on all of the devices across the networks, with subnet of 255.255.255.0

Reading up on setting up the firewall rules as we speak.

6

If you have a vlan with 192.168.207.0, the gateway will be 192 168.207.1. Check the settings in udm where you setup the networks, it will say what the gateway and DHCP server is

Here’s mine, you can see my range is 192.168.40.0/24 is the gateway is 192.168.40.1

Screenshot_20210801_115022_com.android.chrome
Screenshot_20210801_115022_com.android.chrome1080×2244 182 KB

7

That was actually it!

Such a simple fix… All the ESPHome devices had 192.168.1.1 as a gateway because that is my router. My lack of knowledge here convinced me that was correct (and weirdly, has worked for over a year).

I changed one of the devices to the 192.168.207.1 gateway and re-uploaded the newly compiled firmware (over the air via the web portal after switching Wifi) and now I can both ping and use the web portal when I’m connected with my main LAN WiFi network.
That light now also works in the Home Assistant automations again :tada:

This process of still being able to update over the air is a real time saver too, so thanks @exxamalte for that tip.

<3 @Holdestmade, this simple fix has frustrated me for two weeks haha. Thanks so much!

1 Like
8

I’ve only learnt it myself over the past few weeks after upgrading to unifi.

1 Like
9

You need to have mDNS multicast (Bonjour) forwarding service (MDNS Repeater) enabled between both subnets. Also you need to allow conntection to 224.0.0.251:5353 in both subnets. Also allow pinging to ESPHome devices. so allow ICMP to subnet of ESPHome devices. I also use Docker Dashboard with this environment variable "-e ESPHOME_DASHBOARD_USE_PING=true "

10

I use that setting in the esphome add-on config so the esphome nodes show correct colours in the dashboard. Saves having to set any DNS settings.

11

Can you explain in detail what you mean by that?

12

Who are you asking ?

13

Oh sorry, you. I am having mDNS issues with unifi… What esphome config do you mean?

14

Only issue I had is the cards in the esphome dashboard were not showing the correct colour. Apparently it’s because esphome uses mDNS and when using vlans it doesn’t work. I am not a network expert and didn’t know how to solve it with mDNS settings so in the esphome config page there’s an option to use ping instead. I’ve had no issues since

15

Try this https://youtu.be/vz3u6E3Fxi8 @8:30 or so.
Rob explains a lot of settings that might help with mDNS/IGMP snooping across networks, if that is what you’re struggling with.

16

Hi Ronaldt, thank you. I think I have seen this before and set up my vlans accordingly. My problem is sometimes it can resolve hostnames with .local suffix, and sometimes it can not.

17

That’s not true. I’ve got it running and shared you how.