Tracking down an ESPhome integration issue

TripitakaBC · June 1, 2023, 5:11pm

I have an ESP8266 which has been integrated with my ESPhome instance but then, for an unknown reason, dropped off and will not reintegrate. ESPhome connects to it fine and the code and ESPhome firmware update OK, it just will not integrate into the HA ESPhome integration.

I noticed that the logs show as:

[10:00:12][C][ota:093]: Over-The-Air Updates:
[10:00:12][C][ota:094]:   Address: 192.168.18.23:8266
[10:00:12][C][ota:097]:   Using Password.

yet the code in the YAML is:

api:
  encryption:
    key: !secret apikey
  port: 6053
  reboot_timeout: 15min

All the other devices use the apikey flawlessly. What could be causing the device to try to use an API password when the apikey is present and specified and, more importantly, how can I solve this? I tried deleting all the yaml files under /config/esphome for this device but no progress.

Here is the full yaml:

esphome:
  name: esp8266-freezer-os1
  comment: PVIoT | 18.23

esp8266:
  board: d1_mini
  framework:
    version: recommended

logger:
  level: DEBUG

wifi:
  manual_ip:
    static_ip: 192.168.18.23
    gateway: 192.168.15.1
    subnet: 255.255.0.0
    dns1: 192.168.15.1
    dns2: 0.0.0.0
  ap:
    ssid: "Fallback-OutsideFreezer"
    password: ""
    ap_timeout: 1min
  domain: local
  reboot_timeout: 15min
  power_save_mode: NONE
  fast_connect: false
  output_power: 20.0
  networks:
  - ssid: !secret wifi_iot_ssid
    password: !secret wifi_iot_password
    priority: 0.0

api:
  encryption:
    key: !secret apikey
  port: 6053
  reboot_timeout: 15min
ota:
  password: !secret ota_password
  safe_mode: true
  port: 8266
  
switch:
  - platform: restart
    name: Freezer Outside1 Restart  

dallas:
  - pin: D2 #GPIO4
    
sensor:
  - platform: dallas
    address: 0xAF3C01B556D00228
    name: Freezer Outside1 Temperature
    filters:
        - filter_out: nan
        - heartbeat: 15s
        - sliding_window_moving_average:
            window_size: 5
            send_every: 5
            send_first_at: 1
    unit_of_measurement: "°C"

koying · June 2, 2023, 8:28am

That’s the ota port you specified.

ota:
  password: !secret ota_password
  safe_mode: true
  port: 8266

Sure you didn’t use the same static IP twice?

TripitakaBC · June 2, 2023, 2:11pm

Yes. I keep a spreadsheet but I also checked my OPNsense firewall logs and ran Zenmap to check.

LOL! the OTA part you pointed out is another great example of how sometimes, we can get too close to a problem to see the obvious stuff.

I’m thinking that something in the back end of HA is storing an old entry somewhere other than in the yaml files.

aceindy · June 2, 2023, 2:21pm

Wouldn’t it be easier to reserve a DCHP lease on your router?
In that way you do not have to maintain a spreadsheet…
Al my IOT devices are on DHCP lease reservation

koying · June 2, 2023, 2:24pm

Yeah, we don’t even need AI

Protoncek · June 2, 2023, 2:27pm

Been there, done that. It worked, until i decided to replce router with a better one and i’d need to do aaaaalll entries again. From then on i rather have static IP’s. Note that router can also crash, and if you don’t have most recent backup…
And, i did have spreadsheet then and i have it now. Using dhcp reservations doesn’t save you from having spreadsheet (at least if you want some control over your ip’s).

koying · June 2, 2023, 2:35pm

Joking aside, having static addresses might influence mDns, which might be a possible explanation…

TripitakaBC · June 2, 2023, 2:47pm

I have the static addresses in the DHCP server but a spreadsheet is easier to work with when planning and managing blocks, dev/test subnets and so on. As you can see, I have a 255.255.0.0 subnet for a reason.

mDNS is hooped on Ubiquiti systems, it is just more trouble than it is worth and with over 50 ESP devices, I shouldn’t just be getting an issue with one if mDNS was the issue.

paddy0174 · June 2, 2023, 3:33pm

That is the “ok” sign. According to the code, this is the message, that is logged if password != empty. See here:

github.com

esphome/esphome/blob/8bb4c65272b41ccc6fa25a9e39096617c8c041c5/esphome/components/ota/ota_component.cpp#L97


      
            }
          
            this->dump_config();
          }
          
          void OTAComponent::dump_config() {
            ESP_LOGCONFIG(TAG, "Over-The-Air Updates:");
            ESP_LOGCONFIG(TAG, "  Address: %s:%u", network::get_use_address().c_str(), this->port_);
          #ifdef USE_OTA_PASSWORD
            if (!this->password_.empty()) {
              ESP_LOGCONFIG(TAG, "  Using Password.");
            }
          #endif
            if (this->has_safe_mode_ && this->safe_mode_rtc_value_ > 1 &&
                this->safe_mode_rtc_value_ != esphome::ota::OTAComponent::ENTER_SAFE_MODE_MAGIC) {
              ESP_LOGW(TAG, "Last Boot was an unhandled reset, will proceed to safe mode in %" PRIu32 " restarts",
                       this->safe_mode_num_attempts_ - this->safe_mode_rtc_value_);
            }
          }
          
          void OTAComponent::loop() {

Have you enabled the logger specifically for the ESPHome integration in HA? There might be something more useful, as it looks like the HA side is not running properly, not the ESPHome side.

aceindy · June 2, 2023, 3:41pm

And i use

255.255.252.0

also for a reason…

and occasionally i export the dhcp reservatios… from my router

TripitakaBC · June 2, 2023, 9:00pm

@paddy0174 - I do have it enabled but increased to debug level and it seems to deliver the following:

2023-06-02 13:55:13.855 WARNING (MainThread) [aioesphomeapi.connection] 192.168.18.23: Connection error occurred: Connection requires encryption
2023-06-02 13:55:14.412 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 235, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 146, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 180, in post
    return await super().post(request, flow_id)
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper
    result = await method(view, request, data, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 110, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 271, in async_configure
    result = await self._async_handle_step(
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 367, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
  File "/usr/src/homeassistant/homeassistant/components/esphome/config_flow.py", line 177, in async_step_discovery_confirm
    return await self._async_try_fetch_device_info()
  File "/usr/src/homeassistant/homeassistant/components/esphome/config_flow.py", line 149, in _async_try_fetch_device_info
    and await self._retrieve_encryption_key_from_dashboard()
  File "/usr/src/homeassistant/homeassistant/components/esphome/config_flow.py", line 384, in _retrieve_encryption_key_from_dashboard
    noise_psk = await dashboard.api.get_encryption_key(device["configuration"])
  File "/usr/local/lib/python3.10/site-packages/esphome_dashboard_api/__init__.py", line 103, in get_encryption_key
    config = await self.get_config(configuration)
  File "/usr/local/lib/python3.10/site-packages/esphome_dashboard_api/__init__.py", line 93, in get_config
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/esphome_dashboard_api/__init__.py", line 54, in request
    return await resp.json()
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 72, in json
    return await super().json(*args, loads=loads, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1120, in json
    return loads(stripped.decode(encoding))
orjson.JSONDecodeError: unexpected character: line 1 column 1929 (char 1928)

I’m not sure if the first line and the rest are connected but these only seem to show up when I try to add the device to the ESPhome integration.

paddy0174 · June 2, 2023, 10:29pm

Then I have three guesses Before you start, delete both lines port and reboot_timeout, as they are already the defaults. The easier, the better…

#1 The api key you’re using is not the correct format. Are you sure it’s a 32-byte base64 encoded string?
#2 I’m not sure it is allowed to use the api key on more than one device, so you would need different ones for each device.
#3 You’re using the wrong secrets file, the one from HA and not the one from ESPHome.

All guesses could be checked, if you use the randomly generated key from the ESPHome webpage and put it directly under key:. If it works, you’re good to go, if not, let’s see if something else pops up with these changes.

TripitakaBC · June 2, 2023, 11:40pm

#1 API key: Yes, the same key works on 50+ other ESP devices flawlessly and even on the other freezer monitor (code below) which has the same functional code.

#2 Key reuse: I use the same key on all my other ESPs and they function so I’m going to say that it is possible to do it. In fact, all devices use exactly the same api code in the yaml.

#3 Secrets: I use the !include function to populate the ESPhome secrets from the HA secrets to avoid identical duplicate files. Again, each ESP uses the same file, same method etc so I am minded to think that this isn’t a code issue but rather something that has gone screwy in the database or a file reference somewhere.

Identical device which works perfectly:

esphome:
    name: esp8266-freezer-is1
    comment: PVIoT | 18.24
  
  esp8266:
    board: d1_mini
    framework:
      version: recommended
  
  logger:
  
  wifi:
    manual_ip:
      static_ip: 192.168.18.24
      gateway: 192.168.15.1
      subnet: 255.255.0.0
      dns1: 192.168.15.1
      dns2: 0.0.0.0
    ap:
      ssid: "Fallback-InsideFreezer"
      password: ""
      ap_timeout: 1min
    domain: .local
    reboot_timeout: 15min
    power_save_mode: NONE
    fast_connect: false
    output_power: 20.0
    networks:
    - ssid: !secret wifi_iot_ssid
      password: !secret wifi_iot_password
      priority: 0.0
  
  captive_portal:
  
  api:
    encryption:
      key: !secret apikey
    port: 6053
    reboot_timeout: 15min
  ota:
    password: !secret ota_password
    safe_mode: true
    port: 8266
    
  switch:
    - platform: restart
      name: Freezer Inside1 Restart  
  
  dallas:
    - pin: D2 #GPIO4
      
  sensor:
    - platform: dallas
      address: 0x913C01B556383328
      name: Freezer Inside1 Temperature
      filters:
          - filter_out: nan
          - heartbeat: 15s
          - sliding_window_moving_average:
              window_size: 5
              send_every: 5
              send_first_at: 1
      unit_of_measurement: "°C"

paddy0174 · June 3, 2023, 12:50am

That’s unfortunate, I had hoped for one of these… I’m sorry, but I have no idea why it’s not working, all I see seems correct…

Try opening an issue, maybe it is a bug…?