ESPHome devices don't show an update button

I have several ESPHome devices for which I believe they’re supposed to surface an “update” button in Home Assistant; however, that’s missing.

I’m running both Home Assistant (2023.5.4) and ESPHome (2023.5.5) from Docker on their own containers. From the ESPHome site I can see my devices and several of them have “update available” for which I can successfully perform the update OTA. They were auto-discovered in Home Assistant and once I entered the encryption key, Home Assistant was able to connect and I see the sensors.

Am I not supposed to also see an “update” button in Home Assistant when new versions are available for compiling and pushing to the ESP32 devices OTA? Or is that only possible when ESPHome is running as a container inside HA?

I had a similar problem when I switched from using the HA ESPHome add-on to compile, to using the ESPHome command-line interface on my laptop. Except mine was the other way. The devices were updated but the add-on still showed them as needing an update. The add-on didn’t seem to “know” what version the ESP device was, even though it displayed correctly when I connected to the device via HTTP.

I found I could copy just the .yaml and .json files for the devices I’d compiled via the CLI to the like-named directories in HA, and it wouldn’t show them as needing update any more.

One other thing which got confusing during this time was when the add-on and the CLI were at different versions. You want to make sure they match.