Esphome entities not always updating in Homeassistant

I use esphome 2024.11.1 esp-idf on one of my devices.
This device has over 260 entitties. but now I found sometimes that binary sensors are not always updating in HA, when I go to the esphome device webinterface the value is ok. but in HA the value is different and incorrect.

could I do something to make sure all binary state updates are handled correctly?

1 Like

Logs demonstrating the error please.

there is no log as no status update has been triggered
but here is a screenshot from this afternoon showing the value in HA and the value in ESPHome are different

Had several similar cases - missed values/states from sensors, swicthes, …

One was related with WiFi - many devices stopped to report, as well OTA fails with timeout.
Fixed by channel change.

Another one was on device with 89 entities. While other devices, including located in centimeters from problematic one, experienced no issues.
Fixed by changing logging level from DEBUG to INFO.
FYI supply voltage fluctuations before & right after same YAML compiled with reduced logging:
Screenshot 2024-11-27 VCC debug - no debug
So far see no more losses in reporting states to HA - sensors, switches, …

1 Like

thnx I will try reducing the log level, this could be it.

Unfortunately, in my case, again states of binary sensors disappears again.
Will try to get something usefull directly from device through UART cause nothing in log on HA side.

In my environment this issue noticed for only device - ESP-WROOM-32 with arduino framework 2024.10.1. BTW IMHO issue started recently.

Where you do open the ESPhome web ui on the right? I have esphome installed as Home Ass. addon.

I face the same issue with a binary sensor of my heat pump controlling ESP.

http://<ip of your espdevice>
or click on an entity,
go to device info an click the link

This is esphome webinterface v3.
so if you use the default(v2) you might have a different layout.

for me actually the loglevel was already reduces, so that doesn’t help.
I’m on ESP-IDF.

Thanks a lot I overlooked this nice “debugging” functionality:

Web Server Component — ESPHome

I will verify now whether my esp also shows the correct binary sensor state unlike the home assistant state.
Unfortunately I face this only ~ once a week. So a bit difficult to analyse.

My fear is that the ESP device does not send all packets with updates or on the Home Assistant side not all are handled. And for binary sensor not changing that often this is fatal as the value is only sent in case of state changes. So if the first sate change e.g. from true => false is somehow skipped then the value remains on true until it goes back to true and then to false again.

Regards, Christof


My quick analysis of the code, When you call publish_state with a state parameter then only in case the duplicate check (publish_dedup_.next) finds a different value then the send_state internal or filter_list is called. Otherwise in case state parameter is the same then publish_state returns without doing anything.

When now a write of the updated sensor value (see functions at the end) fail for some unknown reason (network e.g.) then the package is not repeated and more important the new state is stored in the Deduplicator instance and any further publish_state with the same value will do nothing (return).

I hope I did not overlook something like a retry.

In my opinion the logic should only update the Deduplicator in case the update package was sent correctly if possible.

// esphome\components\binary_sensor\binary_sensor.cpp:
void BinarySensor::publish_state(bool state) {
  if (!this->**publish_dedup_**.next(state)) // check whether state is the same as before and store new state if different
    return; // NO CHANGE of state detected
  if (this->filter_list_ == nullptr) {
    this->**send_state_internal**(state, false);
  } else {
    this->filter_list_->input(state, false);
  }

template<typename T> class Deduplicator {
 public:
  /// Feeds the next item in the series to the deduplicator and returns whether this is a duplicate.
  bool next(T value) {
    if (this->has_value_) {
      if (this->last_value_ == value)
        return false;
    }
    this->has_value_ = true;
    this->last_value_ = value; // take over new value
    return true;
  }

void BinarySensor::send_state_internal(bool state, bool is_initial) {
  if (is_initial) {
    ESP_LOGD(TAG, "'%s': Sending initial state %s", this->get_name().c_str(), ONOFF(state));
  } else {
    ESP_LOGD(TAG, "'%s': Sending state %s", this->get_name().c_str(), ONOFF(state));
  }
  this->has_state_ = true;
  this->state = state; // take over new value
  if (!is_initial || this->publish_initial_state_) {
     this->state_callback_.call(state); // I assume this now calls APIServer::on_binary_sensor_update
  }
}

void Controller::setup_controller(bool include_internal) {
#ifdef USE_BINARY_SENSOR
  for (auto *obj : App.get_binary_sensors()) {
    if (include_internal || !obj->is_internal())
      obj->add_on_state_callback([this, obj](bool state) { this->on_binary_sensor_update(obj, state); });

// esphome\components\api\api_server.cpp
#ifdef USE_BINARY_SENSOR
void APIServer::on_binary_sensor_update(binary_sensor::BinarySensor *obj, bool state) {
  if (obj->is_internal())
    return;
  for (auto &c : this->clients_)
    c->send_binary_sensor_state(obj, state);
}

// \esphome\components\api\api_connection.cpp:
#ifdef USE_BINARY_SENSOR
bool APIConnection::send_binary_sensor_state(binary_sensor::BinarySensor *binary_sensor, bool state) {
  if (!this->state_subscription_)
    return false;

  BinarySensorStateResponse resp;
  resp.key = binary_sensor->get_object_id_hash();
  resp.state = state;
  resp.missing_state = !binary_sensor->has_state();
  return this->send_binary_sensor_state_response(resp);

#ifdef USE_BINARY_SENSOR

bool APIServerConnectionBase::send_binary_sensor_state_response(const BinarySensorStateResponse &msg) {

#ifdef HAS_PROTO_MESSAGE_DUMP

ESP_LOGVV(TAG, "send_binary_sensor_state_response: %s", msg.dump().c_str());

#endif

return this->send_message_<BinarySensorStateResponse>(msg, 21);

}

bool APIConnection::send_buffer(ProtoWriteBuffer buffer, uint32_t message_type) {
  if (this->remove_)
    return false;
  if (!this->helper_->can_write_without_blocking()) {
    delay(0);
    APIError err = this->helper_->loop();
    if (err != APIError::OK) {
      on_fatal_error();
      ESP_LOGW(TAG, "%s: Socket operation failed: %s errno=%d", this->client_combined_info_.c_str(),
               api_error_to_str(err), errno);
      return false;
    }
    if (!this->helper_->can_write_without_blocking()) {
      // SubscribeLogsResponse
      if (message_type != 29) {
        ESP_LOGV(TAG, "Cannot send message because of TCP buffer space");
      }
      delay(0);
      return false;
    }
  }

  APIError err = this->helper_->write_packet(message_type, buffer.get_buffer()->data(), buffer.get_buffer()->size());
  if (err == APIError::WOULD_BLOCK)
    return false;
  if (err != APIError::OK) {
    on_fatal_error();
    if (err == APIError::SOCKET_WRITE_FAILED && errno == ECONNRESET) {
      ESP_LOGW(TAG, "%s: Connection reset", this->client_combined_info_.c_str());
    } else {
      ESP_LOGW(TAG, "%s: Packet write failed %s errno=%d", this->client_combined_info_.c_str(), api_error_to_str(err),
               errno);
    }
    return false;
1 Like

What coincidence just observed a Connection reset error during the reading of a packet (not the writing which is described in the code above). But if it occurs during reading then it should also occur during writing.

Nice findings,
I get the point about the Deduplicator,
I don’t understand the reader part from the connection reset.
but I also think that after a connection reset all stattusses are send again.

The reader was just a proof that I might face sporadic network issues. Still waiting to see an error during the write of the packet.
Maybe I overlooked some rewrite code.
The only detail I am sure that the bool return value from :send_binary_sensor_* methods is lost with the callbacks. They are void, thus no chance to retry or avoid updating the sensor state in the binary_sensor. The bool result value never reaches the binary_sensor.

if I take a close look at the code I do not see the need for this deduplicator at all. But I’m not familiar enough with c++ to state this for certain.

but then there probably still is the issue of no retry.

The deduplicator avoids that you send over and over again the same values.

True. But this dedeplicator is only used for Boolean like entities. Binary_sensor, switch, lock.
Normal sensors use another method. Assuming this would be true I think this could be changed for those entity types. But then again. I don’t know the source well enough.

And this would explain why I only notice this issue for binary_sensors.