Binary sensors show unavailable after upgrade to 0.112

Dave_B · July 7, 2020, 8:06am

Ok, So this is what I have installed from the Official Add-On Store, I was under the impression that it was Eclipse Mosquitto V5.1? If it’s not Eclipse, what is it?

laca75tn · July 7, 2020, 1:14pm

I have the same configuration. The only difference is that I have Tasmota 6.5 on my RF bridge.

mrh3nk · July 7, 2020, 6:49pm

Hey, this is indeed a change since the last version of Home Assistant, see the release note:

Other noteworthy changes

@balloob has shaved of a couple of seconds from the Home Assistant startup again.
> * Entities that originate from MQTT will now become “unavailable” when the integration is not connected to the MQTT broker. Thanks @elupus!

If you have a Xiaomi vacuum cleaner, @jthure added a service to send it to specific places using a new goto service.

The Smappee integration has been fully rewritten, by @bsmappee themselves!

Auto discovery has been added to the NUT integration. Please note that some NAS devices might be discovered as they support adding an external battery.

Link: 0.112: Making things faster; Logbook & History - Home Assistant

So: when you restart your home assistant instance, first new data has to be sent to have a new data point, for every MQTT sensor.

123 · July 7, 2020, 7:07pm

The change in 0.112 means an MQTT sensor’s state is unavailable during the very brief time before Home Assistant connects to the broker and subscribes to the sensor’s state_topic.

If the broker has a retained message for the state_topic, Home Assistant will receive it immediately.
If the broker does not have a retained message for the state_topic, Home Assistant will receive nothing and the sensor’s state will continue to be unavailable.

I have dozens of MQTT-based lights, switches and sensors and they all correctly report their state on restart using 0.112. The important thing is that all of their state_topics have retained messages residing on the broker.

mrh3nk · July 7, 2020, 7:20pm

Yes, that will probably work fine. For most of my sensors, it is not such a big deal of the sensor shows up as not available. In some cases it is not desired to have the data retained in the MQTT broker. It could lead to incorrect statuses. Therefore, I choose to not retain the data, unless it is necessary. And it is not necessary until now…

123 · July 7, 2020, 7:27pm

So you are saying that you have MQTT sensors whose state_topics do not have retained messages and, in 0.111, if you restarted Home Assistant, these sensors would not display unavailable but would display a state value.

So, in 0.111, where was Home Assistant getting the sensor’s state value on startup?

mrh3nk · July 7, 2020, 11:32pm

I don’t know for sure, but probably the previous database entry in the sqlite database that Home Assistant uses.
But yes, since this update, the sensors are now initially not available, until they have reported something.
By the way, I am using Zigbee2MQTT and I have about 30 devices connected via MQTT. Sensors, switches, lights, etc. None of them I have had issues with up till now (regarding this unavailability).

123 · July 8, 2020, 1:28pm

No, the database is not used for that purpose.

This file:

.storage/core.restore_state

is used to store an entity’s state periodically and just prior to shutdown/restart. Upon startup, the file’s contents (JSON) are read and entity states are restored to their last-known value.

Here’s an example of an item in the file. It is for my pool pump which uses the MQTT integration:

        {
            "last_seen": "2020-07-08T12:27:52.005245+00:00",
            "state": {
                "attributes": {
                    "friendly_name": "Pool Pump"
                },
                "context": {
                    "id": "8e97ca343ce94712b7da0f1eb315734g",
                    "parent_id": null,
                    "user_id": null
                },
                "entity_id": "switch.pool_pump",
                "last_changed": "2020-07-08T12:15:00.049349+00:00",
                "last_updated": "2020-07-08T12:15:00.049349+00:00",
                "state": "on"
            }
        },

The interesting thing is that I have many MQTT-based sensors and binary_sensors but none are listed in core.restore_state. I have retained messages published to their state_topics so, on startup, they all acquire their last-known state from the broker.

As an experiment, I will create a test system with 0.111.4 and inspect its core.restore_state file to see if contains sensor and binary_sensor states.

After reading PR36479 that implemented the new unavailable behavior, I discovered it has also impacted MQTT Binary Sensors that share a state_topic (i.e. multiple binary_sensors subscribed to the same topic published by a Sonoff RF Bridge). Prior to 0.112, this template would work flawlessly:

value_template: >-
  {% if value_json.RfReceived.Data == '545CE9' %}
    {{'ON'}}
  {% else %}
    {{states('binary_sensor.motion_sensor') | upper}}
  {% endif %}

as of 0.112 it can produce this error message:

No matching payload found for entity

The reason is because now, in 0.112, the binary_sensor’s state appears to have a higher chance of having a state of unavailable. So when the received payload is not '545CE9' it executes the else which simply uses the binary_sensor’s current state and converts it to uppercase. That’s fine if the current state is on or off but not if it’s unavailable. By default, a binary_sensor will accept two kinds of payload, ON or OFF. It won’t accept UNAVAILABLE and so, in 0.112, it reports “No matching payload found for entity”.

There is a new PR (37420) that will now allow an MQTT Binary Sensor to accept an “empty output”. That means the payload can be this:

value_template: >-
  {% if value_json.RfReceived.Data == '545CE9' %}
    {{'ON'}}
  {% endif %}

However, it will also report to the log each time that happens …

Ironically, that kind of simple template was acceptable many versions ago but had to change to the else style to avoid getting the error message: “No matching payload found for entity”!

All this to say that the recent alternation has created more than one unintended consequence.

tom_l · July 8, 2020, 1:38pm

I just updated both HA and HasOS. All was well after the HA update but the HasOS update caused all my (retained) MQTT binary sensors to become unavailable too.

I waited 5 minutes with no change. Restarted again, and they came back

123 · July 8, 2020, 3:24pm

I compared the behavior of 0.111.4 to 0.112.3 and the experiments I performed failed to reveal any behavioral difference between the two versions. Based on what I know about MQTT, and how Home Assistant’s MQTT integration operates, the experiments didn’t produce unusual results (i.e. everything behaved as expected).

I invite commentary on how to alter the experiments in order to demonstrate any differences. Otherwise, it is challenging to explain the new observed behavior to the development team beyond the superficial "it now reports unavailable". They would be justified in responding “unable to reproduce the bug”.

Test 1

Publish the binary_sensor’s discovery topic and its state_topic without retained messages.

Click to reveal details

Create a binary_sensor via MQTT Discovery by publishing the following:

topic: homeassistant/binary_sensor/example_door/config
payload: >
  {
    "name":"Example Door",
    "state_topic":"test/example_door",
    "device_class":"door",
    "unique_id":"test_example_door"
  }
retain: false

Default state upon creation of binary_sensor.example_door:

In version 0.111.4: off
In version 0.112.3: off

Publish ON to test/example_door but not as a retained message. In both versions, example_door’s state is now on.
Check .storage/core.restore_state and both versions lack an entry for example_door therefore there is no locally stored state to restore on startup.
Restart both versions.
Initial state of binary_sensor.example_door upon startup:

In version 0.111.4: unavailable
In version 0.112.3: unavailable

Conclusion

Both versions behave the same way. The state of example_door remains unavailable because its discovery topic did not contain a retained message. Therefore, on startup, the entity’s configuration is unknown.

Test 2

Repeat test, this time publishing the discovery topic’s payload as a retained message (but not its state_topic).

Click to reveal details

Create a binary_sensor via MQTT Discovery by publishing the following:

topic: homeassistant/binary_sensor/example_door/config
payload: >
  {
    "name":"Example Door",
    "state_topic":"test/example_door",
    "device_class":"door",
    "unique_id":"test_example_door"
  }
retain: true

Default state upon creation of binary_sensor.example_door:

In version 0.111.4: off
In version 0.112.3: off

Publish ON to test/example_door but not as a retained message. In both versions, example_door’s state is now on.
Restart both versions.
Initial state of binary_sensor.example_door upon startup:

In version 0.111.4: off
In version 0.112.3: off

Conclusion

Both versions behave the same way. The payload ON was not published as a retained message so, on startup, the binary_sensor’s state defaults to off.

Test 3

Repeat test, this time publishing retained messages to both the discovery topic and the state_topic.

Click to reveal details

Create a binary_sensor via MQTT Discovery by publishing the following:

topic: homeassistant/binary_sensor/example_door/config
payload: >
  {
    "name":"Example Door",
    "state_topic":"test/example_door",
    "device_class":"door",
    "unique_id":"test_example_door"
  }
retain: true

Default state upon creation of binary_sensor.example_door:

In version 0.111.4: off
In version 0.112.3: off

Publish ON to test/example_door as a retained message. In both versions, example_door’s state is now on.
Restart both versions.
Initial state of binary_sensor.example_door upon startup:

In version 0.111.4: on
In version 0.112.3: on

Conclusion

Both versions behave the same way. The payload ON was published as a retained message so, on startup, the retained message is received and used to set example_door’s state to on.

123 · July 10, 2020, 1:16pm

FYI

The problem has been identified to be a bug (see Issue 37662). It appears to be due to a race condition on startup.

There are currently two open PR’s to correct it:

laca75tn · July 10, 2020, 1:19pm

Great! I really appreciate your hard work and efforts to fix this. I know a lot of people are going to be happy with the fix as well! Thank you again!

123 · July 10, 2020, 1:29pm

You’re welcome but thanks should go to emontnemery who is correcting the problem. My involvement was simply to help gather information to characterize the problem.

The new intentional behavior is to report unavailable on startup if there’s no connection to the broker.
The new unintentional behavior is that it sometimes reports unavailable on startup even if there is a connection to the broker. This is a bug.

laca75tn · July 10, 2020, 3:22pm

Nevertheless, thank you for your help in channeling the info

fnees · July 17, 2020, 12:34am

Sorry. Missed that…