Persistent version of "last-changed" for the UI?

[johny_mnemonic]

I would like to add my 2c and react on the two parts of your statement @frenck. Thanks for your input here btw!

1. State of the object changes when HA is offline, so restoring the sate would be an assumption, not a reality:

   • For most of the objects, HA should be the only source of truth, so they can’t change without HA telling them to change.
   • For sensors it is normal state of affairs as HA is not aware of their change between two data points either. It just updates it’s own info about the state of the sensor when it receives it.

2. The state actually changes at startup, when connecting:

   • This is actually a bit scary as there should not be a reason for such behavior and it is pretty dangerous thing to do. I will explain later in more detail why I say that, but in general the connect/disconnect action should not mess with the state of the object (i.e. switch, door contact).

As @CentralCommand correctly wrote:

You can’t just persist last_changed without also persisting the state of the entity.

So, as most of the people here agrees that changing the state during restart is in 100% cases wrong (my experience is definitely over 95%), I would propose to add an option to enable state persistence either globally or per entity.
Apart from other systems HA has DB where it keeps the current state of each object, so restoring the states from the DB should be fast and should not add any writes to disk (to not stress SD card). Still it can sometimes come handy to be able to control the persistency per entity.

To give a perspective to my viewpoint, I am not using HA as a main home automation system for my house because when I was constructing my house, HA was a little baby of Paulus not ready for the task of controlling critical home stuff like all lights, power sockets, heating, gates, etc., etc.
The commercial home automation system I use as a main system does support (per entity) opt-in of the state persistence over system restart and I have a few comments to this feature using it for 6+ years:

1. I can’t imagine creating more complex automations without state persistency.
2. My only issues with the state persisting over restart always came from the fact it is opt-in in my system, so whenever any automation broke it was always me forgetting to enable the state persistency for some entity
3. Human beings naturally expects that object state in the system only changes when the real object changes, not when the system does some maintenance task (i.e. reboot). So when writing the automation they do write it the way that they rely on the fact the state will only change when the object changes. That’s why this WTH has so many upvotes I guess

So, I am loving HA and using it for almost 3 years now, but as I already have a main system which has state persistency I haven’t faced this restart issue much apart from few WTH moments when checking stuff in the UI which didn’t make any sense.
For my use-case HA servers mostly as a proxy, integrating stuff I either built myself or bought and which are incompatible with my main commercial system. For that I mostly don’t need persistency as the source of truth is not HA but my main system, which do have the persistency.
I seriously can’t imagine having to use HA as the only system as this lack of persistency would make a lot of things I do either impossible or much more complicated.
Also me being technical I can understand what happened and why, but other family members would be really confused by these state changes caused by restarts.

At last, I would like to get back to the dangerous factor of the possible state change caused by the HA reboot.
Imagine some powerful tool powered by or even controlled by HA. Someone would be cleaning it and suddenly it will turn on injuring him, just because HA rebooted and changed the state of it’s power from off to on.
This is not something imaginary. I have exactly that real issue with HA controlled smart socket powering my 3D printer.
Often when HA reboots that socket turns on powering up my printer. I agree 3D printer is not that dangerous, but still can go rogue and burn your house down…
I originally had the socket running Tasmota, configured to either remember the switch state or start with switch turned off. After two years of random issues I thought it must be Tasmota and a month ago flashed it to ESPHome, configured it with RESTORE_DEFAULT_OFF and hoped for the best. To my disappointment the issue is still present.
After reading the second part of your statement I now blame HA causing the socket to turn-on which is really unfortunate.

[Mincka]

Thanks, and to show these entities in a Markdown card, presented with the most recent changes at the top, with relative time:

## Latest events:
{% set entities = state_attr('sensor.nominal_change_history', 'changes') | dictsort(false, 'value', reverse = True) %}
{% for entity in entities %}
  * {{ states[entity[0]].name }}: {{ relative_time(state_attr('sensor.nominal_change_history', 'changes')[entity[0]]| as_datetime)}} ago
{% endfor %}

[Ildar_Gabdullin]

Since trigger-based binary sensors do not change their states during HA reboot - then why last-changed is reset on HA reboot?
Following a logic described in this thread - last-changed is reset since a state is changed - but here a state is NOT changed, right?
Asked about it here too.

[CentralCommand (Mike)]

So I think there is a case to be made that things which restore state (like trigger template entities) should also persist and restore last changed and last updated. It’s trickier with the many entities that do not restore state but for those that do I think the how is straightforward.

However there is an issue which is mentioned quite a lot in this thread. on and off may be the only states you care about but they aren’t the only possible states. There is also unknown and unavailable. And when the template integration is unloaded all it’s entities briefly become state unavailable. You can see this very clearly if you do “reload templates” from the UI while watching your template entity in developer tools.

Therefore that entity actually did have a state change. It briefly went to unavailable before returning to the state you’re interested in. Which is why what probably would need to happen here is for a new field to be created which tracks the last time an entity changed between two “nominal” states. “Nominal state” essentially being a state other then unknown, unavailable or any other more system states added in the future. This term is an unofficial one, it’s just what people settled on in the thread above.

[Ildar_Gabdullin]

I do agree about the “templates reset” case.
Here what happens with 2 template binary sensors - a “normal” and a trigger-based (both depends on some “input_boolean”, the 2nd is triggered when this “input_boolean” changes):

The “input_boolean” on the top, the trigger-based sensor - on the bottom.
So, both sensors become of “undefined” state (even not specified here - may be “unavailable”, unclear for me), the “normal” sensor becomes “unknown” at HA reboot, but the trigger-based sensor does not change it’s state at HA reboot.
So, I even do agree that both sensors may have reset “last-changed” after “templates reset” - but I do not agree that the trigger-based sensor have “last-changed” reset on HA reboot.

[Mincka]

I share my recipe for a lovelace card that works well with TheFes trick here:

How I "fixed" the real last changed status of sensors (restart/reload resistant) on my cards

Disclaimer: There’s nothing new that does not really exist in this post. It brings together ideas and recipes from other contributors. I’m just sharing them together. Attributions at the end. In this post I explain how I get the real last open/close status of my doors and windows, and the real last detected for PIR. I share the code for dashboard that shows their real last changed and status, and they are also ordered by last changed (optional). Result: [image] You can either use one group …

[Scoff (Scott)]

I’ve been using the solution posted @TheFes for a few days successfully (thank you!), however I’ve noticed the following error message in Studio Code Server:

I believe this has only started showing since updating to 2023.5.3 and Studio Code Server 5.6.0

[TheFes]

Just ignore it.

[GH2user]

Maybe we should raise an all new WTH for a solution.

What about adding two attributes to the sensors;

• last_time_on
• last_time_off

which could be used for doors, alarms and everything with an on/off behaviour.

[1mfaasj (Bowa)]

Does this line needs to be below the template sensor?

{{ state_attr(‘sensor.nominal_change_history’, ‘changes’)[‘input_boolean.test’] | as_datetime }}

Or does it have an own template sensor? Hm I’m not sure how to implement it.
Thanks!

[TheFes]

That line is how you get the data out of the template sensor again.
You can use it where you want, but if course that place needs to allow template sensors

[1mfaasj (Bowa)]

What I meant; Do I need to put the line below the rest? like this

or do I need to create a seperate sensor, like this:

  - sensor:
      - name: "test"
        state: "{{ state_attr('sensor.nominal_change_history', 'changes')['input_boolean.test'] | as_datetime }}"

and when I have multiple entities, I need to put it here twice right?

[TheFes]

It’s not part of the template to store the data, so you should not put it in that template sensor.

You should put it where you want to use the stored datetime, which can be anywhere where usage of templates is allowed

[1mfaasj (Bowa)]

Hm do you have an example? I’ve tried this one but the sensor is telling me unknown.

  - platform: template
    sensors:
      lastchangedsomfybadkamer:
        friendly_name: "Last changed SomfyBadkamer"
        value_template: >-
          state: "{{ state_attr('sensor.nominal_change_history', 'changes')['cover.somfybadkamer'] | as_datetime }}"

or is an input number (helper) a better way?

[1mfaasj (Bowa)]

Not related and looks like a gpt answer?

[burg93 (Adam)]

I’ve seen a few spam bots on here recently, if it’s obviously not helpful it’s best just press report as spam on the post.

[mekaneck (Rick Auch)]

Can I give this topic another nudge? I have a suggestion that maybe makes this more likely to be implemented:

Add two fields to the states table in the database:

1. last_communicated_state:
   • This will normally just reflect the same value that is in state. However:
     • It can only be unknown upon sensor creation if the sensor has a state of unknown or unavailable.
     • Once the sensor has any state besides unknown or unavailable, this value will reflect that state.
     • If the sensor state ever reverts to unknown or unavailable this value will not reflect that. It will not be changed and therefore retain the previous value.
2. last_communicated_state_change_ts
   • This will be a timestamp which changes any time the last_communicated_state has its value changed.

Having the last communicated state in the database prevents having to do lookups when a sensor gets a new state and needs to determine whether this is a “real” change or not. It is also useful in scenarios where users want to ignore unknown/unavailable and just use the last known “real” value. Also it makes it clear that there is no request to change the functionality of the existing last_changed_ts and last_updated_ts fields.

The downside is that I have no idea how much extra storage this would require adding to the database.

Also, in regards to naming of these two fields, I don’t think it matters what it is called. Here’s some other ideas for people to chew on:

• last_known_state
• last_monitored_state
• last_confirmed_state
• last_nominal_state
• last_measured_state
• last_state_that_is_neither_unavailable_nor_unknown
• any_parameter_name_that_the_people_who_create_and_approve_the_PR_agree_on
• ...

[burg93 (Adam)]

Is WTH running again this year?

It might be coming around that time to get this yearly widely requested feature raised again!

[petro]

no

[burg93 (Adam)]

Ah that’s a shame, thanks for confirming though!