Strange Behaviour: Template sensor not being logged in recorder/database. "Show More" also odd

onepointbrewing · June 5, 2023, 7:27am

Hey brains trust.

Either I truly dont understand or my template sensors DONT log/store any historical data. Any ideas why??? I possibly have a fundamental misunderstanding as to how the recorder works or Ive broken something. As a test - Ive just switched to mariadb as the same behaviour also was exhibited under sqllite

Apologies for all the pictures in advance - i just dont know how to explain it adequately…

.
I have a device based sensor based from a real device that DOES does appear to store/log/record historical data.

sensor.shellyplus1_80646fdaca0c_temperature (it would appear that there are caveats to this though)

So as an experiment I created a template sensor using the same temp sensor to compare the way the history/recorder works under - platform: template

The template sensor:

  - platform: template
    sensors:
      fv1t_temp:
        unit_of_measurement: "°C"
        value_template: "{{ states('sensor.shellyplus1_80646fdaca0c_temperature') | float }}"

So when clicking on the cog icon of the real shelly temp sensor this is what is dislayed (to be expected):

…
|
But when I click on a templated version of the same sensor I get this:

Note the timestamps in real shelly sensor arent reflected in the template (nothing prior to 14:12:11) when the shelly obvioulsy had valid data (approx 28.5-ish). Also note that the state in the templated sensor says 28.38degC but the scale says 0-1.0 and theres no graph??? Additionally note that the sensor shows NO prior history to 14:12:11 despite the template being active during that same period and the shelly showing data from 12:30 onwards.

NOTE: the template was active from 12:30. Does this mean that the states of template sensors ARE NOT recorded???

|
Now on to the history of the shelly (see picture below) which Isad had a caveat:

So clicking on the shelly temp entity you see a state/history graph with valid data from 12:30 through to 14:20 (see first pic). However, if you head to “show more” and specify a date and time range that is also within that period (in this case 5 June 12:00 to 5 june 15:00) there is no history BEFORE the point at which you pressed the link in lovelace for the history (in this case 14:30:22) despite the sensor showing history from 12:30-14:20 on the previous page. AND it would appear that when it does show history, it only shows it on a state-change

Any clarification as to why this happens - id be eternally grateful

onepointbrewing · June 5, 2023, 12:54pm

Upon further investigation which created more confusion and no resolution:

It would appear that things dont get logged properly in the db if you dont have state_class: measurement in a template entity (template)

see: unit of measurement

unit_of_measurement string (optional, default: None)
Defines the units of measurement of the sensor, if any. This will also display the value based on the user profile Number Format setting and influence the graphical presentation in the history visualization as a continuous value.

.
So I changed my template to the modern version of the templates that support state_class: measurement as well as device_class: temperature

New template:

template:
  - sensor:
      - name: fv1t_temp
        unit_of_measurement: "°C"
        device_class: temperature
        state_class: measurement
        state: "{{ states('sensor.shellyplus1_80646fdaca0c_temperature') | float }}"

and this is the result:

HOWEVER/BUT/STILL GOT A PROBLEM

click on “show more” = No state history found

I cant believe nobody has discovered this already…

The reason for asking this is because Ive been trying to get the - type: custom:apexcharts-card custom card working and it seems to look at the tables in the db that arent referenced/recorded (which is the reason for me investigating all of this in the first place).

While that “possibly” is a flaw in the apexcharts card - it appears that HA seems to have a fundamental flaw between looking at the “cog” info and the “show more” state history when it comes to recording/logging template entities.

Unless I truly dont know what Im talking about…

Am humbly hoping someone smarter than me can clarify and or possibly correct me.

Otherwise its off to github to report an issue

petro · June 5, 2023, 1:00pm

Nothing in the system is retroactive. When you add an entity, it starts recording from that point forward. So, when you added it, it was showing you it’s current history… which is basically nothing.

All new entities behave this way, not just template entities.

EDIT: Also keep in mind when you click the show more, it has a default time range. It will say “No state history found.” when you don’t have data in that time range.

TLDR: Not a bug, lower your history expectations

onepointbrewing · June 5, 2023, 1:13pm

hey petro
I do understand what you are saying - if the template didnt exist then theres no history - however hence why I said that the template entity had been in existence as long as the sensor had been in existence however the template data was not recorded.

NOTE: the template was active from 12:30. Does this mean that the states of template sensors ARE NOT recorded???

It also does not explain why “show more” does not show data within the same window on the template entity (even within a shorter timeframe)

This has data

“show more” same timeframe = nothing

petro · June 5, 2023, 1:17pm

There are no state changes in that time. Your sensor is most likely unavailable.

The history will show you history. The quick view shows you long term statistics, which is calculated based on the history.

Lastly, if what you’re saying is actually true (your template has existed the whole time as your sensor), then you didn’t include the sensor in recorder. That’s done via the recorder integration.

petro · June 5, 2023, 1:19pm

onepointbrewing:

template:
  - sensor:
      - name: fv1t_temp
        unit_of_measurement: "°C"
        device_class: temperature
        state_class: measurement
        state: "{{ states('sensor.shellyplus1_80646fdaca0c_temperature') | float }}"

You’re missing default on float, so this will be unavailable on startup and it will fail to load from that point forward unless you reload template entities while the system is running.

onepointbrewing · June 5, 2023, 1:20pm

unfortunately that is not the case…

Available and online just constant temp (on purpose)

petro · June 5, 2023, 1:23pm

again, that’s LTS which will show you a calculation based on an interval

petro · June 5, 2023, 1:25pm

there’s 2 types of data in HA’s recording. Long term statistics and short term history. Your short term history defaults to 10 days of history and if you implement the recorder integration, you can filter out what’s stored in short term history. Long term history is a different table that stores information forever.

When you click ‘show more’, you’re seeing the short term history data. When you don’t, you’re seeing long term.

onepointbrewing · June 5, 2023, 1:29pm

Hey petro - thanks for helping me on this one - I am truly diagnostically exhausted in trying to get this to work. Am very appreciative.

You’re missing default on float, so this will be unavailable on startup and it will fail to load from that point forward unless you reload template entities while the system is running.

Not sure what you mean.

Also, the “native” shelly temp sensor ALSO has no “show more” data within a valid timeframe

“show more”

petro · June 5, 2023, 1:30pm

Yes, as i’ve said about 239048230948023984 times, that means you aren’t recording this in recorder.

There’s nothign else to the system.

If you override recorder, it will not record things unless you specifically tell it to record them. If you run recorder with default settings, it will record everything.

Again, that’s just for short term history. I.e. the history tab.

petro · June 5, 2023, 1:34pm

when you make a template sensor and you’re using float, float requires a number as the value coming from the source sensor.

If it’s not a number, it will error.

On startup, most sensors are 'unavialable', meaning your template sensor will error on startup because float cannot convert 'unavailable' to a number.

onepointbrewing · June 5, 2023, 1:35pm

sorry for not quite understanding - i truly dont want to be an imposition

my config.yaml

recorder:
  db_url: mysql://homeassistant:xxxxxxxxx@core-mariadb/homeassistant?charset=utf8mb4

I would have expected it to record everything.

Stupid bit is that it did the same behaviour in sqllite

I have 30+ climate and temp sensors as well as templates for the climate set temps

onepointbrewing · June 5, 2023, 1:37pm

the entity is a number to begin with (its from the shelly)

petro · June 5, 2023, 1:37pm

Yes, I have shelly devices and I can tell you that they do not start up with a number.

onepointbrewing · June 5, 2023, 4:03pm

Further to our discussion this evening on discord - cfm my system has a problem.

I just spun up a VM with same versions of HA installed and used identical template in both as well as ensuring both have cleared/NEW sqllite db using the exact same shelly temp entities.

My BIG system (the one with 42 climate entities and hundreds of templates) = faulty template recording. The new VM (with only one shelly temp sensor and one template) = good template that mimics the shelly sensor regardless of state/reboots etc (it actually records a blank if its unavailable)

I will attempt to explain this as an issue on github tmr morn once both systems have been running overnight. (wish me luck with the explanation)

tom_l · June 5, 2023, 4:04pm

Probably because nobody else has this issue. It’s just your system.

I have no idea what your issue is, but to save you some confusion:

Template sensors only update when the source sensor updates.
The More Info pop up graph displays 5 minute aggregate LTS (long term statistics) if a state class is defined. If no state class is defined it displays state history.
The History panel only displays state history.

Have you changed any recorder integration settings?

Particularly excludes.

petro · June 5, 2023, 4:06pm

Yes, I explained this to him on discord. This is an issue with his installation.

EDIT: Seeing that this is reoccurring on his other VMs, I’d wager this is related to how he’s creating his VMs.

onepointbrewing · June 5, 2023, 4:09pm

Peviously - yes

Diagnostically right now - no. Plus have deleted both mariadb and sqllite db’s many many times in the last 14 hours (been working on this all day) in order to get to the bottom of this.

My config.yaml has NO recorder entries at all on a deleted database pretty much on every reboot.

petro · June 5, 2023, 4:09pm

Then nothing is recorded. At a minimum, you need recorder:, unless you’re running default config and then history enables it.