I call recorder.purge service, asking to keep 2 days worth and to rewrite the whole database. select count(id) from statistics; still gives me the same value as before. No idea how long this is supposed to take either, but mariadb doesn’t appear particularly busy.
did do something: database dump went from 669MB to 250MB
Ok so this is new:
Logger: homeassistant.components.recorder.db_schema
Source: components/recorder/db_schema.py:491
Integration: Recorder (documentation, issues)
First occurred: 13:13:06 (2 occurrences)
Last logged: 13:13:06
State attributes for sensor.solcast_forecast_data exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
I don’t mind that the attribute is not stored but I would like to exclude it from the recorder so I don’t get the warning. However I can only exclude the whole entity and I want the entity state history.
If you are going to exclude attributes from the database can we have that ability (to exclude attributes) in the recorder options?
If thats coming from a core integration, I’d open an issue as its also abusing the state machine a bit by putting that much data in there. That all has to be serialized across the wire (and has to be processed every time it changes), and the frontend has to hold all of that in memory as well.
Integrations can make a recorder platform to have specific attributes excluded from being recorded. Example: core/homeassistant/components/unifiprotect/recorder.py at 3e5e937541dd368f3d70c7557ceaa7a07fd284e2 · home-assistant/core · GitHub
At least for my older Celeron synology box it could convert ~17000 rows per second so I wouldn’t expect it to take more than 2 minutes for 1467600 rows.
It’s a restful sensor:
- resource: https://api.solcast.com.au/rooftop_sites/41bd-6b13-6809-bd99/forecasts?format=json&api_key=<redacted>&hours=72
scan_interval: '00:30:00'
sensor:
- name: "Solcast Forecast Data"
force_update: true
value_template: "{{ value_json.forecasts[0].pv_estimate|round(1) }}"
unit_of_measurement: kW
json_attributes:
- forecasts
Attribute value: https://hatebin.com/mevpnuunlq
forecasts:
- pv_estimate: 2.7088
pv_estimate10: 1.7846
pv_estimate90: 3.7482
period_end: '2023-03-08T04:00:00.0000000Z'
period: PT30M
- pv_estimate: 2.5868
pv_estimate10: 0.7306
pv_estimate90: 4.5601
period_end: '2023-03-08T04:30:00.0000000Z'
period: PT30M
- pv_estimate: 3.0853
pv_estimate10: 0.7394
pv_estimate90: 5.3877
period_end: '2023-03-08T05:00:00.0000000Z'
period: PT30M
- pv_estimate: 2.7455
pv_estimate10: 0.7306
pv_estimate90: 4.5867
period_end: '2023-03-08T05:30:00.0000000Z'
period: PT30M
etc...
It returns 30 minute solar forecasts for the next 3 days with 10% and 90% confidence estimates as well.
I only really need the next 24 hours. Is there a way to limit it?
It would have taken slightly longer than that. But indeed it didn’t take very long.
I can’t think of a way to limit the size of the attributes down without extending the integration to support some type of templating.
This is a case where I would probably make a new integration to process the data down to just what was needed.
The primary motivation for the change was to prevent database/storage failure scenarios from the system getting overloaded (as well as migration failure). The big offenders were storing sets of torrent data in attributes.
Alternatively we could revert the limit or raise the limit but that won’t actually solve the problem of loading all that data into the state machine, having to serialize it, store and update it it in the frontend every time it changes, and bloat the database. I don’t particularly like that idea as the system isn’t designed to store large JSON blobs of attributes and that puts additional burden on users to manually keep their database under control and developers to keep things working.
Ok, I’ll create a template sensor that copies the state and exclude the original sensor from the recorder.
I use the attributes with the Apexcharts generator function so don’t need the attribute history.
the same issue (storing JSON attributes into the recorder) also happens for the Alexa Media Player integration (custom component) - where the “next_alarm” sensor is providing a whole history of past and current alarms…
Not storing this into the recorder is fine for me - but … I would consider a different approach:
Since there was a statement somewhere, that devs should try to create sensors for their data rather than creating tons of attributes, I would consider, that NO attributes will be stored in the database at all.
When you really need to use an attribute somewhere, the integration should create a sensor - or you should create a sensor with an template… that’s an approach that can be found in many topics here on the forum if you really need to work with attributes at all.
It could also be considered to implement this with a global recorder setting, similar to:
db_url: *******
exclude:
entity_attributes:
- *
or: disabled by default, enabled by config flag:
db_url: *******
include:
- entity_attributes
That could be absolutely possible.
Sadly the versions of MariaDB on Synology are updated in loooooong intervals (to say it at least a little bit friendly).
If that would be indeed the cause, then this would be an unsolvable problem for synology users at the moment, as Synology doesn’t seem to adress this.
So it should be noted as this.
at these times I wish we had a way of checking our integrations/entities for being big offenders. I mean, many of us will have a rest sensor like Tom’s, or have some trigger based templates creating attributes with some serious extended content. When are they considered ok, and when are they excessive…
Even a rest sensor made to create template sensor on can be huge (a 48 hour energy cost prediction being a good example probably). How else could we do that?
Having been busy to get some rest sensors weather data to the frontend just this day made me realize that too uses more than a few words in its attributes…
so, to be or not to be an offender, that’s the question?
iow, are these:
considered to be worse than:
?
It should not exceed a maximum size of 16384 bytes
You can roughly estimate it by copy and pasting into a word processor. Use the word count. Each character (including spaces) is a byte.
thanks, now why does that ring a bell. can find a multitude of Google queries on that, but not yet find the real source for the limitation in HA
While I can see the appeal of the new UI change for switches etc, I personally want to see the history when I click on the entity, and being able to control things is very much a secondary requirement. In the new setup I now see the thing I don’t want and have to make a second click to see the history for each entity, which is very annoying.
Is there (can there be) a way to select the behaviour when clicking on entities - i.e. history or control. Preferably setting it as a global default, with an override per entity.
Try a different approach @nick.lawes
For controlling e.g. lights, which is a the most common case, I use e.g. Trådfri remotes in the rooms. They operate, whether there is HA running or not. Automations do their job in the background. Since the lights seem to be the area where those sliders rise their ugly heads, I have actually made controlling lights via UI needless and thus the whole UI is more or less for service & maintenance, checking occasional sensor data with 3rd party stuff like apex charts, history graphs etc., and stuff like that.
That’s to say, since I started to see ugly oversized sliders and similar “trendy” mobile-like stuff too often, I nowadays run HA more or less like PLC-style headless mode. UI is buried somewhere in the normal stack of tens if not hundreds of tabs in various browsers, from where I dig it out, when some point needs checking.
There needs to be freedom to choose. If I go to a shop to buy apples, and the shopkeeper gives me oranges saying they are better for you, I just don’t buy.
Speaking of freedom, It’s been a while since the last time visiting here. When I do not need to try to adjust my UI (controls) to my liking and have automations more or less in place, I do not need to spend my time here either trying to find ways to negate latest UI changes and can use it to something more productive. Also the tone of discussion here has along these UI changes changed to … not to my liking.
Free tip: One form of “plague” today is those ugly rounded corners everywhere. Latest probably being on Google’s Image Search.
If you have uBlock Origin or similar installed, create a custom filter:
*##*:style(border-radius:0px !important;)
…and see what happens 
No rounded corners anywhere. Even some loading “spinners” are rectangular
Checking back again… sometimes.
My lights etc are also fully automated by HASS, with Hue dimmers able to control the lights directly even when HASS is not working. I never touch switches or the UI for controlling them, which is why I don’t need or want to see light controls when I do use the UI. My heating is the same, as is the alarm etc.
I do look at the UI for current state and history, so I can keep my eye on what triggered and when. That way I can confirm things are working, and to help refine the automations.
For me, HASS should be doing everything without intervention, and the UI is just for monitoring and debugging.
I also don’t use the UI for configuring / editing - my configuration is kept in ansible so the more that can be done in yaml the better (although writing HASS jinja2 templates in ansible jinja2 templates can blow the mind :-). It’s another bone of contention for me when feature configuration gets moved from yaml to UI.
Looks like 2023.3.x broke my Lutron Pico BluePrint…again.
I have a simple BP for Lutron Caseta Picos and it seems that the recent updates keep breaking it’s functionality. It worked fine for months and then an update broke it until I commented out the “integration” line. It worked again for a few months since that change, but after the 2023.3.x release, users have reported that the device selector does not list any devices. I tested before updating to the latest and it worked fine. I updated to this release and it immediately broke. I didn’t see any changes listed for the Lutron integration so I can only assume this is a core issue. No error in the logs.
Can anyone suggest/post a link to where in github I should post an issue for this?
domain: automation
input:
pico_remote:
name: Pico Remote
description: "Select the pico remote to configure:"
selector:
device:
#integration: lutron_caseta
model: PJ2-3BRL-GXX-X01 (Pico3ButtonRaiseLower)
I no longer get the setpoint/target temperature trended on the History trend of my thermostat. Just actual and running.
These rest sensors no longer work:
- resource: http://homeassistant:8123/api/hassio/core/stats
verify_ssl: false
headers:
Authorization: !secret ha_api_token
Content-Type: application/json
scan_interval: 31
sensor:
- name: "Core CPU"
value_template: '{{ value_json.data.cpu_percent|round(2) }}'
unit_of_measurement: '%'
- name: "core_memory" # Customized as, friendly_name: Core RAM
value_template: '{{ value_json.data.memory_percent|round(2) }}'
unit_of_measurement: '%'
- resource: http://homeassistant:8123/api/hassio/supervisor/stats
verify_ssl: false
headers:
Authorization: !secret ha_api_token
Content-Type: application/json
scan_interval: 29
sensor:
- name: "Supervisor CPU"
value_template: '{{ value_json.data.cpu_percent|round(2) }}'
unit_of_measurement: '%'
- name: "supervisor_memory" # Customized as, friendly_name: Supervisor RAM
value_template: '{{ value_json.data.memory_percent|round(2) }}'
unit_of_measurement: '%'
I’m assuming this is because of changes made to the home assistant API to mitigate the recently disclosed security vulnerability?
Still no fix for the transition problem of lights 