Recorder domain exclude not working as expected

Configuration
1

I have many of these type of logs in my state_attributes table:

{"type":"browser_mod","last_seen":"2022-04-23T21:44:48.128091","deviceID":"07c1ac69-44667gh77","path":"/lovelace/upstairs","visibility":"visible","userAgent":"Mozilla/5.0 (Linux; Android 7.1.2; Fire Build/NJH47F; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.132 Safari/537.36","currentUser":"xxx","fullyKiosk":false,"width":1103,"height":689,"battery_level":23,"charging":true,"darkMode":false,"userData":{"id":"62df499525574a619ee6254448b84b85","name":"xxx","is_owner":true,"is_admin":true,"credentials":[{"auth_provider_type":"homeassistant","auth_provider_id":null}],"mfa_modules":[{"id":"totp","name":"Authenticator app","enabled":false}]},"config":{"command":"update","name":"upstairs","camera":true}}

And I am trying to exclude them from saving to the DB since they are useless and grow my DB by a few MB per day.
However my exclusion doesn’t work as it still logs these:

db_url: !secret mariaDBurl
purge_keep_days: 7
exclude:
  domains:
      - browser_mod

Can anyone see anything wrong here?

2

Is browser_mod a domain with you?
Try below in Dev > Templates

{{ states | groupby('domain') | map(attribute='0') | list | join('\n') }}
3

Hmm it seems like not which makes sense why my exclusion does not work

alarm_control_panel
automation
binary_sensor
button
camera
climate
cover
device_tracker
group
input_boolean
input_button
input_number
input_select
light
lock
media_player
number
person
remote
script
select
sensor
sun
switch
timer
update
utility_meter
vacuum
weather
zone

Any idea how I might out its domain?
It is this integration here: GitHub - thomasloven/hass-browser_mod: 🔹 A Home Assistant integration to turn your browser into a controllable entity and media player

4

Not using this mod but from what I can read…
You could try to add a prefix for your mods and then exclude on entity_globs using wildcard.

image
image828×212 7.21 KB

image
image736×363 23.6 KB

5

Main question would be…any particular reason why you would want to filter these out? Is youd rec-db too big ?

6

Seeing about 140000 records in my state_attributes DB file so it’s spamming it quite a bit.
Only noticed it as my daily backups are growing in size a few megs each day.

7

Yep…well… unless you are adding more and more ‘stuff’ the db should at some point stabilize as it purges every x days ( I assume you have set this to reasonable value or took default)
On the other hand…why store stuff that no-one uses so I also prefer to keep (any) db clean
EDIT:
I ‘only’ have 64k records and a 10 day purge

How to reduce your database size and extend the life of your SD card - Community Guides - Home Assistant Community (home-assistant.io)

8

I myself am playing with the Recorder and am still learning, but you could do

recorder:
  db_url: !secret mariadb_homeassistant_url
  commit_interval: 60
  auto_purge: false
  include:
    entity_globs:
.
.
 exclude: 
    event_types:
      - browser_mod

I have excluded all the messages from hacs by filtering event_type.
But it does not seem to support globs ("*"), so you need to check in your DB if they are only of the event type: browser_mod or if they have other types also.

I know that the state_attributes table does not have event_type, but maybe if you check the events table, you can find the matching event_type and can exclude it.

Might work, but might not work. Just worth a try.

10

Can you please let me know why you have auto_purge to false? Do you have the separate purge setup in your system …and then why (trying to learn)

11

There is a lot of stuff by default being recorded, ton of them unnecessary.
So finally i ended excluding everything and only including what i really need.

12

I actually may like this approach best … !! :+1:

13

I disabled purge because I want to keep my data in full detail indefinitely (by default, all raw data is purged after 10 days).

However, this does require that you carefully exclude/include in as much detail as possible. I have only included the few entities I want to keep. Everything else (including events, add-ons etc.) is excluded. Also, all logging is set to fatal.

I would not recommend disabling purge if you have not included/excluded on an entity level.

Also, most users seem to prefer the ‘statistics’ to the actual raw data. Statistics is an automatic aggregation that home assistant creates from your raw data. Kind of like an hourly or daily aggregation of data into a reduced number of data points. So if you want to see your power consumption of the last year, you probably don’t need data points for every 5 min (raw data of the sensor, which of course will slow down the data plotting). You will be happy with one point per day (statistics created by home assistant, resulting in only 365 data points to plot).

But I would like to keep all data and then purge once I know what I want to keep and what not. I am monitoring database file size also, so I might enable purging or reduce the amount of tracked entities further at some point.

There is a lot of stuff by default being recorded, ton of them unnecessary.
So finally i ended excluding everything and only including what i really need.

Precisely, only include what you want to keep and exclude the rest.

14

This is not the idea behind the HA/recorder-db… for longer storage one should use e.g. influxdb.
Can you (learning all the time) please explain me what you’d like to keep indefinitely and ‘why’?

EDIT: I am still using mariadb as recorder and recently found out there is no real benefit of use for me and that I too need longer term storage, so I am contemplating to go back to native/sqlite and use influx

15

Well, MariaDB is, if I understand it correctly, suited for long-term storage. So just like InfluxDB it can hold long term data.
Reasons I opted for MariaDB:

  • I prefer the SQL syntax to InfluxDB

  • As the integration of InfluxDB into Lovelace cards is basically non-existent, I don’t like InfluxDB. It more or less requires you to use Grafana to plot old data. With MAriaDB you can plot directly from ApexCharts and all other cards (as far as I know… my database was deleted when I made the switch last week, so I am hoping this will work as planned).

I am only tracking data on power consumption, temperatures and other information on usage or reliability. So nothing like voltage, current, power_yesterday, events, add-ons etc.
I just added a few more, but it was only around 20-30 entities in total. Everything else is not being recorded.
Only exception: I cannot get the events table to stop recording. This I am hoping someone knows how to do (events records state_changed which is useless information given that the actual change is already being recorded in states table).

16

Well… you have a point there as one needs to create sensors to extract influx stuff

I too struggle a bit (old school sql habit) and for now I only use it for long term storage (not affecting my HA db use)
Still…and here I would like to here from other people as well, using the core db of HA for long term storage is a bit odd so I gather. Maybe a secondary Mariadb can be setup instead of the influx one?

17

I would be interested to hear if there is a reason for this, too. Maybe I am missing something and my approach will cause problems.
But at the moment, I don’t know why using the Recorder should be problematic.

HA uses the recorder to store its long term statistics. So the duration is not an issue.
And I see no difference between short and long term storage since all HA Recorder does is add new rows to an existing table.

There might be an increased risk of corruption. That I do not know. But I am creating backups every few days, so I would only lose less than a week of data. Plus, the normal SQLite seems to have been quite prone to corruption and no one seemed to mind.

18

Another solution might be to make the purge more specific, i.e. I cannot care less for events (my largest table)

19

This is not supported by HA, as far as I know. You would need to run an automation to purge the DB for you. Not sure I want to do so. I am trying to streamline my system, not increase the amount of things I need to maintain against breaking changes etc.
Plus, if you change your database you might need to change your syntax. If you exclude/include in Recorder, none of this is necessary.

But yes, I have not found a solution to stop the events table from being populated on state_changed. Hoping someone will be able to support me on that in the other thread.

20

I meant that HA as a solution would provide a more granular way of purging data, i.e. include/exclude events/stats/etc. This one can then run as and when needed…

21

Doesn’t seem to have helped, still logging and I agree having some sort of event purge would be great to have.