GUI Recorder – manage recorder & clean up your DB, no YAML

So i just install this Integration, and i might have “swiped” through the instructions to fast, and missed some.

~~It seems to work, thou in my gui-recorder yaml is Nothing ! ~~
~~In configuration.yaml it has commend out my Include-entities(No big deal) , ~~
But it has wipe out my excludes(vanished)
Wasn’t those suppose to end up in gui-recorder.yaml ? , can i just copy n paste them into this file , or do i have to add the “missing” excludes in the Gui ?

Sorry, i looked at my Logbook :laughing: … i never had any excludes in Recorder, only includes !
So if i get it right, i can just move the entities into the entity_glob block ?

Well i’ll find out,that’s what i did now … Need a little cleanup now

Hmmmm apparently Not, A Lot To Clean Now

  • In HA if one only have Includes, then everything else is Excluded, seems like this Tool Don’t follow this rule ???

Is it me, or Could you write a clear Description in Github, in regards to the rules and exceptions , as i have since long only had Includes, in my DB … Now !
Booom ( cause these includes was not transferred to the GUI )

I did follow the Guide !

NOTE: And Now i deselected ALL(almost :innocent:) Devices and Entities in the GUI, Purged all excluded

  • Please don’t tell me that the Default for this Tool is “Include” so ever time i add a Device it will automatically get “Enabled” ?
    I want to manually include all Devices/Entities ( When i eventually install a Device and Discover which entities it provides( almost 2/3 of my entities is disabled-entities(irrelevant, redundant, bs), and not even half of the Enabled is recorded ) … Currently

Right, I figured that was the issue. I tend to never look at the other two (wal and shm) files because those go away on shutdown or whenever they’re rolled into the .db file. The .db file isn’t increased by the full size of those two files, and tends to stay pretty close to the size it was before those are merged in.

So I figure the .db file size alone is the “real” size of the underlying data for my purposes. I guess it depends on what you’re looking for. I want to know how much data Recorder is actually retaining. If you were severely constrained by limited disk space, I suppose the combined file sizes could have some value.

Well - I used to think very much like you, but with this integration it is definitely time to evaluate that approach.

I very often installed a new gadget and forgot to include right away.
Only when I wanted to look at the data, I realized the data wasn’t there - and then I finally included.

With this integration however, you can install the new gadget, forget about it for as long as you like - and when you feel like it, you look at what it records.

Since you can now easily toggle of the ones you don’t need - AND purge the database from the extra data that is there - well what is not to like?

Always better to have something you don’t need - and you can get rid off - than find out you are missing interesting data?

True, but it seems like i have to “start over again”, as My individual Entities under entity_globs “Block” doesn’t seems to “trigger” a recording any longer, No Records for a day now :thinking:
I wonder why , was just sitting and go through it again, and added a few other separate “entities” under the entity_glob in the gui-editor-> to file, and i’ll wait for “record counts”
Yes i “enabled them in the GUI” , which “confuse” me as it says “Exclusions”, but in fact it should be other way around or ?

Beside it’s kind of “redundant” to have a HUGE list of excluded “Entities” in the gui-recorder-yaml

And then a minor Block in the bottom, with included “Entities” under the Entity_Globs Block

Martin explained that above - when I had the same problem.
If you start with includes, you can only exclude sensors that are in your includes.
You cannot include any other stuff that wasn’t in your includes in the first place - even when the button in the integration says otherwise.

So basically you have to change your approach.

Make a config.yaml that has no includes or excludes - so everything gets recorded and then use the integration to switch stuff on or off.

My solution:
I deleted Gui-recorder
I modified my config.yaml and removed all includes and excludes
Restart HA
Install Gui-recorder and make sure the manual excludes is empty

Now the recorder will record EVERYTHING and the exclude/include buttons in gui-recorder do exactly what it says.

1 Like

I give it another try, cause it still aint recording anything :blush:
i even Purged everything anyways ( That worked fine ) :laughing:

2 Likes

And with over 26000 ents thats a total 100% nonstarter for me. There’s over 400 on the electrical system alone.

I am not. Starting over.

Glad I read first.

1 Like

I did read, but apparently missed Alot ! , like having a gui-recorder.yaml with + 3000 entities :hot_face:
Feels like OP needs to rethink

Now i followed above procedure, and needs to traverse through all Devices & Entities, which rapidly starts to record As they are all enabled by default

And in future, when installing new Devices with + 20 “redundant/irrelevant” Entities , i first have to disable them (Already made records, most likely ) so then i need to go to the GUI and disable them And Purge the DB, for all Excluded

Ok, there is a “Deselect All” so i could just use that( Now, not when installing a New Device ), and then Select the ones i want to Record , oh and then Restart HA for the 10 times, and then … Needs a restart for all changes in the GUI ofcause … and then …

As mentioned i have (Had) way more Entities Excluded, one of 3 reasons to only Include, so now i can always look in the gui-recorder.yaml if i want to see all Excluded :laughing: , my goodness

Well - how many includes do you have?

You can initially switch everything to exclude in the gui-recorder integration with basically 2 clicks.

And then add your includes again?
Many of which probably won’t be a single entity, but a bunch that belong to a device.
You can switch whole devices on and off with one click.

So not sure if it is really as hard as you think ?

:slight_smile: :slight_smile:
It is not as bad as you think mate.
How much data is it going to generate in a week or so until you get around to look at it.

My database was at 15.3 GB when I installed gui-recorder.
Brought it down to 9.5 GB over the weekend - so definitely worth it.

Yeah maybe it’s just another “pattern” one have to get use to, but i still Strongly suggest that new added Devices/Integrations etc is “Excluded” by Default ( Or an Option to have it this way )
Because of mentioned above, some Devices/Integrations have/adds a bunch of (bs) Entities, which i first have to disable in the integration(always) and then in The Recorder GUI, where most likely the newly added Entities is “Included”, before i disable them, and then i have to exclude these, and the ones "enabled in the integration, which i don’t want to record
So maybe, just maybe i eventually could find this as “a pease of cake” , right now …i need a brake :joy: , and it feels like that i most likely goes Back, to “Include” Only in Native-Yaml, Pease of cake in i.e notepad++ over Samba-Share
Not alot of restarts, and not alot of Purge, every time …

1 Like

I’m just trying to figure out why it doesn’t use the same paradigm and therefore import the existing configuration then let one mod from there. If it did that. I’m fine. But I’m in the same boat. Dupes multiple things that might be considered dupe. Some needs to be off all the time because they flood.

@Frank_Beetz I get you’re trying to help but sometimes when you see a list that’s as long as your arm you’re like no I’m not even going down this path. I generally do like the idea. But no I’m jot going through a separate configuration. Simply not interested it should 100% import what’s there as is and do the translation if any is required.

1 Like

My two cents after reading the comments…

If I had a lot of entities (looking at you @nathancu :star_struck:) and I already had recorder filters in yaml, I wouldn’t want to start over either & I probably wouldn’t use this.

I only have ~1200 entities and my db is small, but it’s been increasing lately and I needed to know why and nip it in the bud. I was actually reading the official Recorder docs this week when I came across this thread. I had never done any filtering, so before going the yaml route I decided to give gui-recorder a try. I didn’t have to start over because I never started…in my use case, I think it’s pretty awesome!

2 Likes

In most ways it is, i also like the “overview” and features, hopefully a few “GUI” and “useful” features can be added

Not sure if there’s any different in how HA reads / starts the Recorder during a restart, and it doesn’t seems to affect the time of a restart, but it doesn’t look awsome with a list of all Excluded Entities in gui-recorder.yaml

And Yes, i’ve finished the " Exclusions " :grin: , or i should say " Inclusion ", as i De-selected all, and Selected those i wanted( Need ) , so in a week or so, my DB should be back to “Normal”

2 Likes

One of the reasons is the Default Recorder Setting ( Include All ) , and No GUI for i.e New users and others, so they can get an idea What " Include All " implicate.
So people start i.e adding Devices/Integrations, without disabling entities you dont want/need, then they make a template or 2 dozen or more, and “Trim” their devices/entities so they get i.e Graphs updated (Smooth) i.e every 5 second, and even worse, a wrong template with a loop, etc.
Most work ( For awhile ), but noone have initially any ideas why the DB is huge, or increase, because they are not encurage to “Think” about the DB … Again this “DEV Complex” Naaa that’s to complicated for average users, we fix it for them, by i.e reducing Attributes etc. etc. and optimizing the DB in other ways etc.

With a GUI like this, People can see that it’s nothing but a “storage/closet/drawer”
The more shit you put into it, the more it increases, in most cases you have tons of info, you don’t ever use or need, or even knew you had there.
And in the same time, it’s easy to get an overview over which Devices/Entities “misbehave”
You can have half a dozen of same Device-type(same brand) , thou one is spewing records(and maybe drain it’s battery) … but seems-less work the same as the others, which live a quite stable life :grin:

2 Likes

Exactly! I had no clue until I started reading the Recorder docs this week and I was like, whoa! The gui-recorder definitely makes it easier for me to see what’s going on.

3 Likes

I totally agree that the problem we’re all trying to solve here is the “include all” default in HA, with no way for a beginner to even know they need to do something, or their database will become unmanageable.

There are different approaches. If I knew what I know now, I’d probably start from an “exclude all” position and only include entities I wanted. Instead I started paring down the database by excluding entities which were the worst offenders, one by one, after my database was already bloated.

It is great to see some movement in the direction of a solution which will at least help some users. Hopefully this takes off and eventually supports the “include” philosophy better. But for a beginner, this is a very useful tool and a good start toward understanding and managing Recorder. This is the sort of thing every new user should be encouraged to learn.

1 Like

I did the experiment and started over.

Thought the „heavy“ entities would show up rather fast to exclude them again.

I worked with an include list in the past, but switched to exclude later as I had a few situations where I would have been very happy when history would have existed (but I didn’t include them).

Instead of importing my current excludes (which have more been made based on a „feeling“ instead numbers most of the time, I started fresh.

After a night, it was already very obvious, which entities will be the main drivers for blowing the DB.

Excluded them and purged their data.
So far a happy customer. :slightly_smiling_face::+1:

1 Like

Sorry for going quiet, the thread grew faster than I could keep up with. I also didn’t want to answer @boheme61 with a guess, so I went to read the code first.

Short version: he’s right, it’s a bug, and it’s mine.

If your config has an include: block anywhere, the recorder runs in allowlist mode, and the order HA applies is this one:

include.entities      -> record
exclude.entities      -> skip
include.entity_globs  -> record
exclude.entity_globs  -> skip
include.domains       -> record
anything else         -> skip

The toggles in the panel only ever write to exclude.entities. That is the second line, sitting above your entity_globs on the third. So the moment you switch entities off (and “Deselect all” makes it much worse) those entries land on top of your own globs and kill them. Nothing in the UI tells you that is happening. JC, that is why your globs stopped recording. Sorry for the wasted evening.

It fails the other way round too: switching an entity on only removes it from exclude.entities. If nothing in your include block matches it, the last rule still skips it. Green toggle, no recording. Same thing @phantomsphorever reported in #15.

And the part I like least: since 0.8.32 the migration copies your old include.domains / include.entity_globs into the Manual exclusions field by itself. So migrating drops you straight into the mode where the panel is wrong.

For now, if you need one specific entity recorded, a glob with no wildcards matches only itself:

include:
  entity_globs:
    - sensor.your_entity_id

And if you don’t actually need allowlist behaviour, deleting the include block and staying in plain exclude mode is the saner option today. That is basically what @Frank_Beetz suggested, and it’s why it worked for him.

What I’m doing about it: first a warning in the panel when an include block is active, so at least you can see which entities no include rule covers. Then the real fix, toggles that write to include.entities when you are already in allowlist mode. Only when it’s already active. I’m not going to invent an include block for people who don’t have one, that would break everybody else. No date, because I’d miss it.

@NathanCu, on importing the YAML as it is: fair enough, and redoing 26k entities by hand is not something I would do either. Since 0.8.32 the import does bring exclude.domains, exclude.entity_globs, exclude.event_types, include.domains and include.entity_globs across, verbatim. The one it can’t represent is include.entities, which is the same hole as above. So it’s one fix and not two. If your config is include based I would honestly wait for that release before touching anything.

Anyone about to migrate, please update to 0.8.37 first. 0.8.36 and 0.8.37 fixed two migration bugs that are worse than this one: a custom db_url being dropped, so HA quietly started an empty database, and steps 2 and 3 being clickable without step 1, which could overwrite your recorder config with the defaults. And take a backup, I mean it.

On new entities defaulting to excluded, @boheme61, @CaptTom and @Deedee all asked for it here, so I’ve reopened #7. It also comes almost for free with the allowlist work, since that last rule already skips whatever isn’t matched by anything.

@CaptTom, about the size: yes, that number is .db + .db-wal + .db-shm added together. I did it that way because it’s what the disk really holds, but with a big WAL it’s misleading, so I’ll split it.

@Frank_Beetz, @Thyraz, @Deedee, thanks for the patience and for helping each other out here while I wasn’t around. And thanks in particular to @boheme61, because that report looked like a misconfiguration at first and turned out to be a real bug I would not have found on my own.

2 Likes

No need to be Sorry, i was the one who wanted to test this New Integration :grinning_face_with_smiling_eyes: , normally i give such “new things” time, to grow “mature” ( Let others takes the pitfalls :joy: )
Yes i followed Franks Procedure, so i only have excludes in the Yaml-List and i works fine ( as long as i don’t open the file and starting to scroll , my right pointer finger starts to get old :joy: )

However i would love that one could have a Toggle so ALL New Entities ends as “excluded” as escluded ( Not that i think it’s possible , now that i run with only excluded, hint the reason i only had “included” in recorder.
Some integrations add +30-50 entities or more, where i mostly only want to record 1/5, or less
Ohh,i just read about #7 , so if i “redo” ( Not his week :slight_smile: ) i can use My Include “only” list, and no new( or others ) will be added as “selected” ?

And, thanks for your Work