Script to find items that no longer exist

[NigelL (Nigel)]

Hi @jnvd3b where are you seeing that message?

[Abeksis]

Not working - the old start, works great.

[Mariusthvdb (Marius)]

@NigelL

def process_group_entities(group, grouped_entities, hass, logger, process_group_entities, processed_groups): # logger.warn(“processing group {}, currently {} grouped items”.format(group.entity_id, len(grouped_entities))) processed_groups.append(group.entity_id) for e in group.attributes[“entity_id”]: domain = e.split(“.”)[0] if domain == “group”: g = hass.states.get(e) if (g is not None) and (g.entity_id not in processed_groups): process_group_entities(g, grouped_entities, hass, logger, process_group_entities, processed_groups) else: grouped_entities.add(e) # logger.warn(“finishing group {}, currently {} grouped items”.format(group.entity_id, len(grouped_entities))) def scan_for_ghosts(hass, logger, data, process_group_entities): target_group=data.get(“target_group”,“deaditems”) show_as_view = data.get(“show_as_view”, True) use_custom_ui = data.get(“use_custom_ui”, False) real_entities = set() grouped_entities = set() processed_groups= for s in hass.states.all(): domain = s.entity_id.split(“.”)[0] if domain != “group”: real_entities.add(s.entity_id) else: if ((“view” not in s.attributes) or ( s.attributes[“view”] == False)): real_entities.add(s.entity_id) process_group_entities(s, grouped_entities, hass, logger, process_group_entities, processed_groups) # logger.error(“{} real entities”.format(len(real_entities))) # logger.error(“{} grouped entities”.format(len(grouped_entities))) # logger.error(“{} groups processed”.format(len(processed_groups))) results = grouped_entities - real_entities # logger.error(“{} entities to list”.format(len(results))) entity_ids= if use_custom_ui: summary=“” for e in results: summary = “{}{}\n”.format(summary, e) hass.states.set(‘sensor.ghost_items’, ‘’, { ‘custom_ui_state_card’: ‘state-card-value_only’, ‘text’: summary }) entity_ids.append(“sensor.ghost_items”) else: counter=0 for e in results: name = “weblink.deaditem{}”.format(counter) hass.states.set(name, “javascript:return false”, {“friendly_name”:e}) entity_ids.append(name) counter = counter +1 service_data = {‘object_id’: target_group, ‘name’: ‘Ghost Items’, ‘view’: show_as_view, ‘icon’: ‘mdi:cube-unfolded’, ‘control’: ‘hidden’, ‘entities’: entity_ids, ‘visible’: True} hass.services.call(‘group’, ‘set’, service_data, False) scan_for_ghosts(hass, logger, data, process_group_entities)

result!:

and for custom-ui:

[NigelL (Nigel)]

Hi @Abeksis I copied the code onto a another system and verified that it works.

Can you give me some more info?

[NigelL (Nigel)]

Hi Marius, if you try this again for the non-custom-ui version, can you check if the weblink.deaditem_xx_ items have a friendly_name assigned? It usually seems to work properly, but sometimes I get the same result you did.

[Mariusthvdb (Marius)]

@NigelL

can you check if the weblink.deaditem_xx_ items have a friendly_name assigned?

[NigelL (Nigel)]

Hi Marius, that looks the same as what see on my system and I’m seeing the element IDs.

On line 57

     hass.states.set(name, "javascript:return false", {"friendly_name":e})

Could you try changing the friendly_name to name (still leave it in the quotes) and see if that helps.

[Mariusthvdb (Marius)]

starting to work fine here. Changed it a bit, not to show default as View, and let me show and hide the script results on demand by some simple scripts calling the python, and hitting the groups out of sight again.

both the sensor (custom-ui) and the group show now (took the if then else out of the python, since i want both options together. all packed into the group.developer, that shows when in dev mode:

group:
  developer:
    name: Developer
    icon: mdi:remote-desktop
#    view: true
    entities:
      - group.mode_tiles
      - group.catchall # if made on the fly by python_script
      - group.ghosts # if made on the fly by python_script
      - group.catch_sensor
      - group.ghosts_sensor
      - group.developer_tools
      - group.developer_links
      - group.developer_scripts

  developer_tools:
    name: Developer tools
    icon: mdi:toolbox
    control: hidden
    entities:
      - switch.mode_developer
      - script.catchall
      - script.remove_catchall
      - script.find_ghosts
      - script.remove_ghosts
      - input_select.log_level # package hassio_pi3_system

sensor: (very nice!)

group.ghosts:

must admit the sensor is immediate, and the group takes quite some time to build.

clicking the weblink is a bit awkward, since not knowing what will happen… would be cool btw if a popup would show the link to the ghost, where we can find it and take care of it.

Secondly, what would further enhance is this, if an ignore folder could be set. I use a /put_aside with all my older and test items, which are now scanned. Which is kind of unexpected, since the configuration file doesn’t know of this folder, and Hassio frontend doesnt show them… this script does however

also unexpected:

the earlier populate_catchall script finds all weblinks too, seems undesired, and should maybe be disabled by default in the python? can we add ignore weblink.ghosts* ?

[NigelL (Nigel)]

@Mariusthvdb

clicking the weblink is a bit awkward, since not knowing what will happen… would be cool btw if a popup would show the link to the ghost, where we can find it and take care of it.

As far as I know so far, there is no way I can find out what file an item is defined in.

@Mariusthvdb

Secondly, what would further enhance is this, if an ignore folder could be set. I use a /put_aside with all my older and test items, which are now scanned. Which is kind of unexpected, since the configuration file doesn’t know of this folder, and Hassio frontend doesnt show them… this script does however

That is really strange. This script doesn’t access the file system at all, it asks HASS for a list of all states and works with that. So even though you have the files moved into a /put_aside folder, HASS must know about them somehow.

There might be some way for me to process the filesystem, but that gets much more complicated.

[Mariusthvdb (Marius)]

@NigelL

That is really strange. This script doesn’t access the file system at all, it asks HASS for a list of all states and works with that. So even though you have the files moved into a /put_aside folder, HASS must know about them somehow.

Found it: I had taken out the main group and group items in tab/view setting, and had put_aside the originating sensor itself, but left the buienrdar items in a sub-group of a view… true orphans, prove the script works as it should. cool, sorry for the mixup…

This is also interesting though, and maybe a bug…

• these orphan entries are still in the entity_registry. the could be seen as a feature for future use, but still, wouldn’t it be more sensible if entities were taken out of the system, the entity_registry would be kept up to date and be sanitized ?

Otherwise we would need a script sanitize_entity_registry also btw, would that be a cool feature and addition of the orphan script, offer an extra click to delete all the entries from entity_registry…

finetune thought:

• some entities are created on the fly. several media_players, updater.updater, probably more. I have put these in a group, so when they popup, they popup in the right places.

which means that when they are not there, (media_players are only created when online, as are the remotes for these tool, in case of appleTV eg) the script finds and reports these. As in the other script, it would be very convenient to allow for a entity_non-scan, an ignore setting for entities also.

last findings:

• it sees the custom-ui sensors as ghosts, while they are most definitely defined, and grouped/used:

  ghosts_sensor:
    name: Ghosts sensor
    icon: mdi:toolbox
    entities:
      - sensor.ghosts_sensor
  

• when setting track: false to device_trackers, the script reports the devices as ghosts even though they are grouped and the group is set to display. Found that with the BT carkits of the delivery trucks, which i dont want to track, but BT recognized and shows them as newly tracked devices… have set to track: true, and voila, they’re gone as Ghosts. Busted

[jnvd3b (Jake)]

The message went away I had the script set up wrong

[Mariusthvdb (Marius)]

@NigelL

Could you try changing the friendly_name to name (still leave it in the quotes) and see if that helps.

well, I only just read this, im sorry.

Don’t think it is necessary any longer, both the sensor (custom-ui) and the group work fine, albeit with some of my findings posted above.

Really very very nice this, thank you so much!

Only thing left in the toolbox would be a file path- tracker of sorts. it would be really very cool if a path to the entity definition would be part of the Hassio interface. A sensor can be made in a variety of places, packages, config file, dedicated sensor folders, temp[lates, you name it. Editing the Yaml files can be rather tedious and often start finding where to look. Take for example the resulting list of your Ghost script. Would be really very comfy when a path would be automatically created, and displayed to the origin of the entity in question. Even when trying to be as meticulous as the undersigned…

[anon43302295]

I now have both your scripts working well, populating a ‘debug view’ with the two groups on the flick of a switch from my maintenance package

Excellent work, thanks for sharing them and for putting the effort in to accommodate people’s requests

[edif30]

Did you have to create the groups with your updated version or will just making the service call do it all?

[NigelL (Nigel)]

It should be creating the groups automatically.

[NigelL (Nigel)]

I think the entries in entity_registry allow home-assistant to easily connect the physical item with the HA item. And removing those items automatically would conflict with the idea of keeping the media_player items in groups (as an example).

I will certainly look at setting up an excluded_entities parameter, probably structure it so that it looks the same as entities in a group.

I think the find_ghosts script might classify it’s own groups as ghosts the first run, since they would be defined in group, but not actually exist in the system.

If you can find me information about the entity_registry and how it can be accessed, I’ll take a look at the sanitize_entity_registry idea, but no promises.

As for the idea of letting you know where an entity is defined, that’s certainly an interesting idea, but Home-Assistant doesn’t make the information available that I know of. And I doubt that I would have access to the filesystem in a python-script. I might have to start converting all of this to a component.

[Mariusthvdb (Marius)]

so cool you put all the effort in, will certainly try and assist…

for starters, I was just pointed by @NotoriousBDG to a great resource, maybe you already know of, of really cool templates for doing the things we are at in this thread, have a look here:

Using templates for input_select.set_options (populating options within an input selector)

indeed, thank you! great resource. seems not really easier, but has the advantage of the more flexible template. Will certainly give that further study. on my list: 3, 4, 7, 10 (needs some extra work), 11, 12… wow, this is really great stuff for the dev-toolbox

cheers!

[edif30]

Got mine all up and running. Thanks @NigelL and @anon43302295! Didn’t know I was missing so many things and this helped me find some things that were labeled wrong. Especially some automations!

[finity]

A couple of things I noticed…

@NigelL, you said you changed the name of the group from ‘deaditems’ to ‘ghosts’ in the script. I’m not sure if you forgot to actually change it or I’m looking in the wrong place. I’m using your last script from post #20. I changed it in my script and now it’s working as expected…I think…I’ve got some items that show up as ghosts but I think some actually exist.

Second, I’m not sure how to use the custom_UI setting. What is supposed to happen if i set it to ‘true’? I’m using the custom_UI_state_card from Andrey. If I set it to true with no other changes it turns the list into an empty badge.

And now for another question…

If I wanted to have your two scripts (catchall & ghosts) output to individual groups in a ‘developers’ tab, how would I need to change your scripts? Right now I’m using the example from @arsaboo in the other script thread and when I click the button to run the scripts they both show up as their own views.

I’ll throw my thanks for this work in here now, as well.

[arsaboo]

Take a look at my config…I have added them both to the developer switch. Turning it on adds the developer tab with both the groups and turning it off removes the tabs and the groups.