Script to find items that no longer exist

well, I only just read this, im sorry.

Don’t think it is necessary any longer, both the sensor (custom-ui) and the group work fine, albeit with some of my findings posted above.

Really very very nice this, thank you so much!

Only thing left in the toolbox would be a file path- tracker of sorts. it would be really very cool if a path to the entity definition would be part of the Hassio interface. A sensor can be made in a variety of places, packages, config file, dedicated sensor folders, temp[lates, you name it. Editing the Yaml files can be rather tedious and often start finding where to look. Take for example the resulting list of your Ghost script. Would be really very comfy when a path would be automatically created, and displayed to the origin of the entity in question. Even when trying to be as meticulous as the undersigned… :wink:

I now have both your scripts working well, populating a ‘debug view’ with the two groups on the flick of a switch from my maintenance package :smile:

Excellent work, thanks for sharing them and for putting the effort in to accommodate people’s requests :+1:

1 Like

Did you have to create the groups with your updated version or will just making the service call do it all?

It should be creating the groups automatically.

I think the entries in entity_registry allow home-assistant to easily connect the physical item with the HA item. And removing those items automatically would conflict with the idea of keeping the media_player items in groups (as an example).

I will certainly look at setting up an excluded_entities parameter, probably structure it so that it looks the same as entities in a group.

I think the find_ghosts script might classify it’s own groups as ghosts the first run, since they would be defined in group, but not actually exist in the system.

If you can find me information about the entity_registry and how it can be accessed, I’ll take a look at the sanitize_entity_registry idea, but no promises.

As for the idea of letting you know where an entity is defined, that’s certainly an interesting idea, but Home-Assistant doesn’t make the information available that I know of. And I doubt that I would have access to the filesystem in a python-script. I might have to start converting all of this to a component.

so cool you put all the effort in, will certainly try and assist…

for starters, I was just pointed by @NotoriousBDG to a great resource, maybe you already know of, of really cool templates for doing the things we are at in this thread, have a look here:

cheers!

1 Like

Got mine all up and running. Thanks @NigelL and @anon43302295! Didn’t know I was missing so many things and this helped me find some things that were labeled wrong. Especially some automations!

1 Like

A couple of things I noticed…

@NigelL, you said you changed the name of the group from ‘deaditems’ to ‘ghosts’ in the script. I’m not sure if you forgot to actually change it or I’m looking in the wrong place. I’m using your last script from post #20. I changed it in my script and now it’s working as expected…I think…I’ve got some items that show up as ghosts but I think some actually exist.

Second, I’m not sure how to use the custom_UI setting. What is supposed to happen if i set it to ‘true’? I’m using the custom_UI_state_card from Andrey. If I set it to true with no other changes it turns the list into an empty badge.

And now for another question…

If I wanted to have your two scripts (catchall & ghosts) output to individual groups in a ‘developers’ tab, how would I need to change your scripts? Right now I’m using the example from @arsaboo in the other script thread and when I click the button to run the scripts they both show up as their own views.

I’ll throw my thanks for this work in here now, as well.

Take a look at my config…I have added them both to the developer switch. Turning it on adds the developer tab with both the groups and turning it off removes the tabs and the groups.

Right. That’s what I did too. And now it adds both as their own tabs:

ex

What I would like to do is when I turn on the switch I’d like to add them both as groups to one tab (called for example ‘Development’) and control that tab visibility with the switch.

I’m pretty sure I can figure out how to control the new tab visibility as it will work similarly to the way it works now but I’m not sure what changes I need to make to the scripts so that they don’t show as a view but as standard groups in the view.

That’s how I’ve done it.

In the python_script you have to take out the code that creates it as a view.

Link to my repo is up there :arrow_up: somewhere, with edited versions of the python_scripts and a button that switches on and off a view called debug, containing the ungrouped items on one card and the ghost items on another.

Hope this helps.

1 Like

Hi gang,

Here’s the new version that allows you to ignore items. You can specify either entity_ids or domains. I have given this some testing, but there might still be issues.

@finity one change in this touches on something you said. The sensor is no longer blank, the state in the circle is the number of ghosts, and it has the text “ghosts” below it.

Here’s how I call it using a variation of @anon43302295’s debug button

      turn_on:
        - service: python_script.populate_catchall_group
        - service: python_script.scan_for_ghosts
          data:
            ignore_items:
              - sensor.doesnt_exist
            ignore_domains:
              - media_player
        - service: group.set
          data:
            object_id: debug
            view: true
            visible: true

The ignore_items and ignore_domains options can each take multiple values.

And here’s the code

def process_group_entities(group, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains):
#  logger.warn("processing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))

  processed_groups.append(group.entity_id)
  for e in group.attributes["entity_id"]:
    domain = e.split(".")[0]
    if domain == "group":
      g = hass.states.get(e)
      if (g is not None) and (g.entity_id not in processed_groups):      
        process_group_entities(g, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains)
    else:
      if (domain not in ignore_domains):
        grouped_entities.add(e)
      
#  logger.warn("finishing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))
  
def scan_for_ghosts(hass, logger, data, process_group_entities):
  target_group=data.get("target_group","ghosts")
  show_as_view = data.get("show_as_view", False)
  use_custom_ui = data.get("use_custom_ui", True)
  ignore_items = data.get("ignore_items",[])
  ignore_domains = data.get("ignore_domains",[])
  

  real_entities = set(ignore_items)
  grouped_entities = set()
  processed_groups=[]
  
  for s in hass.states.all():
    domain = s.entity_id.split(".")[0]
    if domain != "group":
      real_entities.add(s.entity_id)
    else:
      if (("view" not in s.attributes) or
          ( s.attributes["view"] == False)):
        real_entities.add(s.entity_id)
      process_group_entities(s, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains)
      
  logger.error("{} real entities".format(len(real_entities)))
  logger.error("{} grouped entities".format(len(grouped_entities)))
  logger.error("{} groups processed".format(len(processed_groups)))
  results = grouped_entities - real_entities
  logger.error("{} entities to list".format(len(results)))
  entity_ids=[]

  if use_custom_ui:
    summary=""
    for e in results:
      summary = "{}{}\n".format(summary, e)
    
    hass.states.set('sensor.ghost_items', len(results), {
        'custom_ui_state_card': 'state-card-value_only',
        'text': summary, 'unit_of_measurement': 'ghosts'
    })
    entity_ids.append("sensor.ghost_items")
    
  else:
    counter=0
    for e in results:
      name = "weblink.ghost{}".format(counter)
      hass.states.set(name, "javascript:return false", {"friendly_name":e})
      entity_ids.append(name)
      counter = counter +1

  service_data = {'object_id': target_group, 'name': 'Ghost Items',
                    'view': show_as_view, 'icon': 'mdi:ghost',
                    'control': 'hidden', 'entities': entity_ids,
                    'visible': True}

  hass.services.call('group', 'set', service_data, False)

scan_for_ghosts(hass, logger, data, process_group_entities)

Worked like a charm! Thanks!

1 Like

magic! this is getting better and better each time you have a look! a big thanks! went from:

53

to

44

because of the ignore settings, just as I would have hoped. so cool.

kind a like this report, is there a way we should see this in the front end too? Don’t think thats happening here, at least I cant find it… Maybe its a ghost or ungrouped item, ive just hidden through the ignore setting :wink: (enter both sensors there, not sure if thats correct?)

Thanks for the update.

I think I’ve got that all worked out except I’m still not sure what setting custom_UI to true accomplishes. Every time I turn the custom_UI to true the card that is supposed to contain the ghost items is empty. If I turn it off it contains items.

But there is a difference now with the new code. it’s no longer an empty badge. Now its just an empty card.

Hi @Finity, to get that part working properly you need to have a “state-card-value_only.html” card defined (alongside the normal state-card-custom-ui.html).

The only working definition I found was here

Important: Don’t forget to add the card to the frontend, see this message for details

If you use the value_only card I pointed you to above, you can play with adding a symbol at the front of the entity name based on domain to get different coloring.

Here’s my card at the moment (playing with coloring it)

ghost_items_sample

I modified the script slightly to use show_as_view = data.get("show_as_view", False) so that the item is not added as a view.

OK, thanks.

I’m not even sure if it’s something I would use, but now I know why it isn’t working. :grinning:

Thanks but I think I’ve got my views/groups all worked out now.

2 small issues here:

  • the ghosts sensor shows fine listing the entities :
    17

    but the ungrouped sensor shows them as list items, and I cant find the spot to change in the python to make it show as the ghosts sensor:

24

  • second: suddenly (i think, didnt notice before) the Ghosts group shows with both the weblink and the entity, while before it only showed the weblinks?

36

must have edited the python incorrectly, but I cant spot it… unless by adding argument e to the summary line, you intended to do just so?

If we take out the e in this line:

busted = "{}{}\n".format(busted,e) to make it into busted = "{}\n".format(busted), the group shows as before, but the sensor is empty… o well.

if its of any help, heres the custom card I adapted some further to use with more coloring and take out the Bold error. gives some extra coloring options I needed elsewhere:

<!--
https://github.com/home-assistant/home-assistant-polymer/blob/master/src/state-summary/state-card-display.html
https://github.com/home-assistant/home-assistant-polymer/blob/master/src/components/entity/state-badge.html
https://github.com/PolymerElements/paper-styles/blob/master/color.html
paper-brown-500: #795548 and google-grey-500: #9e9e9e
-->

<dom-module id="state-card-value_only">
  <template>

    <style is="custom-style" include="iron-flex iron-flex-alignment"></style>
    <style>
      .bold {
        @apply(--paper-font-body1);
        color: var(--primary-text-color);
        font-weight: bold;
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .italic {
        @apply(--paper-font-body1);
        color: var(--primary-text-color);
        font-style: italic;
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .red {
        @apply(--paper-font-body1);
        color: var(--google-red-500);
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .green {
        @apply(--paper-font-body1);
        color: var(--google-green-500);
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .yellow {
        @apply(--paper-font-body1);
        color: var(--google-yellow-500);
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .grey {
        @apply(--paper-font-body1);
        color: #9e9e9e;
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .brown {
        @apply(--paper-font-body1);
        color: #795548;
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .blue {
        @apply(--paper-font-body1);
        color: var(--google-blue-500);
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
      .normal {
        @apply(--paper-font-body1);
        color: #009688;
        margin-left: 8px;
        text-align: left;
        line-height: 20px;
      }
    </style>

    <template is="dom-repeat" items="[[computeStateDisplay(stateObj)]]">
      <div class$="[[computeClass(item)]]">[[computeItem(item)]]</div>
    </template>

  </template>
</dom-module>

<script>
Polymer({
  is: 'state-card-value_only',

  properties: {
    hass: {
      type: Object,
    },

    stateObj: {
      type: Object,
    },
  },

  computeStateDisplay: function (stateObj) {
    var text = stateObj.attributes.text;

    if (text == null) { text = stateObj.state };
    return text.split("\n");
  },

  computeItem: function (item) {
      var value = item.trim();

      switch(value.substring(0,1)) {
      case "*":
      case "/":
      case "!":
      case "+":
      case "=":
      case "%":
      case "$":
      case "#":
          return value.substring(1);
      default:
          return value;
      }
  },

  computeClass: function (item) {
      switch(item.trim().substring(0,1)) {
      case "*": return "bold";
      case "/": return "italic";
      case "!": return "red";
      case "+": return "green";
      case "=": return "yellow";
      case "%": return "grey";
      case "$": return "brown";
      case "#": return "blue";
      default:  return "normal";
      }
  },

});
</script>