Take a look at my config…I have added them both to the developer switch. Turning it on adds the developer tab with both the groups and turning it off removes the tabs and the groups.
Right. That’s what I did too. And now it adds both as their own tabs:
What I would like to do is when I turn on the switch I’d like to add them both as groups to one tab (called for example ‘Development’) and control that tab visibility with the switch.
I’m pretty sure I can figure out how to control the new tab visibility as it will work similarly to the way it works now but I’m not sure what changes I need to make to the scripts so that they don’t show as a view but as standard groups in the view.
That’s how I’ve done it.
In the python_script you have to take out the code that creates it as a view.
Link to my repo is up there 
somewhere, with edited versions of the python_scripts and a button that switches on and off a view called debug, containing the ungrouped items on one card and the ghost items on another.
Hope this helps.
1 Like
Hi gang,
Here’s the new version that allows you to ignore items. You can specify either entity_ids or domains. I have given this some testing, but there might still be issues.
@finity one change in this touches on something you said. The sensor is no longer blank, the state in the circle is the number of ghosts, and it has the text “ghosts” below it.
Here’s how I call it using a variation of @anon43302295’s debug button
turn_on:
- service: python_script.populate_catchall_group
- service: python_script.scan_for_ghosts
data:
ignore_items:
- sensor.doesnt_exist
ignore_domains:
- media_player
- service: group.set
data:
object_id: debug
view: true
visible: true
The ignore_items and ignore_domains options can each take multiple values.
And here’s the code
def process_group_entities(group, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains):
# logger.warn("processing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))
processed_groups.append(group.entity_id)
for e in group.attributes["entity_id"]:
domain = e.split(".")[0]
if domain == "group":
g = hass.states.get(e)
if (g is not None) and (g.entity_id not in processed_groups):
process_group_entities(g, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains)
else:
if (domain not in ignore_domains):
grouped_entities.add(e)
# logger.warn("finishing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))
def scan_for_ghosts(hass, logger, data, process_group_entities):
target_group=data.get("target_group","ghosts")
show_as_view = data.get("show_as_view", False)
use_custom_ui = data.get("use_custom_ui", True)
ignore_items = data.get("ignore_items",[])
ignore_domains = data.get("ignore_domains",[])
real_entities = set(ignore_items)
grouped_entities = set()
processed_groups=[]
for s in hass.states.all():
domain = s.entity_id.split(".")[0]
if domain != "group":
real_entities.add(s.entity_id)
else:
if (("view" not in s.attributes) or
( s.attributes["view"] == False)):
real_entities.add(s.entity_id)
process_group_entities(s, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains)
logger.error("{} real entities".format(len(real_entities)))
logger.error("{} grouped entities".format(len(grouped_entities)))
logger.error("{} groups processed".format(len(processed_groups)))
results = grouped_entities - real_entities
logger.error("{} entities to list".format(len(results)))
entity_ids=[]
if use_custom_ui:
summary=""
for e in results:
summary = "{}{}\n".format(summary, e)
hass.states.set('sensor.ghost_items', len(results), {
'custom_ui_state_card': 'state-card-value_only',
'text': summary, 'unit_of_measurement': 'ghosts'
})
entity_ids.append("sensor.ghost_items")
else:
counter=0
for e in results:
name = "weblink.ghost{}".format(counter)
hass.states.set(name, "javascript:return false", {"friendly_name":e})
entity_ids.append(name)
counter = counter +1
service_data = {'object_id': target_group, 'name': 'Ghost Items',
'view': show_as_view, 'icon': 'mdi:ghost',
'control': 'hidden', 'entities': entity_ids,
'visible': True}
hass.services.call('group', 'set', service_data, False)
scan_for_ghosts(hass, logger, data, process_group_entities)
Worked like a charm! Thanks!
1 Like
magic! this is getting better and better each time you have a look! a big thanks! went from:
to
because of the ignore settings, just as I would have hoped. so cool.
kind a like this report, is there a way we should see this in the front end too? Don’t think thats happening here, at least I cant find it… Maybe its a ghost or ungrouped item, ive just hidden through the ignore setting 
(enter both sensors there, not sure if thats correct?)
Thanks for the update.
I think I’ve got that all worked out except I’m still not sure what setting custom_UI to true accomplishes. Every time I turn the custom_UI to true the card that is supposed to contain the ghost items is empty. If I turn it off it contains items.
But there is a difference now with the new code. it’s no longer an empty badge. Now its just an empty card.
Hi @Finity, to get that part working properly you need to have a “state-card-value_only.html” card defined (alongside the normal state-card-custom-ui.html).
The only working definition I found was here
Important: Don’t forget to add the card to the frontend, see this message for details
If you use the value_only card I pointed you to above, you can play with adding a symbol at the front of the entity name based on domain to get different coloring.
Here’s my card at the moment (playing with coloring it)
I modified the script slightly to use show_as_view = data.get("show_as_view", False) so that the item is not added as a view.
OK, thanks.
I’m not even sure if it’s something I would use, but now I know why it isn’t working. 
Thanks but I think I’ve got my views/groups all worked out now.
2 small issues here:
-
the ghosts sensor shows fine listing the entities :
but the ungrouped sensor shows them as list items, and I cant find the spot to change in the python to make it show as the ghosts sensor:
- second: suddenly (i think, didnt notice before) the Ghosts group shows with both the weblink and the entity, while before it only showed the weblinks?
must have edited the python incorrectly, but I cant spot it… unless by adding argument e to the summary line, you intended to do just so?
If we take out the e in this line:
busted = "{}{}\n".format(busted,e) to make it into busted = "{}\n".format(busted), the group shows as before, but the sensor is empty… o well.
if its of any help, heres the custom card I adapted some further to use with more coloring and take out the Bold error. gives some extra coloring options I needed elsewhere:
<!--
https://github.com/home-assistant/home-assistant-polymer/blob/master/src/state-summary/state-card-display.html
https://github.com/home-assistant/home-assistant-polymer/blob/master/src/components/entity/state-badge.html
https://github.com/PolymerElements/paper-styles/blob/master/color.html
paper-brown-500: #795548 and google-grey-500: #9e9e9e
-->
<dom-module id="state-card-value_only">
<template>
<style is="custom-style" include="iron-flex iron-flex-alignment"></style>
<style>
.bold {
@apply(--paper-font-body1);
color: var(--primary-text-color);
font-weight: bold;
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.italic {
@apply(--paper-font-body1);
color: var(--primary-text-color);
font-style: italic;
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.red {
@apply(--paper-font-body1);
color: var(--google-red-500);
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.green {
@apply(--paper-font-body1);
color: var(--google-green-500);
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.yellow {
@apply(--paper-font-body1);
color: var(--google-yellow-500);
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.grey {
@apply(--paper-font-body1);
color: #9e9e9e;
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.brown {
@apply(--paper-font-body1);
color: #795548;
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.blue {
@apply(--paper-font-body1);
color: var(--google-blue-500);
margin-left: 8px;
text-align: left;
line-height: 20px;
}
.normal {
@apply(--paper-font-body1);
color: #009688;
margin-left: 8px;
text-align: left;
line-height: 20px;
}
</style>
<template is="dom-repeat" items="[[computeStateDisplay(stateObj)]]">
<div class$="[[computeClass(item)]]">[[computeItem(item)]]</div>
</template>
</template>
</dom-module>
<script>
Polymer({
is: 'state-card-value_only',
properties: {
hass: {
type: Object,
},
stateObj: {
type: Object,
},
},
computeStateDisplay: function (stateObj) {
var text = stateObj.attributes.text;
if (text == null) { text = stateObj.state };
return text.split("\n");
},
computeItem: function (item) {
var value = item.trim();
switch(value.substring(0,1)) {
case "*":
case "/":
case "!":
case "+":
case "=":
case "%":
case "$":
case "#":
return value.substring(1);
default:
return value;
}
},
computeClass: function (item) {
switch(item.trim().substring(0,1)) {
case "*": return "bold";
case "/": return "italic";
case "!": return "red";
case "+": return "green";
case "=": return "yellow";
case "%": return "grey";
case "$": return "brown";
case "#": return "blue";
default: return "normal";
}
},
});
</script>Do you have your version of the scripts up somewhere I can look at? Or could you message them to me?
consider it done, message sent.thx!
had a few t.trim errors because of trying to have Unit_of_measurement show the len(results) or len(entity_ids), but changed that to the Friendly_name, and bow have these 2 little neat badges show up when the scripts run and find Orphans or Ghosts…
Orphans:
hass.states.set('sensor.orphans_badge', len(entity_ids), {
'text': caught,
'unit_of_measurement': 'Orphans',
'friendly_name': len(entity_ids),
'entity_picture': '/local/badges/orphans.png'
})
Ghosts:
hass.states.set('sensor.ghosts_badge', len(results), {
'text': busted,
'unit_of_measurement': 'Ghosts',
'friendly_name': len(results),
'entity_picture': '/local/badges/ghosts.png'
})
just add these to the python script (of course change the appropriate name if needed) and add the badges to the developer group .
catch the weekend!
cheers.
Since you’ve been experimenting with the weblink.ghosts{}, I had a little try too. Found that when using a fake domain, we can show them without any trouble of truly linking and the javascript safeguard isnt necessary?
tried it like this now:
else:
counter=0
for e in results:
name = "left_entity.ghosts{}".format(counter) #weblink.ghosts{}
parent = "Left behind in {}".format(len(processed_groups))
hass.states.set(name, parent, {"friendly_name":e}) #"javascript:return false"
entity_ids.append(name)
counter = counter +1
as you can see, ive made space for a parent to expose. No idea yet how to calculate the parent, so format them by narrowing them down to the processed_groups for now.
showing in frontend:
and more-info
rather basic, but it gets the job done.
found the way to format the custom-ui card as desired:
fyi here’s the python that does it:
busted=""
for e in results:
busted = "{}!- {}\n".format(busted,e) #"{}\n".format(e) # "{}{}\n".format(busted,e,)
ghost_card = "*=========== Ghosts ===========\n{}\n*========= Entity count =========\n!- {} entities to list\n#- {} real entities\n+- {} grouped entities\n+- {} groups processed\n*========== Settings ===========\n/- targetting group: {}\n$- ignoring {} domains: {}\n$- ignoring {} items: {}".format(busted,len(results),len(real_entities),len(grouped_entities),len(processed_groups),target_group,len(ignore_domains), ignore_domains, len(ignore_items), ignore_items ) #"{}\n".format(e) #
hass.states.set('sensor.ghosts_sensor', len(results), {
'custom_ui_state_card': 'state-card-value_only',
'text': ghost_card,
'unit_of_measurement': 'ghosts'
})
Now how to find the parenting groups…
had a few side-sessions with @NigelL, and this is the result for the ‘Ghosts’ script. Besides further tweaking and formatting, please note the great added functionality of disclosure of the group the ghosts are found in. Making this an indispensable tool for debugging your configuration.
##########################################################################################
# Script to find all grouped but non-existing entities
#
# https://community.home-assistant.io/t/script-to-find-items-that-no-longer-exist/56146/52
# original author: https://community.home-assistant.io/u/NigelL
# customizing, further tweaking and pushing it a bit by https://community.home-assistant.io/u/Mariusthvdb/
##########################################################################################
# set ignore_items and ignore_domains in the yaml file calling the script
# script:
# scan_ghosts:
# alias: Scan Ghosts
# sequence:
# - service: python_script.scan_ghosts
# data:
# ignore_items:
# - updater.updater
# - sensor.ghosts_sensor
# - sensor.ghosts_badge
# - sensor.orphans_sensor
# - sensor.orphans_badge
# - python_script.scan_for_orphans
# ignore_domains:
# - media_player
# - remote
#
##########################################################################################
# Codes for text_colors declared in
# Custom card: /custom_ui/state-card-value_only.html
##########################################################################################
# case "*": return "bold";
# case "/": return "italic";
# case "!": return "red";
# case "+": return "green";
# case "=": return "yellow";
# case "%": return "grey";
# case "$": return "brown";
# case "#": return "blue";
# default: return "normal";
##########################################################################################
def process_group_entities(group, grouped_entities, hass, logger, process_group_entities,\
processed_groups, ignore_domains, ignore_items):
# logger.info("processing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))
processed_groups.append(group.entity_id)
for e in group.attributes["entity_id"]:
domain = e.split(".")[0]
if domain == "group":
g = hass.states.get(e)
if (g is not None) and (g.entity_id not in processed_groups):
process_group_entities(g, grouped_entities, hass, logger, process_group_entities,\
processed_groups, ignore_domains, ignore_items)
else:
if (domain not in ignore_domains):
if not e in grouped_entities:
grouped_entities[e] = set()
grouped_entities[e].add(group.entity_id)
# logger.info("finishing group {}, currently {} grouped items".format(group.entity_id, \ len(grouped_entities)))
def scan_ghosts(hass, logger, data, process_group_entities):
target_group=data.get("target_group","group.ghosts")
show_as_view = data.get("show_as_view", False)
use_custom_ui = data.get("use_custom_ui", True)
ignore_items = data.get("ignore_items",[])
ignore_domains = data.get("ignore_domains",[])
real_entities = set(ignore_items)
grouped_entities = {}
processed_groups=[]
for s in hass.states.all():
domain = s.entity_id.split(".")[0]
if domain != "group":
real_entities.add(s.entity_id)
else:
if (("view" not in s.attributes) or
( s.attributes["view"] == False)):
real_entities.add(s.entity_id)
process_group_entities(s,grouped_entities,hass,logger, \
process_group_entities,processed_groups,ignore_domains, \
ignore_items)
logger.info("{} real entities".format(len(real_entities)))
logger.info("{} grouped entities".format(len(grouped_entities)))
logger.info("{} groups processed".format(len(processed_groups)))
results = grouped_entities.keys() - real_entities
logger.info("{} entities to list".format(len(results)))
entity_ids=[]
# if (use_custom_ui): ## show card
busted=""
busted_badge=""
for e in results:
line = "{} in: {}".format(e, ",".join(grouped_entities[e]))
busted = "{}!- {}\n".format(busted,line)
busted_badge = "{}{}\n".format(busted_badge,line)
ignore_items_unlist = ', '.join(ignore_items)
ignore_domains_unlist = ', '.join(ignore_domains)
ghost_card = '*=========== Ghosts ===========\n' \
'{}\n' \
'*========= Entity count =========\n' \
'!- {} entities to list\n' \
'#- {} real entities\n' \
'+- {} grouped entities\n' \
'+- {} groups processed\n' \
'*========== Settings ===========\n' \
'/- targetting group: {}\n' \
'$- ignoring {} items:\n' \
'%|-> {}\n' \
'$- ignoring {} domains:\n' \
'%|-> {}' \
.format(busted,
len(results),
len(real_entities),
len(grouped_entities),
len(processed_groups),
target_group,
len(ignore_items),
ignore_items_unlist,
len(ignore_domains),
ignore_domains_unlist)
hass.states.set('sensor.ghosts_sensor', len(results), {
'custom_ui_state_card': 'state-card-value_only',
'text': ghost_card,
# 'unit_of_measurement': 'ghosts'
})
# else: ##show group with found ghosts:
counter=0
for e in results:
name = "left_entity.ghost{}".format(counter)
parent = "In: {}".format(",".join(grouped_entities[e]))
hass.states.set(name, parent, {"friendly_name":e, "icon": "mdi:ghost"})
entity_ids.append(name)
counter = counter +1
service_data = {'object_id': 'ghosts','name': 'Ghosts',
'view': show_as_view,'icon': 'mdi:ghost',
'control': 'hidden','entities': entity_ids,
'visible': True }
hass.services.call('group', 'set', service_data, False)
# show badge
hass.states.set('sensor.ghosts_badge', len(results), {
'text': busted_badge,
'unit_of_measurement': 'Ghosts',
'friendly_name': len(results),
'entity_picture': '/local/badges/ghosts.png'
})
scan_ghosts(hass, logger, data, process_group_entities)
Frontend:
3 Likes
updated script, now better handling when 0 Ghosts are found:
##########################################################################################
# Script to find all grouped but non-existing entities
#
# https://community.home-assistant.io/t/script-to-find-items-that-no-longer-exist/56146/52
# original author: https://community.home-assistant.io/u/NigelL
# customizing, further tweaking and pushing it a bit by https://community.home-assistant.io/u/Mariusthvdb/
##########################################################################################
# set ignore_items and ignore_domains in the yaml file calling the script
# script:
# scan_ghosts:
# alias: Scan Ghosts
# sequence:
# - service: python_script.scan_ghosts
# data:
# ignore_items:
# - updater.updater
# - sensor.ghosts_sensor
# - sensor.ghosts_badge
# - sensor.orphans_sensor
# - sensor.orphans_badge
# - python_script.scan_for_orphans
# ignore_domains:
# - media_player
# - remote
#
##########################################################################################
# Codes for text_colors declared in
# Custom card: /custom_ui/state-card-value_only.html
##########################################################################################
# case "*": return "bold";
# case "/": return "italic";
# case "!": return "red";
# case "+": return "green";
# case "=": return "yellow";
# case "%": return "grey";
# case "$": return "brown";
# case "#": return "blue";
# default: return "normal";
##########################################################################################
def process_group_entities(group, grouped_entities, hass, logger, process_group_entities,\
processed_groups, ignore_domains, ignore_items):
# logger.info("processing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))
processed_groups.append(group.entity_id)
for e in group.attributes['entity_id']:
domain = e.split('.')[0]
if domain == 'group':
g = hass.states.get(e)
if (g is not None) and (g.entity_id not in processed_groups):
process_group_entities(g, grouped_entities, hass, logger, process_group_entities,\
processed_groups, ignore_domains, ignore_items)
else:
if (domain not in ignore_domains):
if not e in grouped_entities:
grouped_entities[e] = set()
grouped_entities[e].add(group.entity_id)
# logger.info("finishing group {}, currently {} grouped items".format(group.entity_id, \ len(grouped_entities)))
def scan_ghosts(hass, logger, data, process_group_entities):
target_group=data.get('target_group','group.ghosts')
show_as_view = data.get('show_as_view', False)
use_custom_ui = data.get('use_custom_ui', True)
ignore_items = data.get('ignore_items',[])
ignore_domains = data.get('ignore_domains',[])
show_if_empty = data.get('show_if_empty', False)
min_items_to_show = data.get('min_items_to_show', 0)
real_entities = set(ignore_items)
grouped_entities = {}
processed_groups=[]
for s in hass.states.all():
domain = s.entity_id.split('.')[0]
if domain != 'group':
real_entities.add(s.entity_id)
else:
if (('view' not in s.attributes) or
( s.attributes['view'] == False)):
real_entities.add(s.entity_id)
process_group_entities(s,grouped_entities,hass,logger, \
process_group_entities,processed_groups,ignore_domains, \
ignore_items)
logger.info('{} real entities'.format(len(real_entities)))
logger.info('{} grouped entities'.format(len(grouped_entities)))
logger.info('{} groups processed'.format(len(processed_groups)))
results = grouped_entities.keys() - real_entities
logger.info('{} entities to list'.format(len(results)))
entity_ids=[]
## if (use_custom_ui): ## show card
busted=""
busted_badge=""
if (len(results)) > min_items_to_show or show_if_empty:
visible = True
for e in results:
line = '{} in: {}'.format(e, ','.join(grouped_entities[e]))
busted = '{}!- {}\n'.format(busted,line)
busted_badge = '{}{}\n'.format(busted_badge,line)
friendly_name_badge = len(results)
else:
visible = False
busted = '!- All busted, {} Ghosts found!\n'.format(len(results))
busted_badge = 'No Ghosts found\n'
friendly_name_badge = 'All busted'
ignore_items_unlist = ', '.join(ignore_items)
ignore_domains_unlist = ', '.join(ignore_domains)
ghost_card = '*=========== Ghosts ===========\n' \
'{}\n' \
'*========= Entity count =========\n' \
'!- {} entities to list\n' \
'#- {} real entities\n' \
'+- {} grouped entities\n' \
'+- {} groups processed\n' \
'*========== Settings ===========\n' \
'/- targetting group: {}\n' \
'$- ignoring {} items:\n' \
'%|-> {}\n' \
'$- ignoring {} domains:\n' \
'%|-> {}' \
.format(busted,
len(results),
len(real_entities),
len(grouped_entities),
len(processed_groups),
target_group,
len(ignore_items),
ignore_items_unlist,
len(ignore_domains),
ignore_domains_unlist)
hass.states.set('sensor.ghosts_sensor', len(results), {
'custom_ui_state_card': 'state-card-value_only',
'text': ghost_card,
# 'unit_of_measurement': 'ghosts'
})
# else: ##show group with found ghosts:
counter=0
for e in results:
name = 'left_entity.ghost{}'.format(counter)
parent = 'In: {}'.format(','.join(grouped_entities[e]))
hass.states.set(name, parent, {'friendly_name':e, 'icon': 'mdi:ghost'})
entity_ids.append(name)
counter = counter +1
service_data = {'object_id': 'ghosts','name': 'Ghosts',
'view': show_as_view,'icon': 'mdi:ghost',
'control': 'hidden','entities': entity_ids,
'visible': visible }
hass.services.call('group', 'set', service_data, False)
# show badge
hass.states.set('sensor.ghosts_badge', len(results), {
'text': busted_badge,
'unit_of_measurement': 'Ghosts',
'friendly_name': friendly_name_badge,
'entity_picture': '/local/badges/ghosts.png'
})
scan_ghosts(hass, logger, data, process_group_entities)
1 Like
I am a total newbie here, and part of my ramping up to HomeAssistant ended up creating too many orphans.
Is there a step by step to add the script to my setup so I can debug using the front end?
Reading the thread is exactly what I need to identify items which are not valid.
Not all my entities are in a group.
Can you help telling me how to go about implementing this functionality? I use VS code within HA to get my yaml code done, helps me with auto complete and real time debug.
Maybe a dumb question but how do I delete the old ghost entities? lol
EDIT: That was annoying. deleted the python_script entry in conf.yaml and removed the states from teh database. Im sure there is a better way face_palm. But all gone.