Hi Marius, if you try this again for the non-custom-ui version, can you check if the weblink.deaditem_xx_ items have a friendly_name assigned? It usually seems to work properly, but sometimes I get the same result you did.
Hi Marius, that looks the same as what see on my system and I’m seeing the element IDs.
On line 57
hass.states.set(name, "javascript:return false", {"friendly_name":e})
Could you try changing the friendly_name to name (still leave it in the quotes) and see if that helps.
starting to work fine here. Changed it a bit, not to show default as View, and let me show and hide the script results on demand by some simple scripts calling the python, and hitting the groups out of sight again.
both the sensor (custom-ui) and the group show now (took the if then else out of the python, since i want both options together. all packed into the group.developer, that shows when in dev mode:
group:
developer:
name: Developer
icon: mdi:remote-desktop
# view: true
entities:
- group.mode_tiles
- group.catchall # if made on the fly by python_script
- group.ghosts # if made on the fly by python_script
- group.catch_sensor
- group.ghosts_sensor
- group.developer_tools
- group.developer_links
- group.developer_scripts
developer_tools:
name: Developer tools
icon: mdi:toolbox
control: hidden
entities:
- switch.mode_developer
- script.catchall
- script.remove_catchall
- script.find_ghosts
- script.remove_ghosts
- input_select.log_level # package hassio_pi3_system
sensor: (very nice!)
group.ghosts:
must admit the sensor is immediate, and the group takes quite some time to build.
clicking the weblink is a bit awkward, since not knowing what will happen… would be cool btw if a popup would show the link to the ghost, where we can find it and take care of it.
Secondly, what would further enhance is this, if an ignore folder could be set. I use a /put_aside with all my older and test items, which are now scanned. Which is kind of unexpected, since the configuration file doesn’t know of this folder, and Hassio frontend doesnt show them… this script does however 
also unexpected:
the earlier populate_catchall script finds all weblinks too, seems undesired, and should maybe be disabled by default in the python? can we add ignore weblink.ghosts* ?
As far as I know so far, there is no way I can find out what file an item is defined in.
That is really strange. This script doesn’t access the file system at all, it asks HASS for a list of all states and works with that. So even though you have the files moved into a /put_aside folder, HASS must know about them somehow.
There might be some way for me to process the filesystem, but that gets much more complicated.
Found it: I had taken out the main group and group items in tab/view setting, and had put_aside the originating sensor itself, but left the buienrdar items in a sub-group of a view… true orphans, prove the script works as it should. cool, sorry for the mixup… 
This is also interesting though, and maybe a bug…
- these orphan entries are still in the entity_registry. the could be seen as a feature for future use, but still, wouldn’t it be more sensible if entities were taken out of the system, the entity_registry would be kept up to date and be sanitized ?
Otherwise we would need a script sanitize_entity_registry also 
btw, would that be a cool feature and addition of the orphan script, offer an extra click to delete all the entries from entity_registry…
finetune thought:
- some entities are created on the fly. several media_players, updater.updater, probably more. I have put these in a group, so when they popup, they popup in the right places.
which means that when they are not there, (media_players are only created when online, as are the remotes for these tool, in case of appleTV eg) the script finds and reports these. As in the other script, it would be very convenient to allow for a entity_non-scan, an ignore setting for entities also.
last findings:
-
it sees the custom-ui sensors as ghosts, while they are most definitely defined, and grouped/used:
ghosts_sensor: name: Ghosts sensor icon: mdi:toolbox entities: - sensor.ghosts_sensor
- when setting
track: falseto device_trackers, the script reports the devices as ghosts even though they are grouped and the group is set to display. Found that with the BT carkits of the delivery trucks, which i dont want to track, but BT recognized and shows them as newly tracked devices… have set totrack: true, and voila, they’re gone as Ghosts. Busted
The message went away I had the script set up wrong
well, I only just read this, im sorry.
Don’t think it is necessary any longer, both the sensor (custom-ui) and the group work fine, albeit with some of my findings posted above.
Really very very nice this, thank you so much!
Only thing left in the toolbox would be a file path- tracker of sorts. it would be really very cool if a path to the entity definition would be part of the Hassio interface. A sensor can be made in a variety of places, packages, config file, dedicated sensor folders, temp[lates, you name it. Editing the Yaml files can be rather tedious and often start finding where to look. Take for example the resulting list of your Ghost script. Would be really very comfy when a path would be automatically created, and displayed to the origin of the entity in question. Even when trying to be as meticulous as the undersigned…
I now have both your scripts working well, populating a ‘debug view’ with the two groups on the flick of a switch from my maintenance package 
Excellent work, thanks for sharing them and for putting the effort in to accommodate people’s requests 
1 Like
Did you have to create the groups with your updated version or will just making the service call do it all?
It should be creating the groups automatically.
I think the entries in entity_registry allow home-assistant to easily connect the physical item with the HA item. And removing those items automatically would conflict with the idea of keeping the media_player items in groups (as an example).
I will certainly look at setting up an excluded_entities parameter, probably structure it so that it looks the same as entities in a group.
I think the find_ghosts script might classify it’s own groups as ghosts the first run, since they would be defined in group, but not actually exist in the system.
If you can find me information about the entity_registry and how it can be accessed, I’ll take a look at the sanitize_entity_registry idea, but no promises.
As for the idea of letting you know where an entity is defined, that’s certainly an interesting idea, but Home-Assistant doesn’t make the information available that I know of. And I doubt that I would have access to the filesystem in a python-script. I might have to start converting all of this to a component.
so cool you put all the effort in, will certainly try and assist…
for starters, I was just pointed by @NotoriousBDG to a great resource, maybe you already know of, of really cool templates for doing the things we are at in this thread, have a look here:
cheers!
1 Like
Got mine all up and running. Thanks @NigelL and @anon43302295! Didn’t know I was missing so many things and this helped me find some things that were labeled wrong. Especially some automations!
1 Like
A couple of things I noticed…
@NigelL, you said you changed the name of the group from ‘deaditems’ to ‘ghosts’ in the script. I’m not sure if you forgot to actually change it or I’m looking in the wrong place. I’m using your last script from post #20. I changed it in my script and now it’s working as expected…I think…I’ve got some items that show up as ghosts but I think some actually exist.
Second, I’m not sure how to use the custom_UI setting. What is supposed to happen if i set it to ‘true’? I’m using the custom_UI_state_card from Andrey. If I set it to true with no other changes it turns the list into an empty badge.
And now for another question…
If I wanted to have your two scripts (catchall & ghosts) output to individual groups in a ‘developers’ tab, how would I need to change your scripts? Right now I’m using the example from @arsaboo in the other script thread and when I click the button to run the scripts they both show up as their own views.
I’ll throw my thanks for this work in here now, as well.
Take a look at my config…I have added them both to the developer switch. Turning it on adds the developer tab with both the groups and turning it off removes the tabs and the groups.
Right. That’s what I did too. And now it adds both as their own tabs:
What I would like to do is when I turn on the switch I’d like to add them both as groups to one tab (called for example ‘Development’) and control that tab visibility with the switch.
I’m pretty sure I can figure out how to control the new tab visibility as it will work similarly to the way it works now but I’m not sure what changes I need to make to the scripts so that they don’t show as a view but as standard groups in the view.
That’s how I’ve done it.
In the python_script you have to take out the code that creates it as a view.
Link to my repo is up there 
somewhere, with edited versions of the python_scripts and a button that switches on and off a view called debug, containing the ungrouped items on one card and the ghost items on another.
Hope this helps.
1 Like
Hi gang,
Here’s the new version that allows you to ignore items. You can specify either entity_ids or domains. I have given this some testing, but there might still be issues.
@finity one change in this touches on something you said. The sensor is no longer blank, the state in the circle is the number of ghosts, and it has the text “ghosts” below it.
Here’s how I call it using a variation of @anon43302295’s debug button
turn_on:
- service: python_script.populate_catchall_group
- service: python_script.scan_for_ghosts
data:
ignore_items:
- sensor.doesnt_exist
ignore_domains:
- media_player
- service: group.set
data:
object_id: debug
view: true
visible: true
The ignore_items and ignore_domains options can each take multiple values.
And here’s the code
def process_group_entities(group, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains):
# logger.warn("processing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))
processed_groups.append(group.entity_id)
for e in group.attributes["entity_id"]:
domain = e.split(".")[0]
if domain == "group":
g = hass.states.get(e)
if (g is not None) and (g.entity_id not in processed_groups):
process_group_entities(g, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains)
else:
if (domain not in ignore_domains):
grouped_entities.add(e)
# logger.warn("finishing group {}, currently {} grouped items".format(group.entity_id, len(grouped_entities)))
def scan_for_ghosts(hass, logger, data, process_group_entities):
target_group=data.get("target_group","ghosts")
show_as_view = data.get("show_as_view", False)
use_custom_ui = data.get("use_custom_ui", True)
ignore_items = data.get("ignore_items",[])
ignore_domains = data.get("ignore_domains",[])
real_entities = set(ignore_items)
grouped_entities = set()
processed_groups=[]
for s in hass.states.all():
domain = s.entity_id.split(".")[0]
if domain != "group":
real_entities.add(s.entity_id)
else:
if (("view" not in s.attributes) or
( s.attributes["view"] == False)):
real_entities.add(s.entity_id)
process_group_entities(s, grouped_entities, hass, logger, process_group_entities, processed_groups, ignore_domains)
logger.error("{} real entities".format(len(real_entities)))
logger.error("{} grouped entities".format(len(grouped_entities)))
logger.error("{} groups processed".format(len(processed_groups)))
results = grouped_entities - real_entities
logger.error("{} entities to list".format(len(results)))
entity_ids=[]
if use_custom_ui:
summary=""
for e in results:
summary = "{}{}\n".format(summary, e)
hass.states.set('sensor.ghost_items', len(results), {
'custom_ui_state_card': 'state-card-value_only',
'text': summary, 'unit_of_measurement': 'ghosts'
})
entity_ids.append("sensor.ghost_items")
else:
counter=0
for e in results:
name = "weblink.ghost{}".format(counter)
hass.states.set(name, "javascript:return false", {"friendly_name":e})
entity_ids.append(name)
counter = counter +1
service_data = {'object_id': target_group, 'name': 'Ghost Items',
'view': show_as_view, 'icon': 'mdi:ghost',
'control': 'hidden', 'entities': entity_ids,
'visible': True}
hass.services.call('group', 'set', service_data, False)
scan_for_ghosts(hass, logger, data, process_group_entities)
Worked like a charm! Thanks!
1 Like