Add group.all_* with AppDaemon

petro · January 19, 2020, 6:35pm

Add Domain Group (group.all_`<domain>`s)

Group All app for AppDaemon.

Adds a domain group to Home Assistant. The added domain groups will have the name group.all_<domain>s. e.g. if the domain light is specified, group.all_lights will be created.

With the legacy configuration, this will restore all hidden group.all_* groups that were removed in Home Assitant v0.104:

group.all_automations
group.all_covers
group.all_devices
group.all_fans
group.all_lights
group.all_locks
group.all_plants
group.all_remotes
group.all_scripts
group.all_switches
group.all_vacuum_cleaners
group.calendar
group.remember_the_milk_accounts

Installation

Download the group_all directory from inside the apps directory to your local apps directory, then add the configuration to enable the hacs module.

Example App configuration

Legacy Configuration

# Creates all legacy groups.  All newly created entity_id's will be added to the correct legacy group.
groups:
  module: group_all
  class: GroupAll

Only all switches

# Creates group.all_switches, the group will update when a new switch is added to home assistant.
groups:
  module: group_all
  class: GroupAll
  domains: 
  - switch

Advanced

# Creates groups for all domains.
# Renames group.all_switches to group.switch_master
# Renames group.all_lights to group.light_master
# Any new entity_id that is created will be added to the correct group.
groups:
  module: group_all
  class: GroupAll
  domains: all
  track_new_entities: true
  group_config:
    switch:
      name: Switch Master
    light:
      name: Light Master
  log_level: INFO

App Configuration

key	optional	type	default	description
`module`	False	string	group_all	The module name of the app.
`class`	False	string	GroupAll	The name of the Class.
`track_new_entities`	True	bool	`true`	Dynamically adds new entity_id’s to the entity list of a domain group.
`domains`	True	`'all'` \| `'legacy'` \| list	`'legacy'`	A list of domain names. `'all'` will create all available domains at startup. `'legacy'` will create all avialable legacy domains. Adding your own list will only create those domain groups.
`group_config`	True	map		A map for domain names.
`log_level`	True	`'INFO'` \| `'DEBUG'`	`'INFO'`	Switches log level.

Domain Map Configuration

key	optional	type	default	description
`name`	False	string		Friendly Name of the created group.

EDIT

Version 1.1 is out. There are a few breaking changes. This should be easier to configure and it’s more optimized. Unfortunately, the trade off for better performance was to remove the ability to find new domains.

Mariusthvdb · January 21, 2020, 7:23am

HI Petro,

though my legacy groups are created the AD log throws this:

2020-01-21 08:15:26.603265 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:26.604559 WARNING AppDaemon: message: (legacy entity_id tracking) Adding domain script., args: ('DEBUG',)
2020-01-21 08:15:26.652417 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:26.653732 WARNING AppDaemon: message: (legacy entity_id tracking) Updating group.all_scripts with 275 entities., args: ('DEBUG',)
2020-01-21 08:15:26.819863 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:26.821061 WARNING AppDaemon: message: (legacy entity_id tracking) Adding domain light., args: ('DEBUG',)
2020-01-21 08:15:26.854826 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:26.856060 WARNING AppDaemon: message: (legacy entity_id tracking) Updating group.all_lights with 98 entities., args: ('DEBUG',)
2020-01-21 08:15:26.902577 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:26.904107 WARNING AppDaemon: message: (legacy entity_id tracking) Adding domain switch., args: ('DEBUG',)
2020-01-21 08:15:26.932342 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:26.933974 WARNING AppDaemon: message: (legacy entity_id tracking) Updating group.all_switches with 75 entities., args: ('DEBUG',)
2020-01-21 08:15:26.974627 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:26.976150 WARNING AppDaemon: message: (legacy entity_id tracking) Adding domain automation., args: ('DEBUG',)
2020-01-21 08:15:27.020514 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:27.021675 WARNING AppDaemon: message: (legacy entity_id tracking) Updating group.all_automations with 208 entities., args: ('DEBUG',)
2020-01-21 08:15:27.092467 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:27.094044 WARNING AppDaemon: message: (legacy entity_id tracking) Adding domain device_tracker., args: ('DEBUG',)
2020-01-21 08:15:27.131735 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:27.133073 WARNING AppDaemon: message: (legacy entity_id tracking) Updating group.all_devices with 107 entities., args: ('DEBUG',)
2020-01-21 08:15:27.238837 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:27.240384 WARNING AppDaemon: message: (legacy entity_id tracking) Adding domain remote., args: ('DEBUG',)
2020-01-21 08:15:27.252379 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:27.253490 WARNING AppDaemon: message: (legacy entity_id tracking) Updating group.all_remotes with 5 entities., args: ('DEBUG',)
2020-01-21 08:15:27.280510 WARNING AppDaemon: Log formatting error - 'not all arguments converted during string formatting'
2020-01-21 08:15:27.282107 WARNING AppDaemon: message: (legacy entity_id tracking) Creating listen state handle., args: ('DEBUG',)

probably not a true issue, but nonetheless…

btw for my AD learning purpose: why did you call this App ‘events’? And not ‘group_all’ ? or is that simply a c&p remnant from another app…

petro · January 21, 2020, 12:44pm

It hasn’t been updated for appdaemon 4. I believe logging changed, not entirely sure, I’ll look into and post a change. I haven’t updated to AD4, when that happens, I’ll make official builds.

petro · January 21, 2020, 12:50pm

I’m not sure why logging was changed. It apparently needs some extra single argument? Not entirely sure. Doc’s don’t mention the required argument but all the examples do.

@ReneTode care to comment?

Is this a valid log line?

self.log('foo', level='DEBUG')

if not, why? EDIT: Nevermind.

petro · January 21, 2020, 12:52pm

change line 175

self.log(f"({tracking} tracking) {msg.capitalize()}", self._level)

to

self.log(f"({tracking} tracking) {msg.capitalize()}", level=self._level)

Mariusthvdb · January 21, 2020, 1:03pm

Correct, reverted to Ad3 and all log errors were gone

ReneTode · January 21, 2020, 1:55pm

you already found out it is
yeah keyword for level is now required, more in the line from regular logging.

great that you created this app.
but i would only listen for the selected domains. that will reduce the amount of callbacks quite a bit.

petro · January 21, 2020, 2:03pm

Couldn’t find out how to do that with the documentation. I wanted to do that. What’s the call?

EDIT: Namespace seems like it was integrations, not domain.

ReneTode · January 21, 2020, 2:38pm

self.listen_state(self.cb, “domain”)

entity ( str , optional ) – name of an entity or device type. If just a device type is provided, e.g., light, or binary_sensor. listen_state() will subscribe to state changes of all devices of that type. If a fully qualified entity_id is provided, listen_state() will listen for state changes for just that entity.

namespace is something else.
in general is a namespace a collection from entities and events.

everything from HA is in 1 namespace (if not set in the plugin settings it is default and doesnt need to be provided)
every plugin has 1 or more namespace, and you can use AD internal namespaces.

petro · January 21, 2020, 2:41pm

Ah yes, but you can’t feed it a list of domains to track. So I’d need to make n listeners with less callbacks or 1 listener with n callbacks. Not sure which one is less taxing.

ReneTode · January 21, 2020, 2:54pm

the difference wouldnt be great when you got all possible groups (allthoug sensors were not grouped, so if someone has a lot of sensors, it would be a great difference too)

but for sure if someone only uses 1 domain, the difference would be significant.
and you could also create the option that the group is only created once, and not updated. (make use of ONESHOT)

petro · January 22, 2020, 4:02am

I’ve been running this a few days on all, the performance hit is non existent. I think you’re over estimating how many resources the callback is actually using. It’s 2 if statements that require almost no computing power. The only ‘taxing’ manipulation that happens is the string split.

How does appdaemons core work? I’ve never bothered looking into it. Does each callback register a new thread? Most apps* keep the callback in the same thread. And if that’s the case, then creating more listeners would be worse for the system then having less callbacks. Typically more threads = more cpu tax.

EDIT: By apps i mean applications. Not referring to appdeamon apps.

ReneTode · January 22, 2020, 10:45am

callbacks from an app are all in 1 thread.
so you use 1 app, so 1 thread.

and believe me, with a very busy system, you want to reduce the amount of callbacks.
but take a look at the admin interface. you will see the amount of callbacks your system has. i think it opens your eyes about how busy you keep the system if you listen for every state change.

thats why i also suggested the oneshot option. groups would only be build on startup, without the cost of extra cpu.

my system is already up to 100.000 callbacks a day. if i add your app as is, i would probably double that easy.

if you just add 1 listener for each domain it would also be possible to create 1 app for each domain.
using 1 thread for each domain. which would reduce the risk of building up a queue on very busy systems.

petro · January 22, 2020, 12:48pm

Ah ok, so the callbacks from all listeners go into a queue?

If you turn on the dynamic updating. You can turn that off so it only updates on startup:

events:
  module: group_all
  class: GroupAll
  track_new_domains: none
  track_new_entities: false

petro · January 22, 2020, 12:50pm

Know the thread count makes a difference. I’ll swap the domain listeners. That won’t be too difficult.

There still isn’t a better way for finding new domains. Maybe an event gets created when new items are added. I’ll play around and see if that’s the case.

ReneTode · January 22, 2020, 12:55pm

i dont see you using any constraints, or oneshot option.
so it only updates once, but still generates callbacks for every state change.

im not sure, but i believe there is an event for new entities.
but it would make your app more complicated, because then you need to take care of 2 type of events (startup/reload and entity added)

i dont know if that would be worth the trouble, but for complicated systems it would make a difference.

petro · January 22, 2020, 12:57pm

Nope, it does not create the ~~callback~~ listener:

        if track_domains != OPTION_NONE or self._track_entities:
            # Listener to create new everything on state changes.
            self.applog("creating listen state handle.")
            self.handle = self.listen_state(self.track_new_items)

petro · January 22, 2020, 12:59pm

It wouldn’t take much effort. It’s registering for an event vrs listening to all state changes. I built the app to be easily modifiable because I knew these conversations would likely happen.

ReneTode · January 22, 2020, 1:22pm

ah, ok. i missed that you left out the listen_state and called the add_group from initialise directly.
i would have chosen to create the groups on the first state change, i think.

just trying to give you info to optimise, because ts an app that has a chance to be used by quite a few people.

and i really appreciate your efford!

petro · January 22, 2020, 1:28pm

Yeah, I’m going to optimize it. I was under the impression that listeners were in their own threads. I’ve tailored most of my apps this way.

Edit: Plus I need to update the one log line to use the kwarg level. So I’m going to do it in optimizing/fixing in one shot.

Where is this admin interface? I asked you before about the move from HACS to that interface but I realized that I don’t know what interface you are referring to.