ControllerX. Bring full functionality to light and media player controllers

duceduc · July 27, 2020, 10:02pm

This looks like a cool setup. What media players are using with the dimmer controller?

Sennevds · July 28, 2020, 6:26am

I use a combination of multiple Chromecasts and a Google Home. But in theory you could add anything that’s controllable from Hass

xaviml · July 30, 2020, 5:01pm

Hi @madrian,

Glad that it worked What you are mentioning is too specific to be added. This is something that can be done with Custom Controllers and callback constraints from AppDaemon, however, the switch between light with a double click will not work since the E1810 does not support double click. If you are interested, I can help you build the configuration for such behaviour.

Cheers,
Xavi M.

xaviml · July 30, 2020, 5:10pm

Hi @pav,

The z2m integration as well as the state integration will listen to changes on the state, so if there is no change on the state, it will not trigger the action. For the case of z2m is done like this since Zigbee2MQTT always clears out the entity state so every event will change from empty to the name of the entity. Knowing that your state does not change I am guessing that you are not either using Zigbee2MQTT or you have an special setting to that device that makes that happens. However, if this is how your controller works and what it changes is an attribute state, then I am afraid that as of now this is not supported in ControllerX, but I could add an argument to the state integration (a more generic one) to specify the attribute we are listening.

Let me know what you think about this.

Regards,
Xavi M.

pav · July 30, 2020, 6:53pm

Hi @xaviml,

So it seems my ‘diagnosis’ was spot on …

I also have noticed this behaviour previously, BUT it definitely is useless to detect subsequent identical events. This is because it takes anywhere from several seconds to several minutes, maybe hours (just tested it: took over half an hour …), before the state is cleared (timespan depending on god knows what) - waaaaaaay too long for being useful to cause detection of a state change from null to some-state !

I am definitely using z2m as my gateway, without any special setting for my Hue Dimmer controllers.

I think it’s fair to say that Hue Dimmers are not of the exotic kind with strange behaviour, and they certainly do NOT work by changing just an attribute. They do supply extra attributes (eg link quality), but they definitely supply state in the form of the button-press action name as well.

I was a bit disappointed to find out that ControllerX’s z2m Integration seems to be no more than the State Integration, having nothing to do with z2m as such ? The more so because Z2M DOES publish the controller’s state for EVERY event/button press, duplicate or not …

Food for thought, I hope ?
Best, Paul

xaviml · July 30, 2020, 7:09pm

Hi @pav again,

So interesting that you have just z2m working normally. Do you experience this behavior with other controllers? I have experience that once with the symfonisk controller with Z2M and it was due to the debounce attribute on z2m device configuration.

There is something else you could try which I have not mentioned before. z2m integration works just with the state of the sensor that is linked to HA. However, z2m works with MQTT and ControllerX leverages that to skip one layer in the communication. For this, I recommend you using the MQTT integration, all you need to change is the integration: z2m for integration: mqtt and the controller attribute will be the zigbee2mqtt topic to listen for the changes, it is normally in the form of zigbee2mqtt/<friendly name>/action or zigbee2mqtt/<friendly name>/click, you can check Z2M logs after pressing a button and you will see the topic that z2m fired the event. In addition to this, you will need to change your appdaemon.yaml. Please follow the MQTT section if you are interested: https://xaviml.github.io/controllerx/others/integrations#mqtt

Regards,
Xavi M.

xaviml · July 31, 2020, 7:17am

As much simple as it sounds, it is how Zigbee2MQTT works with Home Assistant. Each button press changes the state from empty to the action and immidiatly from the action to empty, as simple as that. For now z2m and state integration in ControllerX are almost the same. However, state integration will serve the generic purpose for any actions that are fired by state changes, like listening to state attribute changes. Whereas z2m will be only related to zigbee2mqtt.

For this reason, I recommend you using the mqtt integration that will work better for zigbee2mqtt. I have plans of changing the z2m integration to work directly with MQTT, but I didn’t want to do that yet due to breaking changes.

Cheers,
Xavi M

pav · July 31, 2020, 8:30am

Hi @xaviml,

Thanks for bearing with me

As I explained, that may be the theory (although I’ve never seen it stated like this) but in practice it definitely does NOT change immediately from action to empty, taking minutes, if not hours, to revert to empty. And thus one cannot count on this to detect presses of ‘same’ buttons. If you’re interested, and if it would be useful to you, I could explain what I think is the reason for this, and why it’s not an easy thing to solve. I have come to believe that the only way to reliably deal with controllers/remotes is by monitoring events, not states (my 2 cents) …

I had no doubts about this, but I made sure testing with the one ‘other’ controller I’m using (a Xiaomi WXKG01LM). Sorry to say, but I can confirm it exhibits the exact same problem.

I’m not one that starts complaining before testing and trying everything possible I can think of , so I tried the mqtt integration already before I even hit upon my problems - spurred by the rationale of cutting an intermediate level and the expected speed benefit. Alas, I could not get it to fire any event yet, and I fail to see what I did wrong - as all logs showed ok, with appdaemon connecting to mqtt, with setting the app’s parameters exactly like you suggest. I will however try to get to the bottom of this, and keep you posted on any success or further failure.

Xavi, now that I have your attention , please allow me to throw in a feature request: I think it would greatly enhance the scope and the ease-of-use of ControllerX if you would see fit to allow for templating some of the app’s parameters, e.g. the ‘controller’ entry. This would in many cases obviate the need to have to resort to using the CallServiceController iso the more simple and straightforward basic Controllers. Although it’s easy for me to say I think this is very doable, as Appdaemon features a simple API call to let HASS resolve any given template.
What do you think ?

Have a nice weekend,
Paul

pav · July 31, 2020, 1:13pm

Hi @xaviml,

Update: I’ve been testing with the MQTT integration, and the only reasonable conclusion is that it simply doesn’t work - at least not in combination with Z2M. Could it be that the relevant parts of ControllerX and the docs are based upon an older version of Z2M that did things differently, at least as integrated with HASS ?

I have been able to establish, watching AD & Z2M logs and the AD administrative interface, that ControllerX receives each and every mqtt-published message on the watched topic, but then does NOTHING with it - not even with the first button press, let alone with following presses.

My findings:

the only topic Z2M publishes to, is of the form zigbee2mqtt/<friendly name>, not suffixed with action, switch or whatever
what’s worse, and what breaks the integration, is that that topic’s payload contains a json, and NOT the direct action’s name. It looks like this : {“battery”:100,“linkquality”:0,“counter”:1,“brightness”:1,“update_available”:false,“action”:“down-press”,“duration”:0} - note the ‘action’ key in there …

I guess this is fixable ?

Cheers,
Paul

xaviml · July 31, 2020, 5:36pm

Hi @pav

No worries, thank you for bearing with me too. I am jut trying to help as much as I can

I am not able to find this behaviour on the Zigbee2mqtt documentation, but I have seen this behaviour since I started using Zigbee2Mqtt one year ago. In fact, I have never seen anyone saying that the state changes after hours, this is why I am telling you that this is not normal and zigbee2mqtt should send 3 topics everytime a button is pressed:

zigbee2mqtt:info  2020-07-31 19:20:04: MQTT publish: topic 'zigbee2mqtt/livingroom_controller', payload '{"linkquality":92,"update_available":false,"battery":47,"action":"toggle"}'
zigbee2mqtt:info  2020-07-31 19:20:04: MQTT publish: topic 'zigbee2mqtt/livingroom_controller', payload '{"linkquality":92,"update_available":false,"battery":47,"action":""}'
zigbee2mqtt:info  2020-07-31 19:20:04: MQTT publish: topic 'zigbee2mqtt/livingroom_controller/action', payload 'toggle'

You can see that the action and the empty one are sent one after the other and they are sent consecutively in the same second. I do not have any special configuration on Z2M and I believe this is what it is expected to do.

Could you give me an example of a configuration with templating in the controller attribute? I do not quite understand what you mean I think. Could you also point to the AppDaemon documentation for the API call to HA to resolve any given template? This could be interesting.

Which version of z2m are you using? Do you use the z2m addon? I use zigbee2mqtt-1.14.2, the latest stable release and as I showed before, it publishes 3 topics, 2 with zigbee2mqtt/<friendly name> and another one with zigbee2mqtt/<friendly name>/action. Could you share the z2m logs when a button is pressed?

Cheers,
Xavi M.

pav · July 31, 2020, 6:58pm

Hi @xaviml,

I know, Xavi, and I sincerely appreciate not only the creation of a very nice app, but also your support which is second to none.

About the templating: one of my use cases would be for my custom mediaplayer control, for which I am now forced to use the CallServiceController because I want/need it to control not a fixed media player, but rather a dynamic one - or even a set. Now I’m using templating within the called scripts, but most of that could be done away with if I could e.g. template the ‘controller’ entry. So, if I could set the controller to e.g. “{{ states(‘sensor.current_media_player’) }}”, that would be really helpful.
The API I was referring to is this one : render_template ( self , template , **kwargs )
https://appdaemon.readthedocs.io/en/latest/HASS_API_REFERENCE.html#appdaemon.plugins.hass.hassapi.Hass.render_template

Regarding Z2M behaviour: weird how its behaviour is so different. I have no idea what could cause such a difference, as I run a very standard setup. FYI: My Z2M is still on 1.13.0, a separate install on a HASS env/HASSBIAN installation (latest version). The change from some-state to empty typically takes about half an hour and I believe it only happens when the device sends one of its regular automatic ‘attribute update’ messages with e.g. up-to-date info on link quality or so - but then without an ‘action’ field, causing the action to reset to empty.

This is the only entry in the z2m log after a button press :

jul 31 20:42:30 HASSBIAN npm[719]: zigbee2mqtt:info 2020-07-31 20:42:30: MQTT publish: topic ‘zigbee2mqtt/Music Remote’, payload ‘{“battery”:100,“linkquality”:23,“counter”:1,“brightness”:1,“update_available”:false,“action”:“down-press”,“duration”:0}’

I may install the latest z2m tomorrow to find out if this accounts for the big difference. Who knows, this may solve the mistery and my problem
I’ll keep you posted, ok ?

Greetz,
Paul

pav · August 1, 2020, 11:40am

Hi @xaviml,

Following up on things I upgraded Z2M this morning to latest version, but as I somewhat expected this did not make any difference whatsoever.

In the meantime I also did some extra research, which makes me wonder about a couple of things.
The first being: regarding the mqtt integration you insist on using the ‘zigbee2mqtt/friendly name/action’ topic. Strangely enough such topic does not normally exist, according to Z2M’s documentation : https://www.zigbee2mqtt.io/information/mqtt_topics_and_message_structure.html#zigbee2mqttfriendly_name
The latter confirms the use of the topic as it appears in my setup, the one of the form ‘zigbee2mqtt/friendly name’, and the fact that the data in there is in JSON format.
This is further confirmed here: https://www.zigbee2mqtt.io/integration/home_assistant.html#via-mqtt

All of this leads me to wondering how exactly you included your devices. Could it be that you somehow manually added these ‘action’ topics, instead of relying on the automatic MQTT discovery as I do ?

Finally, some good news: although my Python-skills are virtually non-existant, I managed to modify your controllerx.py to deal with json content instead. Happy to say that first tests show it then solves my initial problem by properly detecting subsequent presses of the same button.
So, all’s not lost

Cheers,
Paul

htvekov · August 1, 2020, 9:49pm

hi’ @pav

I’ve no idea what causes all these z2m problems at your end ?
I’m pretty sure though, that you and Xavi are much more competent finding a solution for your strange issues

But something is definitely wrong with your z2m setup, since you don’t get states cleared immediately after action payload has been sent ? All my some 10 remotes (five different brands/versions or so) all reports in the same manner as Xavi describes. Remotes clears action payloads instantly after issuing first payload. I’ve been using z2m for 1½ years, and never experienced any other behaviour.

zigbee2mqtt:info  2020-08-01 23:06:53: MQTT publish: topic 'zigbee2mqtt/Office Ikea 5 button', payload '{"linkquality":92,"update_available":false,"battery":37,"action":"toggle"}'
zigbee2mqtt:info  2020-08-01 23:06:53: MQTT publish: topic 'zigbee2mqtt/Office Ikea 5 button', payload '{"linkquality":92,"update_available":false,"battery":37,"action":""}'
zigbee2mqtt:info  2020-08-01 23:06:53: MQTT publish: topic 'zigbee2mqtt/Office Ikea 5 button/action', payload 'toggle'

At the moment I can only think of Hue remotes (via HA Hue Bridge integration) that keeps their last state until new key is pressed.

Hope you get your issues resolved soon

Ciao !

xaviml · August 2, 2020, 8:03am

Hi @pav

Thanks for pointing this out. I created an issue in the GitHub repository for this enhancement.

This is interesting. As @htvekov said, there must be something different and probably the controller does not support the /action topic. However, it is still even stranger that the state does not change immediately as it should. This makes me thing that you might be running to many processes in a small machine. I currently use Raspberry 4 with 2GB.

This is also not clear to me since I really tried to find documentation on the /action or /click topic and I could not find anything. We could probably reach out the z2m community to solve this and now what is happening.

Indeed, you are right. I know this topic exists and I also receive it in my logs, however, I did not decide to integrate this topic in ControllerX due to having the other topic. Seeing this is not very clear, I will give support to the zigbee2mqtt/<friendly_name> topic in ControllerX since it seems to be the official by documentation. I created this issue for this.

No, I do not have any special setting for z2m. In fact, I recently reinstalled the addon so I have all the default config.

Nice! Did you parse the json object to get the action field? As I mentioned, I opened an issue so people can use the z2m integration with mqtt capabilities. It is something I had planned to do anyway

Thanks for all your insights,
Xavi M.

pav · August 2, 2020, 1:34pm

Hi @xaviml,

Looking forward to your templating enhancement I made a little comment on Github.

Well, mine is a Pi 3B with 1GB - not as powerful as yours, but certainly not stressing under its load. And anyway, performance problems could account for say a couple of secs, but surely not for delays of over half an hour ?
I guess for now it remains a mistery what causes these substantial Z2M’s differences

About the mqtt topic issue: also on this issue I took the liberty of leaving a comment on github.

Indeed, I just added a couple of parsing statements in a very quick and dirty way (just extracting the ‘action’ key) which (much to my surprise ) did the job. No more than a rude proof-of-concept …

Cheers amigo - and keep safe !
Paul

carsten_h · August 2, 2020, 4:21pm

Hello!

Is it real that the Ikea ICTCG1 only allows those left and right events and no keypresses?
Does someone use it for anything?

Is it possible to eg. switch off a lamp when it is completely dimmed and switch it on when dimming back to right?

htvekov · August 3, 2020, 12:02am

Hi’ @xaviml and @pav

It’s definitely not the controller that ‘misbehaves’ using z2m.
WXKG01 remote works perfectly here and z2m issues the usual three mqtt topics.
Issue must lie elsewhese

Note: its not an action but click payload for… action
Ciao !

office4_controller:
  module: controllerx
  class: WXKG01LMLightController
  controller: 'zigbee2mqtt/Doorbell Utilitydoor/click'
  integration: mqtt
  
  #controller: sensor.0x00158d00035a660f_click #Xiaomi round remote
  #integration: z2m
  
  automatic_steps: 8
  delay: 400
  smooth_power_on: true
  light: light.0xec1bbdfffed45c3b_light

zigbee2mqtt:info  2020-08-03 01:55:46: MQTT publish: topic 'zigbee2mqtt/Doorbell Utilitydoor', payload '{"battery":100,"voltage":3022,"linkquality":120,"click":"single"}'
zigbee2mqtt:info  2020-08-03 01:55:46: MQTT publish: topic 'zigbee2mqtt/Doorbell Utilitydoor', payload '{"battery":100,"voltage":3022,"linkquality":120,"click":""}'
zigbee2mqtt:info  2020-08-03 01:55:46: MQTT publish: topic 'zigbee2mqtt/Doorbell Utilitydoor/click', payload 'single'

pav · August 3, 2020, 8:41am

Hi Henning,

I do not doubt that your controllers work perfectly. And my WXKG01 (using ‘click’ iso ‘action’ - of course) works as ‘good’ as my Hue Dimmers, but suffers from the exact same problem.

But the fact remains that ‘my’ Z2M publishes topics strictly according to the official docs - and yours (and Xavi’s) does not quite, as Xavi admitted freely and as you can easily check for yourself. But no worries, as it’s rather easy to solve this discrepancy, and I’m sure Xavi is on top of it.

Cheers and thanks for chiming in,
Paul

htvekov · August 3, 2020, 11:33am

Hi’ @pav & @xaviml

Well, ‘Google is your friend’

It took some time to actually ‘filter out’ the answer I was looking for, but evetually I DID find it.
Both yours and mine (and Xavi’s) setup and MQTT commands are actually as expected and documented. But I must admit, that it was very hard to find some documentation on the specific Home Assistant legacy entity state implementation commands.

Please check this issue and especially check out Koenkk’s replies April 22nd, 2020

With homeassistant: true set in configuration.yaml you’ll get the extra MQTT command that clears state, in order to be able to trigger directly on state change in HA. Otherwise it would be impossible (as you’ve experienced) to detect multiple state changes on same entity_id. Makes perfect sense for all z2m click devices.

If you only use MQTT device triggers, it’s possible to completely disabe the HA entity integration with homeassistant_legacy_triggers: false

Recommended method is still using MQTT device triggers

Can’t test the above as I’m at work, but I’m quite sure that’s why we experience different MQTT commands in the logs.

Do you have the homeassistant: true set in your configuration.yaml, @pav ?

Ciao !

xaviml · August 3, 2020, 5:21pm

Hi @htvekov,

Thanks for finding this! I really tried to find why this was happening, but I could not. However, the documentation does not say what the homeassistant: true fully does. I did just try to set homeassistant to false in the configuration and indeed, I do not receive the /action and /click topic in the MQTT.

Anyway, I will give full support to the other topic in ControllerX since it is the official topic from Z2M.

Cheers,
Xavi M.