Native MQTT push updates from Hue Hub

Configuration
1

Hi,

I’m trying to achieve real-time updates of my Hue Dimmer Switches (+ potentially motion sensors in future), connected via a Hue Bridge (2.1).
I know most of you guys are probably preferring to do it using using zigbee on the HA host directly, but to me, this is mostly about the challenge of making it work using that proprietary embedded device called “Hue bridge” - I’m aware there’s an easier route :slight_smile:

OK, so first I gained root access to the Hue bridge using pin2pwn, described here

( Adapted for bridge 2.1 here )

Voilá:
grafik

After poking around on the device (based on OpenWRT), tweaked a few things - got OPKG working, wrote a few basic scripts.

Along the path, I noticed that the device is using MQTT for internal communication. It’s just not reachable from outside by default.
… But after adding a custom rule to iptables on the Hue bridge it now is :slight_smile:

Then I bridged MQTT messages from Hue to HA’s MQTT plugin as described by Harry for cloudMQTT here

I can now see MQTT messages from Hue, with the interesting topic being “cmd/clip/event/publish”, which publishes status updates for all connected devices in json. Here’s an example to show the syntax:

{
    "data": [
        {
            "creationtime": "2021-04-14T18:00:46Z",
            "data": [
                {
                    "dimming": {
                        "brightness": 50
                    },
                    "id": "104b88d7-9322-43db-ba48-a77248f06fab",
                    "id_v1": "/lights/4",
                    "type": "light"
                }
            ],
            "id": "3ccd7574-5c44-44b4-bdb4-ad967d51f2c7",
            "type": "update"
        }
    ],
    "metadata": [
        {
            "key": "event_origin",
            "value": "zigbee"
        }
    ]
}

Now I’m looking for directions to an elegant way to proceed from here. Ideally, I would like to retain my current configuration using the default Philips Hue integration (API) and add the ability to update my device states with data from mqtt.

Any help would be much appreciated.

Regards
John

2

Still looking for help - Shortened content to make my goal more comprehensive

3

Hacking the Hue hub was still on my to do list, but I had little use for it… but nobody mentioned before that mosquitto is running on the hub :open_mouth:. So after reading your initial post I hacked my hub too and got the same results. Used a similar construction to bridge the mqtt messages to HA so I now have the ‘cmd/clip/event/publish’ topic available in my HA mosquitto instance.

Next day I tried to integrate in HA, but for some reason the Hue hub was not accepting mqtt connections anymore. I suspected that my rooting was found out and some anti-tamper protection had kicked in :face_with_raised_eyebrow:. Took me a while to figure out that exactly a day after rooting the Hub, Signify decided to release a new firmware containing a new version of mosquitto. This new versions defaults to localhost access only, so I needed to configure a new listener in order to make the bridging work again (see: Migrating from 1.x to 2.0 | Eclipse Mosquitto). Anyway… today I had some time to continue with the HA side of things.

My goal was to get instant motion sensor events available in HA. An example messages of a motion sensor in Hue mqtt is:

{
   "data":[
      {
         "creationtime":"2021-04-19T18:33:15Z",
         "data":[
            {
               "id":"1ab6b9b1-5790-4634-8e0f-0b8ac2ab4e43",
               "id_v1":"/sensors/63",
               "motion":{
                  "motion":false
               },
               "type":"motion"
            }
         ],
         "id":"a59c45b1-d4bb-407f-9bc7-febae7835cb7",
         "type":"update"
      }
   ],
   "metadata":[
      {
         "key":"event_origin",
         "value":"zigbee"
      }
   ]
}

Unfortunately all events of all devices are published in this single topic and the structure also contains some nested json attributes. So we need to parse this in 2 steps:

  1. Create a generic MQTT sensor to get the events available in HA and use the value_template to get to the interesting part of the json:
#---------------------------------------------------------------------------------------#     
sensor:
  - platform: mqtt  
    name: "Hue event"  
    state_topic: "cmd/clip/event/publish"  
    value_template: "{{ value_json.data[0].data[0].id_v1 }}"
    json_attributes_topic: "cmd/clip/event/publish"
    json_attributes_template: "{{ value_json.data[0].data[0] | tojson }}"
#---------------------------------------------------------------------------------------#

The value_template contains the ‘id_v1’ attribute and the the json_attributes contains the rest:

{
   "id":"1ab6b9b1-5790-4634-8e0f-0b8ac2ab4e43",
   "id_v1":"/sensors/63",
   "motion":{
      "motion":false
   },
   "type":"motion"
}

Using this hue event sensor we now can create template (binary)sensors for the different Hue devices. In this case I used a template binary_sensor to create a motion sensor in HA.

#---------------------------------------------------------------------------------------#
binary_sensor
  - platform: template
    sensors:
      hue_motion_hallway:
        friendly_name: "Motion hallway"
        value_template:  "{{ (state_attr('sensor.hue_event', 'id_v1') == '/sensors/63') and (((state_attr('sensor.hue_event', 'motion') | tojson).split(': ')[1]).split('}')[0] == 'true') }}"
        delay_off: 00:00:15
        device_class: occupancy
#---------------------------------------------------------------------------------------#

The tricky bit in the attribute json is the ‘motion’ attribute; it’s value is ‘{“motion”:false}’ which is not easy parse in HA. In this case the only way to parse…I mean ‘hack’…this, is by first converting it to json and then use a double split function to parse out the value. It’s nasty but it works.

If this works for you, please consider to re-add the part how to root the Hue bridge back in your initial post so others can replicate this method for Hue event integrations :slightly_smiling_face:.

4

Happy to see you’re sharing my excitement for this approach :v:
& Thanks a lot for sharing your findings!!

It looks like we have been digging into HA at roughly the same time, even hit the same issue with MQTT 2.0.
That must have been caused by firmware update 1944102110 of the hub.
Difference is that my learning curve with templating was probably a bit steeper than yours.

I had my hub hacked for a while now, and this is not the first time patches break my use case, I can only recommend to turn automatic updates off and only update when looking for trouble :expressionless:
On one of the previous occasions, curl had been crippled on my hub (noticed it after upgrading to 1943082030)
In case it helps anyone… Managed to get it working again by reinstalling libcurl from here:
https://archive.openwrt.org/chaos_calmer/15.05.1/ar71xx/generic/packages/base/libcurl_7.40.0-3_ar71xx.ipk

Another interesting thing I found was that you can also “press” the push-link button on the hub via MQTT. Will implement this in HA when I have more time, should be a piece of cake after this templating adventure :slight_smile: Anyways, here’s how to simulate the buttonpress via shell:

mosquitto_pub -t button/link -m pressed
sleep 1
mosquitto_pub -t button/link -m released

I don’t have my motion sensors yet, but with your input they will surely have a smooth start :+1:
My primary goal so far was to get state changes of my dimmer switch to assign custom functionality, and I used a very similar approach.
Although I went for 2 mqtt sensors to work around the “nested-json” issue and had to use the cryptic ID, because all buttons on the dimmer switch share the same ‘ID_V1’.

Soo… results of my recent HA journey - Example dimmerswitch message from hue:

{
    "data": [
        {
            "creationtime": "2021-04-18T22:56:40Z",
            "data": [
                {
                    "button": {
                        "last_event": "short_release"
                    },
                    "id": "9f72f4c5-4af5-4695-b33a-9c7814bcdd76",
                    "id_v1": "/sensors/8",
                    "type": "button"
                }
            ],
            "id": "2b18f2b1-493e-4ace-ac3f-2588c91767f9",
            "type": "update"
        }
    ],
    "metadata": [
        {
            "key": "event_origin",
            "value": "zigbee"
        }
    ]
}

sensor:
  - platform: mqtt
    name: "HueMQTT"
    state_topic: "cmd/clip/event/publish"
    value_template: '{{ value_json.data[0].creationtime }}'
    json_attributes_topic: "cmd/clip/event/publish"
    json_attributes_template: '{{ value_json.data[0].data[0] | tojson }}'
  - platform: mqtt
    name: "HueMQTT_button"
    state_topic: "cmd/clip/event/publish"
    value_template: '{{ value_json.data[0].creationtime }}'
    json_attributes_topic: "cmd/clip/event/publish"
    json_attributes_template: '{{ value_json.data[0].data[0].button | tojson }}'

automation:
- id: '1613684302736'
  alias: Dimmschalter K OFF
  description: ''
  trigger:
  - platform: template
    value_template: '{{ is_state_attr(''sensor.huemqtt_button'', ''last_event'', ''initial_press'')
      }}'
  condition:
  - condition: template
    value_template: '{{ is_state_attr(''sensor.huemqtt'', ''id'', ''9f72f4c5-4af5-4695-b33a-9c7814bcdd76'')
      }}'
  action:
  - service: media_player.toggle
    target:
      entity_id: media_player.squeezebox_boom
  mode: single

Next on my list until I get some motion sensors:
Extracting availability state of Hue Bulbs from MQTT to use as trigger. But it’s getting late :v:

5

I have been studying the Hue mqtt messages, but the ‘cmd/clip/event’ topic only seems to contain a very limited set of events. For instance: the motion sensors also measure lightlevels and temperatures, but these events do not get posted in this topic. Also not all light status information seems to be posted. It feels like the MQTT service on the Hue hub may be part of the Homekit interface which has been bolted on afterwards.

On topic of usefullness of this Hue MQTT interface: To deduct the state of e.g. the bulbs (and other Hue devices) from ‘listening’ to state changes seems to be a very complicated way to do this. But then again, I could be an interesting challenge… :wink: The other topics in ‘dt/clip/*’ do seem to contain the state information you want, but I have not yet managed to decypher the binary content.

At the moment I have migrated all my motion sensors from z2m to this mqtt method and the responsiveness of the motion sensors is instant. Z2m is also quick, but I noticed that the motion sensors do disconnect sometimes from the z2m zigbee network and need time (and battery energy) to reconnect. They don’t seem to do this when connected to the Hue hub.

6

Luckily the hub sends specific messages when connection to a zigbee device is lost. So here’s an example of triggers I ended using for my automations to detect when lights have been switched of using the normal wall-lightswitch:

{{ is_state_attr(''sensor.huemqtt'', ''status'', ''connectivity_issue'') }}
{{ is_state_attr(''sensor.huemqtt'', ''status'', ''connected'') }}

I agree, listening to these state changes feels kind of wrong. However, listening is still way better than polling like with the normal hue integration :slight_smile:

And you’re probably right that mosquitto on the bridge stems from homekit. I have seen some mqtt messages that may indicate this. What’s more is that I don’t recall having seen mosquitto on the bridge when I had initially hacked it a few years ago. But I think I can live with the temperature/lightlevel information not being real-time (Assuming the default hue integration can provide these values). I’m wondering though why Homekit would need to be able to “press” the push-link button on the Hub (I’m sure this was done differently pre-mqtt), but I have no experience with Homekit.

All in all, getting HA to utilize Hue mqtt has been a fun digression and for real-time button presses and motion detection I think it’s definitely useful and will keep me from buying a zigbee stick for my HA host :slight_smile:

…For sake of completeness, here is my script for simulating push-link button press from HA:

'1619022870074':
  alias: Hue Push-Link
  sequence:
  - service: mqtt.publish
    data:
      topic: button/link
      payload: pressed
  - delay:
      hours: 0
      minutes: 0
      seconds: 1
      milliseconds: 0
  - service: mqtt.publish
    data:
      topic: button/link
      payload: released
  mode: single
7

What does this technique, using MQTT, provide that the existing homekit_controller integration does not?

If MQTT serves as the internal means by which the Hue Hub implements the HomeKit protocol, then I assume it provides no additional features beyond what is already accessible via homekit_controller.

For example, the homekit_controller integration does not provide a means of controlling groups (whereas the Philips Hue integration represents a group as a light). Are groups controllable via MQTT?

8

Short answer: it provides not much more than the Homekit controller method. In fact, even less, since a.t.m. it provides only one event topic where you would have to filter out the events you want and create sensors yourself. For now I only ‘consume’ events of the motion sensors that I use complementary to the default Hue integration, since it allows for quicker responding motion sensors. The MQTT topic does provide for group information, but I don’t know if it is even possible to publish messages to the hub and have the hub react to it. So I don’t think it is a worthy replacement for the existing integrations.

And you are right: I can accomplish the same by using the default Hue integration and Homekit controller integration (and then disable all the entities except for the motion sensors). But I like hacking :wink:

Update: I have added the Hue bridge via Homekit controller and only enabled the motion sensors, just to compare them side by side. The MQTT method is faster than the Homekit version of the same motion sensor, not by a lot, but still noticeable.

9

Hello,

I’ve also planned to use MQTT instead of HTTP to manage my hue devices.

I’m root on the bridge and on my side, I added a configuration file in /etc/mosquitto/inc to publish cmd/clip/event/publish messages to my main broker.

I’m using Node-RED to do my automations. Currently I’m parsing MQTT in messages from Hue to publish messages in topics like:

  • home/<room>/light/<light-name>/state: json value with state, brightness, color, etc
  • home/<room>/light/<light-name>/availability: online or offline built with zigbee_connectivity
  • home/<room>/light/<light-name>/command: published by home-assistant

I push configurations in homeassistant/light/<room>_<name>/config with state_topic, command_topic and availability_topic.

Currently, Home Assistant is able to get (through MQTT) all my device states (lights, dimmer, motions). But, I think that my solution needs a lot of “code” already written in other lib / modules…

Today, my goal is to use an existing Node-RED plugin which currently use HTTP polling but with a “manual” trigger when a MQTT message is received. With this solution, I just have to handle command topic (where home assistant will publish its commands) to control my devices.

I’m open if someone wants to work on it with me. Do not hesitate if you had already did something like that or if you have a better idea.

Thomas.

10

Hi,

@rclsilver Would it be possible to share your work? I’m having trouble getting those MQTT-messages out of the bridge :confused:
What’s in your conf file you added to mosquitto/inc? Did you add something to rc.local as wel? (i.e. iptables…)
And if possible the code used to parse it with Node-RED.

Cheers!

11

I see my former message was not replied to @rclsilver , to make sure you get notified I’ll post it again.
Are you willing to share your mosquitto.conf here? I could use some help to get the messages to my main broker.
Please let me know, huge thanks! :slight_smile:

12

Connecting MQTT brokers is relatively easy and well explained here: Mosquitto MQTT Bridge-Usage and Configuration (steves-internet-guide.com)

My solution was to open the firewall on the Hue bridge (port 1883) and configure my main MQTT broker to collect the messages from the Hue MQTT broker.

connection hue_mqtt
address 192.168.xxx.yyy:1883
topic cmd/clip/event/publish in

You probably can use the same method the other way around and push the massages using:

connection main_mqtt
address 192.168.xxx.yyy:1883
topic cmd/clip/event/publish out
13

hmm, that’s exactly what I did after reading that exact same guide.
I opened port 1883 with:

iptables -A OUTPUT -p tcp --dport $port -d $ip -m state --state NEW,ESTABLISHED -j ACCEPT

Did you use a different iptables rule?

14

I just edited /etc/rc.local on the Hue bridge and added the following line to open port 1883 to access MQTT on the hub:

iptables -I INPUT -p tcp --dport 1883 -j ACCEPT

Note… to get things working with the new version of MQTT on the Hue bridge I needed to re-configure the listening interfaces. Maybe in your situation where you try to connect to a remote MQTT broker from the local-only MQTT broker on the Hue hub you may need to change some other settings too.
Migrating from 1.x to 2.0 | Eclipse Mosquitto

15

How about that, finally I am getting messages from the hue :slight_smile:
Thanks a bunch for your assistance, you just made me (a Domoticz-user) very happy :wink:

16

No problem, glad it works! :slightly_smiling_face:

And feel welcome; there are for sure a lot of (ex) Domoticz-users on this forum and we’re all in the home automation endeavor together. Please do stick around on this forum and you might find out some new things you may like :wink:

17

Hi,

Sorry, I didn’t seen your messages.

I created a configuration file in /etc/mosquitto/inc/ on the bridge, with the following content:

connection domotique
address <my-mqtt-server>:1883

topic cmd/clip/event/publish out 0 "" hue/bridge/

I’ve also created a Node-RED plugin. Development version is available here: GitHub - rclsilver/node-red-contrib-hue-mqtt at dev. I have to finish it with:

  • documentation / examples
  • translations

I’ve created it from GitHub - Foddy/node-red-contrib-huemagic: Philips Hue node to control bridges, lights, groups, motion sensors, temperature sensors and Lux sensors using Node-RED.

node-red-hue
node-red-hue1134×817 53.2 KB

I wrote a local library to use in functions in node-red: GitHub - rclsilver/node-red-utils. You have to git pull this repository in /data in Node-RED container and add following settings in settings.js node-red configuration file:

    functionGlobalContext: {
        utils: require('./utils'),
        // os:require('os'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})
    },

You can find bellow my implementation in Node-RED:

[
  {
    "id": "6912c440.c1421c",
    "type": "mqtt in",
    "z": "fd782201.29744",
    "name": "Raw hue events",
    "topic": "hue/bridge/cmd/clip/event/publish",
    "qos": "2",
    "datatype": "json",
    "broker": "971e1087.5d6b5",
    "nl": false,
    "rap": true,
    "rh": 0,
    "x": 140,
    "y": 120,
    "wires": [["53f1b951.c88a28"]]
  },
  {
    "id": "53f1b951.c88a28",
    "type": "hue-mqtt-in",
    "z": "fd782201.29744",
    "name": "MQTT updates",
    "bridge": "4a6301b0.fb715",
    "x": 340,
    "y": 120,
    "wires": [[]]
  },
  {
    "id": "33a1c624.b4176a",
    "type": "hue-cache",
    "z": "fd782201.29744",
    "name": "Hue cache",
    "bridge": "4a6301b0.fb715",
    "x": 310,
    "y": 220,
    "wires": [["7ae4286.6d705d8"], []]
  },
  {
    "id": "7ae4286.6d705d8",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Discovery",
    "func": "const topicPrefix = 'homeassistant';\nconst { hue } = global.get('utils');\n\nif (msg.type === 'temperature') {\n    return hue.temperatureToHomeassistant(msg, topicPrefix);\n} else if (msg.type === 'brightness') {\n    return hue.brightnessToHomeassistant(msg, topicPrefix);\n} else if (msg.type === 'motion') {\n    node.send(hue.motionBatteryToHomeassistant(msg, topicPrefix));\n    node.send(hue.motionEnabledToHomeassistant(msg, topicPrefix));\n    node.send(hue.motionToHomeassistant(msg, topicPrefix));\n} else if (msg.type === 'light') {\n    return hue.lightToHomeassistant(msg, topicPrefix);\n}",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 480,
    "y": 220,
    "wires": [["341f0151.e0b2fe"]]
  },
  {
    "id": "341f0151.e0b2fe",
    "type": "mqtt out",
    "z": "fd782201.29744",
    "name": "",
    "topic": "",
    "qos": "2",
    "retain": "true",
    "respTopic": "",
    "contentType": "",
    "userProps": "",
    "correl": "",
    "expiry": "",
    "broker": "971e1087.5d6b5",
    "x": 630,
    "y": 220,
    "wires": []
  },
  {
    "id": "af5c245c.a2e728",
    "type": "inject",
    "z": "fd782201.29744",
    "name": "Manual refresh",
    "props": [],
    "repeat": "",
    "crontab": "",
    "once": false,
    "onceDelay": 0.1,
    "topic": "",
    "x": 130,
    "y": 220,
    "wires": [["33a1c624.b4176a"]]
  },
  {
    "id": "3fd2690b.2ae9b6",
    "type": "hue-temperature",
    "z": "fd782201.29744",
    "name": "Entrée",
    "bridge": "4a6301b0.fb715",
    "sensor": "4",
    "x": 130,
    "y": 300,
    "wires": [["d36b0833.8c9a28"]]
  },
  {
    "id": "2d53b396.109aec",
    "type": "hue-temperature",
    "z": "fd782201.29744",
    "name": "Couloir",
    "bridge": "4a6301b0.fb715",
    "sensor": "8",
    "x": 130,
    "y": 380,
    "wires": [["1013304c.64039"]]
  },
  {
    "id": "d36b0833.8c9a28",
    "type": "change",
    "z": "fd782201.29744",
    "name": "Set label",
    "rules": [
      { "t": "set", "p": "label", "pt": "msg", "to": "Entrée", "tot": "str" }
    ],
    "action": "",
    "property": "",
    "from": "",
    "to": "",
    "reg": false,
    "x": 260,
    "y": 300,
    "wires": [["a9dd1b6b.fc22c8"]]
  },
  {
    "id": "1013304c.64039",
    "type": "change",
    "z": "fd782201.29744",
    "name": "Set label",
    "rules": [
      { "t": "set", "p": "label", "pt": "msg", "to": "Couloir", "tot": "str" }
    ],
    "action": "",
    "property": "",
    "from": "",
    "to": "",
    "reg": false,
    "x": 260,
    "y": 380,
    "wires": [["a9dd1b6b.fc22c8"]]
  },
  {
    "id": "a9dd1b6b.fc22c8",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Build message",
    "func": "const { datetime, hue } = global.get('utils');\nconst { area } = hue.parseName(msg.info.name);\nconst timestamp = datetime.toTimestamp(msg.payload.updated);\n\nreturn {\n    topic: `home/${area}/sensor/temperature`,\n    payload: {\n        value: msg.payload.celsius,\n        battery: msg.info.battery,\n        timestamp: timestamp,\n        label: msg.label,\n    }\n};",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 460,
    "y": 340,
    "wires": [["37443c5c.b6e3b4"]]
  },
  {
    "id": "37443c5c.b6e3b4",
    "type": "mqtt out",
    "z": "fd782201.29744",
    "name": "",
    "topic": "",
    "qos": "1",
    "retain": "true",
    "respTopic": "",
    "contentType": "",
    "userProps": "",
    "correl": "",
    "expiry": "",
    "broker": "971e1087.5d6b5",
    "x": 630,
    "y": 420,
    "wires": []
  },
  {
    "id": "96ada3bb.813fa",
    "type": "hue-brightness",
    "z": "fd782201.29744",
    "name": "Entrée",
    "bridge": "4a6301b0.fb715",
    "sensor": "3",
    "x": 130,
    "y": 460,
    "wires": [["d0f56618.ad0348"]]
  },
  {
    "id": "88728839.f2a828",
    "type": "hue-brightness",
    "z": "fd782201.29744",
    "name": "Couloir",
    "bridge": "4a6301b0.fb715",
    "sensor": "7",
    "x": 130,
    "y": 540,
    "wires": [["3315e3b2.d48f7c"]]
  },
  {
    "id": "d0f56618.ad0348",
    "type": "change",
    "z": "fd782201.29744",
    "name": "Set label",
    "rules": [
      { "t": "set", "p": "label", "pt": "msg", "to": "Entrée", "tot": "str" }
    ],
    "action": "",
    "property": "",
    "from": "",
    "to": "",
    "reg": false,
    "x": 260,
    "y": 460,
    "wires": [["5fa4f379.28695c"]]
  },
  {
    "id": "3315e3b2.d48f7c",
    "type": "change",
    "z": "fd782201.29744",
    "name": "Set label",
    "rules": [
      { "t": "set", "p": "label", "pt": "msg", "to": "Couloir", "tot": "str" }
    ],
    "action": "",
    "property": "",
    "from": "",
    "to": "",
    "reg": false,
    "x": 260,
    "y": 540,
    "wires": [["5fa4f379.28695c"]]
  },
  {
    "id": "5fa4f379.28695c",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Build message",
    "func": "const { datetime, hue } = global.get('utils');\nconst { area } = hue.parseName(msg.info.name);\nconst timestamp = datetime.toTimestamp(msg.payload.updated);\n\nreturn {\n    topic: `home/${area}/sensor/brightness`,\n    payload: {\n        value: msg.payload.lux,\n        battery: msg.info.battery,\n        timestamp: timestamp,\n        label: msg.label,\n    }\n};",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 460,
    "y": 500,
    "wires": [["37443c5c.b6e3b4"]]
  },
  {
    "id": "24267098.776ca",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Build message",
    "func": "const { datetime, hue } = global.get('utils');\nconst { area, name } = hue.parseName(msg.info.name);\nconst timestamp = datetime.toTimestamp(msg.payload.updated);\n\nreturn {\n    topic: `home/${area}/button/${name}`,\n    payload: {\n        button: msg.payload.name,\n        action: msg.payload.action,\n        battery: msg.info.battery,\n        timestamp,\n    },\n};",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 460,
    "y": 620,
    "wires": [["4d8c7418.f4677c"]]
  },
  {
    "id": "b04994ee.564598",
    "type": "hue-switch",
    "z": "fd782201.29744",
    "name": "Switches",
    "bridge": "4a6301b0.fb715",
    "sensor": "",
    "x": 140,
    "y": 620,
    "wires": [["24267098.776ca"]]
  },
  {
    "id": "4d8c7418.f4677c",
    "type": "mqtt out",
    "z": "fd782201.29744",
    "name": "",
    "topic": "",
    "qos": "1",
    "retain": "false",
    "respTopic": "",
    "contentType": "",
    "userProps": "",
    "correl": "",
    "expiry": "",
    "broker": "971e1087.5d6b5",
    "x": 630,
    "y": 620,
    "wires": []
  },
  {
    "id": "5795579c.357888",
    "type": "mqtt out",
    "z": "fd782201.29744",
    "name": "",
    "topic": "",
    "qos": "1",
    "retain": "true",
    "respTopic": "",
    "contentType": "",
    "userProps": "",
    "correl": "",
    "expiry": "",
    "broker": "971e1087.5d6b5",
    "x": 1090,
    "y": 780,
    "wires": []
  },
  {
    "id": "75dee6be.de3d98",
    "type": "hue-motion",
    "z": "fd782201.29744",
    "name": "Motions",
    "bridge": "4a6301b0.fb715",
    "sensor": "",
    "x": 720,
    "y": 740,
    "wires": [["b395f324.09108"]]
  },
  {
    "id": "b395f324.09108",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Build message",
    "func": "const { datetime, hue } = global.get('utils');\nconst { area, name } = hue.parseName(msg.info.name);\nconst timestamp = datetime.toTimestamp(msg.payload.updated);\n\nreturn {\n    topic: `home/${area}/sensor/${name}`,\n    payload: {\n        value: msg.payload.motion,\n        enabled: msg.payload.active,\n        battery: msg.info.battery,\n        timestamp: timestamp,\n    }\n};",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 920,
    "y": 740,
    "wires": [["5795579c.357888"]]
  },
  {
    "id": "97dbf0a3.89307",
    "type": "hue-light",
    "z": "fd782201.29744",
    "name": "Lights",
    "bridge": "4a6301b0.fb715",
    "light": "",
    "x": 730,
    "y": 820,
    "wires": [["2795da08.900126"]]
  },
  {
    "id": "2795da08.900126",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Build message",
    "func": "const { hue } = global.get('utils');\nconst { area, name } = hue.parseName(msg.info.name);\n\n// Availability topic\nif (msg.payload.reachable !== undefined) {\n    node.send({\n        topic: `home/${area}/light/${name}/availability`,\n        payload: msg.payload.reachable ? 'online' : 'offline',\n    });\n}\n\n// State topic\nlet message = {\n    topic: `home/${area}/light/${name}/state`,\n    payload: {\n        state: msg.payload.on ? 'ON' : 'OFF',\n    },\n};\n\nif (msg.payload.brightness !== undefined) {\n    message.payload.brightness = msg.payload.brightness;\n}\n\nif (msg.payload.xy !== undefined) {\n    message.payload.color = msg.payload.xy;\n}\n\nif (msg.payload.colorTemp !== undefined) {\n    message.payload.color_temp = msg.payload.colorTemp;\n}\n\nnode.send(message);",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 920,
    "y": 820,
    "wires": [["5795579c.357888"]]
  },
  {
    "id": "2b17af94.a22cb",
    "type": "mqtt in",
    "z": "fd782201.29744",
    "name": "Motions commands",
    "topic": "home/+/sensor/+/set",
    "qos": "2",
    "datatype": "json",
    "broker": "971e1087.5d6b5",
    "nl": false,
    "rap": true,
    "rh": 0,
    "x": 150,
    "y": 740,
    "wires": [["c3bac75a.7d26b8"]]
  },
  {
    "id": "458d2f58.b715",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Build message",
    "func": "const { mqtt } = global.get('utils');\nconst { area, name } = mqtt.parseTopic(msg.topic);\n\nreturn {\n    payload: {\n        sensorName: `${area}_${name}`,\n        enabled: msg.payload,\n    }\n};",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 550,
    "y": 740,
    "wires": [["75dee6be.de3d98"]]
  },
  {
    "id": "c3bac75a.7d26b8",
    "type": "switch",
    "z": "fd782201.29744",
    "name": "Filter motions",
    "property": "topic",
    "propertyType": "msg",
    "rules": [
      {
        "t": "regex",
        "v": "^home/[^/]+/sensor/motion[^/]*/set$",
        "vt": "str",
        "case": false
      }
    ],
    "checkall": "true",
    "repair": false,
    "outputs": 1,
    "x": 360,
    "y": 740,
    "wires": [["458d2f58.b715"]]
  },
  {
    "id": "5f979bb3.e0c984",
    "type": "mqtt in",
    "z": "fd782201.29744",
    "name": "Lights commands",
    "topic": "home/+/light/+/set",
    "qos": "2",
    "datatype": "json",
    "broker": "971e1087.5d6b5",
    "nl": false,
    "rap": true,
    "rh": 0,
    "x": 140,
    "y": 820,
    "wires": [["756a1bd3.18a854"]]
  },
  {
    "id": "756a1bd3.18a854",
    "type": "function",
    "z": "fd782201.29744",
    "name": "Build message",
    "func": "const { mqtt } = global.get('utils');\nconst { area, name } = mqtt.parseTopic(msg.topic);\nconst payload = {\n    lightName: `${area}_${name}`,\n};\n\nif (msg.payload.state !== undefined) {\n    payload.on = msg.payload.state === 'ON';\n}\n\nif (msg.payload.brightness !== undefined) {\n    payload.brightness = msg.payload.brightness;\n}\n\nif (msg.payload.color !== undefined) {\n    payload.xy = msg.payload.color;\n}\n\nif (msg.payload.color_temp !== undefined) {\n    payload.colorTemp = msg.payload.color_temp;\n}\n\nreturn {\n    payload,\n};",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "libs": [],
    "x": 360,
    "y": 820,
    "wires": [["97dbf0a3.89307"]]
  },
  {
    "id": "971e1087.5d6b5",
    "type": "mqtt-broker",
    "name": "Local MQTT",
    "broker": "mosquitto.mosquitto.svc.cluster.local",
    "port": "1883",
    "clientid": "node-red",
    "usetls": false,
    "protocolVersion": "4",
    "keepalive": "60",
    "cleansession": true,
    "birthTopic": "",
    "birthQos": "0",
    "birthPayload": "",
    "birthMsg": {},
    "closeTopic": "",
    "closeQos": "0",
    "closePayload": "",
    "closeMsg": {},
    "willTopic": "",
    "willQos": "0",
    "willPayload": "",
    "willMsg": {},
    "sessionExpiry": ""
  },
  {
    "id": "4a6301b0.fb715",
    "type": "hue-bridge",
    "name": "Hue Bridge",
    "bridge": "192.168.1.42",
    "key": "secret",
    "interval": "60000"
  }
]
1 Like
18

The other topics in ‘dt/clip/*’ do seem to contain the state information you want, but I have not yet managed to decypher the binary content.

The encoding for dt/clip/+/+ is just zlib-compressed JSON, but with no zlib header or trailer. You can use Python zlib.decompress(payload, wsize=-15) to decompress it.

When you initially connect, you get a series of dt/clip messages with retained=1, which seem to represent the known state of all Hue devices. New dt/clip messages arrive whenever the state is updated or when a device is added or removed. Multiple data messages may be followed by a single cmd/clip/event/publish message.

Here’s an example program that outputs those messages in real time. You might need to adjust the connection parameters.

#!/usr/bin/env python3
# dlitz 2021, public domain
import base64
import json
import paho.mqtt.client as mqtt     # pip install paho-mqtt
import sys
import zlib

def main():
    client = mqtt.Client(protocol=mqtt.MQTTv5, transport="tcp")
    client.enable_logger()
    client.on_connect = on_connect
    client.message_callback_add("dt/clip/+/+", handle_data_message)
    client.message_callback_add("cmd/clip/event/publish", handle_event_message)
    #client.tls_set()
    #client.username_pw_set("username", "password")
    client.connect("localhost", 1883)
    client.loop_forever()

def on_connect(client, userdata, flags, rc, properties):
    client.subscribe("dt/clip/+/+")
    client.subscribe("cmd/clip/event/publish")

def handle_data_message(client, userdata, msg):
    out = {
        'timestamp': msg.timestamp,
        'state': msg.state,
        'dup': msg.dup,
        'mid': msg.mid,
        'topic': msg.topic,
        'payload_b64': base64.b64encode(msg.payload).decode('UTF-8'),
        'payload_decoded': json.loads(zlib.decompress(msg.payload, wbits=-15)),
        'qos': msg.qos,
        'retain': msg.retain,
        'info': {
            'mid': msg.info.mid,
            'rc': msg.info.rc,
        },
        'properties': msg.properties.json(),
    }

    #print(json.dumps(out))
    print(json.dumps(out, indent=2))
    sys.stdout.flush()

def handle_event_message(client, userdata, msg):
    out = {
        'timestamp': msg.timestamp,
        'state': msg.state,
        'dup': msg.dup,
        'mid': msg.mid,
        'topic': msg.topic,
        'payload_b64': base64.b64encode(msg.payload).decode('UTF-8'),
        'payload_decoded': json.loads(msg.payload),
        'qos': msg.qos,
        'retain': msg.retain,
        'info': {
            'mid': msg.info.mid,
            'rc': msg.info.rc,
        },
        'properties': msg.properties.json(),
    }

    #print(json.dumps(out))
    print(json.dumps(out, indent=2))
    sys.stdout.flush()

if __name__ == '__main__':
    main()
1 Like
19

The Hue hub also has a couple really nice DIY/remote-admin features: it supports pointing syslog at a custom host, and SSH pubkey authentication, and you can save these parameters in the boot variables, so they are preserved across firmware updates.

From the u-boot prompt, I did something like:

# Remove root user password (disables remote password auth)
setenv security
# Set the contents of root's authorized_keys file
setenv authorized_keys 'ssh-rsa AAAA[...] user@host'
# syslog destination
setenv logdest '192.0.2.1:514'
# save environment vars
saveenv

Once I had SSH with pubkey auth enabled, I set up a service on my main Linux mosquitto server to maintain a persistent tunnel to the Hue’s MQTT service:

# /etc/systemd/system/sshfwd-hue0-mqtt.socket
[Unit]
Description=SSH port-forward to mosquitto running on Philips Hue bridge
#Documentation=

[Socket]
#ListenStream=/run/sshfwd/hue0-mqtt/sock
#SocketMode=0660
#SocketGroup=mosquitto
# unfortunately mosquitto currently doesn't support connecting via a unix socket, so use some arbitrary port on localhost
ListenStream=127.0.0.1:10002
Accept=yes

[Install]
WantedBy=sockets.target

# /etc/systemd/system/[email protected]
[Unit]
Description=SSH port-forward to mosquitto running on Philips Hue bridge
#Documentation=

[Service]
Type=oneshot
User=sshfwd
ExecStart=/usr/bin/ssh -F /etc/sshfwd/config -qW localhost:1883 hue0
StandardInput=socket

# /etc/sshfwd/config
IdentityFile /etc/sshfwd/id_rsa
IdentityFile /etc/sshfwd/id_ed25519
UserKnownHostsFile /etc/sshfwd/known_hosts
HashKnownHosts no
CheckHostIP no
BatchMode yes

Host hue0 hue0.lan hue0.example.net
    HostKeyAlias hue0.example.net
    User root
    #HostName hue0.lan
    HostName 192.0.2.211
    KexAlgorithms +diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

# /etc/mosquitto/conf.d/hue-bridge.conf
# bridge for philips-hue
connection hue0
address 127.0.0.1:10002
topic # in 2 hue0/ ""
1 Like
20

This is what I came here for, how to decode the Hue’s Zigbee signals! For a couple years I’ve been using a rules file in Openhab (yes I do realize this is a Home Assistant forum). Openhab, and probably HA too, handle the dimmers by periodically polling the Hue hub API to check for changes to their most recent status. Once I learned I could root the Hue, I wrote a bash script that monitors the MQTT messages published by “clipd” (that script is available if anyone wants it), which reduced the latency quite a bit. But I really wanted the raw messages from the sensors to eliminate “clipd” as yet another middleman… This discussion filled in the missing link to do that (namely the Zlib encoding).

So now I’ve written a variant of one of the above scripts which can run on the Hue itself (at least with the latest firmware) using the included micropython, and using curl to reach out to Openhab (or whatever else accepts HTTP requests). It doesn’t require any libraries via PIP. I think I had to install curl via opkg, but the other necessary libraries are all on the Hue already. I’ve set this up to control a non-Hue light bulb with a Hue dimmer. It launches at startup on the Hue by adding micropython /root/hue_zigbee_monitor.py & to /etc/rc.local.

I’m a long-time coder but have never done much with Python so forgive me if it’s pretty crude. Right now the handler is a giant if statement… Anyway, it works. At some point I’ll figure out a nicer way to map the sensor IDs.

#!/usr/bin/env python3
# hue_zigbee_monitor.py
import json
import uasyncio as asyncio
from mqtt import Mqtt
import zlib
import sys
import os

# this code adapted from /usr/lib/micropython/mqtt/example.py:

async def text_listener(topic, mqtt):
    async for msg in mqtt.subscribe(topic):
        handle_event_message(topic, msg)                          
                  
async def data_listener(topic, mqtt):
    async for msg in mqtt.subscribe(topic):
        handle_data_message(topic, msg)                          
                  
async def main():
    async with Mqtt("pymosquitto") as m:
        tasks = []                                     
        tasks.append(asyncio.create_task(text_listener("cmd/clip/event/publish",m))) 
        tasks.append(asyncio.create_task(data_listener("dt/clip/+/+",m)))
        await asyncio.gather(*tasks)

def handle_data_message(topic, msg):
      # print(topic, msg)
      item = json.loads(zlib.decompress(msg["payload"], -15).decode())

      # your handler stuff goes here -- one would probably want it to be a bit more sophisticated, but this is just a POC
      if "button" in item:
        cmd = ""
        request = item["id_v1"] + ":" + item["button"]["last_event"] + ":" + str(item["metadata"]["control_id"])
        print(request) # use this to easily map as-yet unmapped buttons
        if request == "/sensors/10:short_release:4":
          cmd = "/usr/bin/curl http://openhabserver:8080/rest/items/MyLight -H 'Content-Type: text/plain' --data-raw OFF"
        elif request == "/sensors/10:short_release:1":
          cmd = "/usr/bin/curl http://openhabserver:8080/rest/items/MyLight -H 'Content-Type: text/plain' --data-raw ON"
        if cmd != "":
          # print(cmd)
          os.system(cmd)
      sys.stdout.flush()


def handle_event_message(topic, msg):
#    print(msg) # whatever you want to do with this, if anything
    sys.stdout.flush()

asyncio.run(main())
asyncio.get_event_loop().run_forever()
3 Likes