Make sure to check out monitor as well, as it is more advanced than presence
Introduction
Here’s the problem I faced: all presence detection options I tried didn’t work reliably for my family.
The various GPS-based options (e.g., OwnTracks, iOS App, HomeKit, etc.) periodically would stop working if an app on our phones crashed. Plus the minimum radius for ‘home’ or ‘away’ was often way larger than our house. I would register as ‘home’ when I was on a dog walk a few blocks away. Not great.
Similarly, the Wi-Fi options (e.g., nmap, ubiquiti, etc.) were neither fast nor reliable because we have iOS devices that disconnect to save battery. Plus, our ubiquiti APs do not report a disconnect for quite some time - in some cases, fifteen minutes would go by. Yes, I know we can configure the timeouts, but that was not workable either. Still too slow. Also, the phones would disconnect when charging at night. Bad.
Also, Bluetooth LE options (e.g., Happy Bubbles, built-in bluetooth presence sensor, Radius Networks Script, room-assisant, etc.) either required BTLE tags ($$$ + regular batter change), were limited to detection within bluetooth’s range, or became expensive to implement (five happy bubbles + btle tags = ~$180). Don’t get me wrong, I love the happy bubbles stuff and that project is awesome. But for us, it did not work reliably enough to justify the cost. Plus, the server spammed the mqtt broker and slowed Home Assistant down. Not good.
In sum, these were the problems I had when implementing the current presence solutions:
- Too slow to respond (wifi)
- Too unreliable (GPS, wifi, btle)
- Too expensive (happy bubbles + btle tags)
- Too big of a radius (GPS, wifi)
- Too small of a radius (bluetooth, happy bubbles)
- False positives and false negatives. Switching from home to away when we are at the house is not ok. Similarly, a false positive when we are not home is a huge problem. (GPS)
I wanted a cheaper solution that would meet these specs:
- Fast to respond. Arrival should be recognized in no more than 5 seconds when I’m on the curb or in the garage.
- Reliable without adding anything to my wife’s phone or keychain or purses. Should work without her caring or knowing that it’s working.
- Cheap. Less than $50 would be awesome.
- Tight radius around the house. When I’m at at the curb walking the dog, I should be marked as ‘away’.
- Let me know when guests arrive without slowing down the search for my ‘owner’ devices
- Respond to BTLE devices too, as a backup
Here’s my solution that I’ve been tinkering with for the past six months or so. Still working on it and tuning, but it works great so far. I use Raspberry Pi Zero Ws positioned around the house, each running a shell script that I wrote and that is available on github here. I have the four devices positioned throughout the house, each connected to a cheap power brick (anker or monoprice) via a monoprice usb cable. Each configuration is about $16 ($10 pi, $6 usb micro power supply). For four of these (which is way overkill for my house, probably), the total price is $64. That’s much more palatable to me than the $180 I spent on the five happy bubbles that are now sitting in my drawer, and it still is able to track BTLE devices! (as of version 0.4)
This reports when my wife and I are home or away, and reports when a registered ‘guest’ device arrives home. I may implement guest device ‘away’ in the future, but I decided that the feature was not quite necessary for my use case.
Summary
The script generates a JSON-formatted MQTT message that is reported to a broker whenever a specified bluetooth device responds to a name query. If the device responds, the JSON message includes the name of the device and a confidence of 100.
After a delay, another name query is sent and, if the device does not respond, a verification-of-absence loop begins that queries for the device (on a shorter interval) a set number of times. Each time the device does not respond, the confidence is reduced, eventually to 0. This value is averaged with the other sensors to give an aggregate confidence that can be used for automations. To eliminate false negatives, I also use a condition for “away” state that requires recent main floor door access. I’ll talk about that a bit later.
Also, a configuration file accessed by the script defines ‘owner devices’ and another defines ‘guest devices.’ The script only scans for guest devices when not scanning for owner devices; detection of owner devices is prioritized over detection of guest devices.
Topics are formatted like this:
location/owner/pi_zero_location/00:00:00:00:00:00
location/guest/pi_zero_location/00:00:00:00:00:00
Messages are JSON formatted and contain name and confidence fields, in addition to other fields that might be interesting to end-users:
{ name : "Andrew's iPhone", confidence : 100, timestamp : "[javascript formatted timestamp]", scan_duration_ms : "0000"}
{ name : "", confidence : 0}
The timestamp is formatted according to what javascript expects, namely:
+%a %b %d %Y %H:%M:%S GMT%z (%Z)
When the script starts, the configuration file is analyzed and estimations of time for each event transition.
As an example, with two owner devices, with beacon detection off and default settings:
presence 0.x.xx - Started. Performance predictions based on current settings:
> Est. to verify all (2) owners as 'away' from all 'home': 60 seconds to 210 seconds.
> Est. to verify one owner is 'away': 30 to 121 seconds.
> Est. to recognize one owner is 'home': 0.15 seconds to 12 seconds.
Tweaking your settings will adjust these numbers. The script is set up to detect devices as quickly as possible, and taking time to verify that a device is actually gone.
Example Use with Home Assistant
The presence script can be used as an input to a number of mqtt sensors in Home Assistant.. Output from these sensors can be averaged to give a highly-accurate numerical occupancy confidence.
In order to detect presence in a home that has three floors and a garage, we might inclue one Raspberry Pi per floor. For average houses, a single well-placed sensor can probably work, but for more reliability at the edges of the house, more sensors are better.
- platform: mqtt
state_topic: 'location/owner/first floor/00:00:00:00:00:00'
value_template: '{{ value_json.confidence }}'
unit_of_measurement: '%'
name: 'Andrew First Floor'
- platform: mqtt
state_topic: 'location/owner/second floor/00:00:00:00:00:00'
value_template: '{{ value_json.confidence }}'
unit_of_measurement: '%'
name: 'Andrew Second Floor'
- platform: mqtt
state_topic: 'location/owner/third floor/00:00:00:00:00:00'
value_template: '{{ value_json.confidence }}'
unit_of_measurement: '%'
name: 'Andrew Third Floor'
- platform: mqtt
state_topic: 'location/owner/garage/00:00:00:00:00:00'
value_template: '{{ value_json.confidence }}'
unit_of_measurement: '%'
name: 'Andrew Garage'
These sensors can be combined/averaged using a min_max:
- platform: min_max
name: "Andrew Home Occupancy Confidence"
type: mean
round_digits: 0
entity_ids:
- sensor.andrew_garage
- sensor.andrew_third_floor
- sensor.andrew_second_floor
- sensor.andrew_first_floor
So, as a result of this combination, we use the entity sensor.andrew_home_occupancy_confidence in automations to control the state of an input_boolean that represents a very high confidence of a user being home or not.
As an example:
- alias: Andrew Occupancy
hide_entity: true
trigger:
- platform: numeric_state
entity_id: sensor.andrew_home_occupancy_confidence
above: 10
action:
- service: home-assistant.turn_on
data:
entity_id: input_boolean.andrew_occupancy
Another automation is used to set the away mode, this time triggered based on a value being above 10. I have also added a condition that a first floor door has been opened in the last 10 minutes. This prevents false exit events.
That’s it! The state of input_boolean.andrew_occupancy very reliably holds my occupancy state. I have the same setup for my wife’s iPhone as well. We’ve been using this setup for roughly six months, and it’s the solution I’m going to continue to use. It works excellently, despite how complicated it seems.
Truth be told, the automation above is not my primary automation to turn on the input_booleans. I use node-red and some smoothing and statistics nodes.
Example Use with Node-Red
Recently, I transitioned many of my automations to node-red. I used the hass.io add-on as my node-red server.
This is a portion of my automation used to set home and away:
Looks super complex and cumbersome, I know. To save you a bit of boring detail, I’ll go through the nodes from left to right:
-
Leftmost nodes subscribe to the mqtt topics for individual Raspberry Pi Zero Ws. The node on the bottom takes the average of the four (from the max_min sensor above). By combining all four raspberry pi’s with their average, I’ve effectively low-pass filtered the entire bunch right off the bat.
-
Next, each of the mqtt messages received are passed to a JSON formatter. Not strictly necessary, but a bit more convenient. After formatting as a JSON message, the payloads are passed into a function that recasts the string ‘confidence’ into a number. This is important for the next step. The function also adds a topic to the message. The topic is a plain-english name for each sensor. The function looks like this:
return {topic: 'Garage', payload: parseInt(msg.payload.confidence) || 0};
-
I recast the string to an integer so that these values can be smoothed with a smooth node. Right now, I average the value of the last five values received by this node.
-
After smoothing, the smoothed value is sent to an RBE node that will only pass the value if combined and twice-filtered/averaged confidence has changed.
-
After change-filtering, I pass the value to a function that asks whether the confidence is greater than a threshold. This function outputs true or false only. I’m still playing with the threshold, but right now, 5 seems to be working well since I have four sensors. As an example, if at least one sensor detects that I am home, it will output 100. Averaging this value against three other zero values returns 25. Averaging 25 (twice) with three zeros results in a value of 10 (25+25+0+0+0)/5. In other words, I mark as “away” only after most sensors don’t see me at all, and the last sensor to recognize my phone is only half-sure that I’m there.
-
After reducing to a boolean confidence, I pass the occupancy value to a rate limiter that passes only 1 message per every ten seconds. This prevents rapid changes from home to away when I’m right on the edge of the ‘occupied’ zone. Probably not strictly necessary, but since my node-red installation is running on the same box as home assistant, anything I can do to prevent node-red from working on parallel tasks improves performance.
-
After rate limiting, I pass the boolean confidence to a switch that bifurcates the path. If true, I set my occupancy input boolean to true.
-
On the other hand, fi the input boolean is false, I set the input boolean to false only if an exterior door on the main floor (including the garage lift) has been opened or closed in the last 10 minutes. This is a final check against false exit events.
Now, astute readers will note that I have two alternate paths shown. The top alternate path allows me to bypass the smoothing if one of the sensors switches from 0 to 100 (meaning that I have arrived home). This function looks like this:
var confidence = parseInt(msg.payload);
if (confidence >= 5 ){
return {payload: true};
}else{
return null
}
This allows an ‘arrival’ event to be recognized slightly sooner. This alternate path is not strictly necessary.
Lastly, I also have an alternate path that notifies me if any of the raspberry pis go offline. ‘Offline’ is defined as not receiving any message for at least 10 minutes.
The entire flow is available by copying the following and importing to node-red:
[{"id":"dcd7a3e9.622e6","type":"inject","z":"247a6012.ad0588","name":"30s ","topic":"","payload":"","payloadType":"date","repeat":"30","crontab":"","once":false,"onceDelay":"","x":130,"y":440,"wires":[["ae864ca4.e676d8"]]},{"id":"723329a9.df4fb8","type":"function","z":"247a6012.ad0588","name":"> 5","func":"var confidence = parseInt(msg.payload);\nreturn {payload: confidence > 5 };","outputs":1,"noerr":0,"x":1250,"y":330,"wires":[["33baaee6.5e5fca"]]},{"id":"a1928cf9.98ba08","type":"rbe","z":"247a6012.ad0588","name":"","func":"rbe","gap":"","start":"","inout":"out","property":"payload","x":1080,"y":330,"wires":[["723329a9.df4fb8","1a73a15d.55ade7"]]},{"id":"3b25dc0f.50915c","type":"api-call-service","z":"247a6012.ad0588","name":"Set Home","server":"58ba2a40.903c6c","service_domain":"homeassistant","service":"turn_on","data":"{\"entity_id\" : \"input_boolean.andrew_occupancy\"}","mergecontext":"","x":1770,"y":280,"wires":[[]]},{"id":"d68fd19.42fabb","type":"comment","z":"247a6012.ad0588","name":"ANDREW IS HOME OR AWAY","info":"","x":170,"y":160,"wires":[]},{"id":"d8da0e0c.544c78","type":"switch","z":"247a6012.ad0588","name":"Switch","property":"payload","propertyType":"msg","rules":[{"t":"true"},{"t":"false"}],"checkall":"true","repair":false,"outputs":2,"x":1590,"y":330,"wires":[["3b25dc0f.50915c"],["43fad02d.31854"]]},{"id":"3e0bf278.fe9c6e","type":"api-call-service","z":"247a6012.ad0588","name":"Set Away","server":"58ba2a40.903c6c","service_domain":"homeassistant","service":"turn_off","data":"{\"entity_id\" : \"input_boolean.andrew_occupancy\"}","mergecontext":"","x":2210,"y":380,"wires":[[]]},{"id":"5ce2a8c4.825378","type":"mqtt in","z":"247a6012.ad0588","name":"First Floor","topic":"location/owner/first floor/34:08:BC:15:24:F7","qos":"2","broker":"6e9ecea1.694bf8","x":120,"y":380,"wires":[["635652e4.54013c"]]},{"id":"635652e4.54013c","type":"json","z":"247a6012.ad0588","name":"Format","property":"payload","action":"","pretty":true,"x":320,"y":380,"wires":[["82b05d1f.66f7b8"]]},{"id":"82b05d1f.66f7b8","type":"function","z":"247a6012.ad0588","name":"Parse","func":"return {topic: 'First Floor', payload: parseInt(msg.payload.confidence) || 0};","outputs":1,"noerr":0,"x":530,"y":380,"wires":[["57cddb63.1cb484","43318e46.1cdbc8","90ce7582.c2eaf"]]},{"id":"ae864ca4.e676d8","type":"api-current-state","z":"247a6012.ad0588","name":"Occupancy","server":"58ba2a40.903c6c","halt_if":"","entity_id":"sensor.andrew_occupancy_confidence","x":310,"y":440,"wires":[["468f7799.a3085"]]},{"id":"468f7799.a3085","type":"function","z":"247a6012.ad0588","name":"Parse Confidence","func":"return {topic: 'timer', payload: parseInt(msg.payload)};","outputs":1,"noerr":0,"x":500,"y":440,"wires":[["57cddb63.1cb484","43318e46.1cdbc8"]]},{"id":"57cddb63.1cb484","type":"smooth","z":"247a6012.ad0588","name":"","property":"payload","action":"mean","count":"5","round":"0","mult":"single","x":920,"y":330,"wires":[["a1928cf9.98ba08","16deef63.23a421"]]},{"id":"16deef63.23a421","type":"debug","z":"247a6012.ad0588","name":"","active":false,"console":"false","complete":"false","x":1080,"y":390,"wires":[]},{"id":"43fad02d.31854","type":"api-current-state","z":"247a6012.ad0588","name":"Recent Door Access","server":"58ba2a40.903c6c","halt_if":"","entity_id":"input_boolean.1fl_recent_door_access","x":1810,"y":380,"wires":[["ed941d94.211df"]]},{"id":"ed941d94.211df","type":"switch","z":"247a6012.ad0588","name":"ON","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"on","vt":"str"},{"t":"eq","v":"off","vt":"str"}],"checkall":"true","outputs":2,"x":1990,"y":380,"wires":[["3e0bf278.fe9c6e"],[]]},{"id":"43318e46.1cdbc8","type":"function","z":"247a6012.ad0588","name":">= 5 || null","func":"var confidence = parseInt(msg.payload);\nif (confidence >= 5 ){\n return {payload: true};\n}else{\n return null\n}","outputs":1,"noerr":0,"x":1060,"y":180,"wires":[["33baaee6.5e5fca"]]},{"id":"90ce7582.c2eaf","type":"interval-length","z":"247a6012.ad0588","format":"mills","bytopic":true,"minimum":"","maximum":"","window":"","timeout":false,"msgTimeout":"10","minimumunit":"msecs","maximumunit":"msecs","windowunit":"msecs","msgTimeoutUnit":"mins","startup":true,"msgField":"timeout","repeatTimeout":true,"name":"10m Timeout","x":1090,"y":680,"wires":[[],["640dbdc7.3ceeb4"]]},{"id":"f8c3a.085ee3c7","type":"api-call-service","z":"247a6012.ad0588","name":"Notify Andrew","server":"4c32fd70.c128cc","service_domain":"notify","service":"ios_andrews_iphone","data":"{}","mergecontext":"","x":1550,"y":680,"wires":[[]]},{"id":"640dbdc7.3ceeb4","type":"function","z":"247a6012.ad0588","name":"Alert! Notification","func":"var payload = msg.payload;\nvar msg = {payload : {data: { title : \"853 Elm: Notice\", message: \"Heartbeat from \" + msg.topic + \" not detected for 10 minutes. The server may be down.\", data : { push : {badge : 0}}}}};\n\nreturn msg;","outputs":1,"noerr":0,"x":1300,"y":680,"wires":[["f8c3a.085ee3c7"]]},{"id":"42067d42.2dbd04","type":"mqtt in","z":"247a6012.ad0588","name":"Second Floor","topic":"location/owner/second floor/34:08:BC:15:24:F7","qos":"2","broker":"6e9ecea1.694bf8","x":130,"y":330,"wires":[["ab3f1893.36425"]]},{"id":"b8497c97.b7384","type":"mqtt in","z":"247a6012.ad0588","name":"Third Floor","topic":"location/owner/third floor/34:08:BC:15:24:F7","qos":"2","broker":"6e9ecea1.694bf8","x":120,"y":280,"wires":[["3e8b4e94.d422a2"]]},{"id":"2d869c7a.84ee6c","type":"mqtt in","z":"247a6012.ad0588","name":"Garage","topic":"location/owner/garage/34:08:BC:15:24:F7","qos":"2","broker":"6e9ecea1.694bf8","x":110,"y":220,"wires":[["bbc4e61c.fd6ac8"]]},{"id":"ab3f1893.36425","type":"json","z":"247a6012.ad0588","name":"Format","property":"payload","action":"","pretty":true,"x":320,"y":330,"wires":[["3bc286a8.6c76aa"]]},{"id":"3bc286a8.6c76aa","type":"function","z":"247a6012.ad0588","name":"Parse","func":"return {topic: 'Second Floor', payload: parseInt(msg.payload.confidence) || 0};","outputs":1,"noerr":0,"x":530,"y":330,"wires":[["90ce7582.c2eaf","57cddb63.1cb484","43318e46.1cdbc8"]]},{"id":"3e8b4e94.d422a2","type":"json","z":"247a6012.ad0588","name":"Format","property":"payload","action":"","pretty":true,"x":320,"y":280,"wires":[["4588a594.1564c4"]]},{"id":"4588a594.1564c4","type":"function","z":"247a6012.ad0588","name":"Parse","func":"return {topic: 'Third Floor', payload: parseInt(msg.payload.confidence) || 0};","outputs":1,"noerr":0,"x":530,"y":280,"wires":[["90ce7582.c2eaf","57cddb63.1cb484","43318e46.1cdbc8"]]},{"id":"bbc4e61c.fd6ac8","type":"json","z":"247a6012.ad0588","name":"Format","property":"payload","action":"","pretty":true,"x":320,"y":220,"wires":[["bdf10878.08cde"]]},{"id":"bdf10878.08cde","type":"function","z":"247a6012.ad0588","name":"Parse","func":"return {topic: 'Garage', payload: parseInt(msg.payload.confidence) || 0};","outputs":1,"noerr":0,"x":530,"y":220,"wires":[["57cddb63.1cb484","90ce7582.c2eaf","43318e46.1cdbc8"]]},{"id":"1a73a15d.55ade7","type":"interval-length","z":"247a6012.ad0588","format":"mills","bytopic":false,"minimum":"","maximum":"","window":"","timeout":false,"msgTimeout":"7","minimumunit":"msecs","maximumunit":"msecs","windowunit":"msecs","msgTimeoutUnit":"mins","startup":true,"msgField":"timeout","repeatTimeout":false,"name":"","x":1460,"y":470,"wires":[[],["89b2dee8.2b689"]]},{"id":"89b2dee8.2b689","type":"function","z":"247a6012.ad0588","name":"FALLBACK TIMEOUT","func":"var confidence = parseInt(msg.payload);\n\n//after the timeout, if the confidence is zero, set to AWAY\nif (confidence === 0 ){\n return {payload : \"\", debug: \"Fallback\"}; \n}\n\nreturn null;","outputs":1,"noerr":0,"x":1720,"y":470,"wires":[["d2a06aed.f094c","3e0bf278.fe9c6e"]]},{"id":"d2a06aed.f094c","type":"debug","z":"247a6012.ad0588","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"debug","x":2190,"y":470,"wires":[]},{"id":"66d0cb36.e41b54","type":"comment","z":"247a6012.ad0588","name":"ALERT FOR FAILURE","info":"","x":1120,"y":620,"wires":[]},{"id":"33baaee6.5e5fca","type":"delay","z":"247a6012.ad0588","name":"1msg/10s","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"10","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"x":1430,"y":330,"wires":[["d8da0e0c.544c78"]]},{"id":"58ba2a40.903c6c","type":"server","z":"","name":"Home Assistant","url":"https://ADDRESS REDACTED:443","pass":"PASSWORD REDACTED"},{"id":"6e9ecea1.694bf8","type":"mqtt-broker","z":"","broker":"192.168.1.27","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"willTopic":"","willQos":"0","willRetain":"false","willPayload":"","birthTopic":"","birthQos":"0","birthRetain":"false","birthPayload":""},{"id":"4c32fd70.c128cc","type":"server","z":"","name":"Home Assistant","url":"https://ADDRESS REDACTED:443","pass":"PASSWORD REDACTED"}]
Example Use with AppDaemon
@PianSom has detailed instructions on using this with AppDaemon in this comment. Thanks, @PianSom!