[presence] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

Last Will and Testement? Isn’t that what it’s for?

yes, there is an option called “expire_after” that will change the mqtt sensor to “unknown” if it doesnt receive a msg after the defined time. (see here)

you can setup an automation when the state changes to “unknown”

Thanks for that, @simbesh.
How long are you guys observing Monitor (not presence) to take to set a user as away? I just observed monitor take 9 minutes to set my wife as away. Recognising someone ‘home’ is much quicker. For the record I only have one Pi Zero at the moment (more on the way).

Although…looking at my Mosquitto logs…it may have more to do with the Zero’s ability to stay connected to Mosquitto than the script. Unsure how to debug.

There’s either a LOT of this in my Hass mosquitto log

1535289038: New client connected from 192.168.0.104 as mosqsub|598-raspberrypi (c1, k60, u'').
1535289047: New connection from 192.168.0.104 on port 1883.
1535289047: New client connected from 192.168.0.104 as mosqpub|1095-raspberryp (c1, k60, u'').
1535289048: Client mosqpub|1095-raspberryp disconnected.
1535289052: New connection from 192.168.0.104 on port 1883.
1535289052: New client connected from 192.168.0.104 as mosqpub|1182-raspberryp (c1, k60, u'').
1535289053: Client mosqpub|1182-raspberryp disconnected.
1535289064: New connection from 192.168.0.104 on port 1883.
1535289064: New client connected from 192.168.0.104 as mosqpub|1354-raspberryp (c1, k60, u'').
1535289064: Client mosqpub|1354-raspberryp disconnected.

or JUST

1535288719: Saving in-memory database to /data/mosquitto.db.

I have a feeling when it’s just saving in-memory the Pi Zero isn’t connected to Hass.

I have a house with three WiFi APs but all broadcasting the same SSID. It’s not a mesh, though. Works with other devices…do you think it might be giving the Zero problems?

Edit: 16:03 Just discovered the “watch” function. Seeing lots of “arrive scan denied, hardware busy” messages.

Drawing

Sorry - don’t want to spam this thread with my issues - is there somewhere better to ask for technical help? HASS is DEFINITELY not being kept informed by monitor of roughly half the events that are going on - I suspect connectivity issues. Either the MQTT link or the WiFi connection on the Pi Zero.

What OS are you using? You need to get some type of MQTT broker so that you can subscribe to your MQTT server and see the events that are being sent to the service. Once you have this information you will know exactly what to tell HASS to look for.

On my mac I use MQTT.fx just to see what is happening on the MQTT server. If you read the monitor readme it tells you how the program works. its not going to keep sending the messages constantly unless you set it to do so with one of the parameters defined in the readme.

Depart takes longer to happen because of the confidence level idea. It will continue to get lower and lower until it reaches ZERO. But once you arrive it goes straight to 100 instead of slowly building like the depart does.

If you want a quicker depart you need to have a MQTT trigger sent to the MQTT server that forces a depart. Such as (depart when you dont see a ping from my device), or (depart when you see gate open).

also there is a behavior preferences file that you may want to modify for a quicker depart. This is located in the Monitor folder.

Hopefully someone can help me out with how to use the Presence tracker within my HA install.

I’ve successfully installed Presence on a Pi-Zero that I use to run Pi-Hole (I can see Presence communicating with my MQTT broker). I would like to combine the Presence sensor with the other device trackers that I am using to track at home presence (Router and ownTracks).

How would I take the numerical output from Presence and turn it into a sensor that either indicates home (=100) or away (say less than 60). Once I have this I was thinking that I could combine the three tracking sensors into a Bayesian sensor.

You combine those three values/sensors IN the Bayesian sensor, then make adjustments using the probability values.

Hello @Texangeek - take a look at the first post of this thread ^^^

One thing - you would be better moving from presence to monitor, which is Andrew’s up to date, more reliable and more supported latest work. Here

I noted that Presence appears to have been superseded by Andrew’s monitor software while skimming through this thread (all 548 posts of it!) however I thought it best to get things working with Presence before moving to Monitor.

I’m very new to HA so please bear with me if I’m asking what seem like stupid questions. I’ve inserted below the code that I have in my configuration.yaml file to manage the MQTT bluetooth trackers.

  - platform: mqtt
    name: Peter bluetooth
    state_topic: "location/owner/study/00:00:00:00:00:00"
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'

  - platform: mqtt
    name: Jane bluetooth
    state_topic: "location/owner/study/00:00:00:00:00:00"
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'

Also attached are a few lines from my MQTT broker log which appears to show the Pi connecting and publishing a message to the server (I have anonymized the IP and bluetooth addresses).

2018-08-27 18:54:00: New connection from xx.xx.xx.xx on port 15575.
2018-08-27 18:54:00: New client connected from xx.xx.xx.xx as mosqpub|16185-raspberry (c1, k60, u'Peter_BT').
2018-08-27 18:54:00: Sending CONNACK to mosqpub|16185-raspberry (0, 0)
2018-08-27 18:54:00: Received PUBLISH from mosqpub|16185-raspberry (d0, q0, r1, m0, 'Home/owner/Study/00:00:00:00:00:00', ... (124 bytes))
2018-08-27 18:54:00: Received DISCONNECT from mosqpub|16185-raspberry
2018-08-27 18:54:00: Client mosqpub|16185-raspberry disconnected.

There doesn’t appear to be anything in the log that shows the MQTT broker publishing the messages received to other clients (i.e. my HA installation). Is this as expected?

Also is my code in the configuration file correct to capture the output of the published MQTT messages (assuming they are being published by the broker)?

Lastly, you pointed me to the first post to presumably understand how to utilize Presence within HA. However I am only using a single Pi rather than the three in the example code. I’m trying to figure how I can use the numerical probability returned by presence together with the the two other device trackers that only show a binary presence (“Home”/“Away”)

Thanks

IIRC correctly, when a client makes an MQTT publication then that publication does not normally hit the log. You will be able to see it if you subscribe to the broker and look at what is being published. There are a number of ways of doing that - my personal preference is to run a command at a prompt, like

mosquitto_sub -u BROKER_USERNAME -P BROKER_PASSWORD -v -t \#

though there are programs with whizzy gui’s available - eg mqtt_spy, I think.

So - first task: make sure that monitor (and I strongly suggest you move asap) is successfully publishing to MQTT. You can then check that your sensor is picking up the publication (have a look at <YOUR_HA_IP>:8123/dev_state )

On a quick view, yes, your code looks fine.

For your final point, personally I use a collection of sensor’s (which I then combine to determine presence, using Appdaemon - which is perhaps a bit outside the scope of your query right now). But here are some suggestions on where to start:

- platform: mqtt
  name: "upstairsbt_ianiphone"
  state_topic: !secret ian_ups_mqtt
  value_template: "{{ value_json.confidence }}"

- platform: template
  sensors:
    upstairs_ianiphone:
      friendly_name: "Ian iPhone - Upstairs"
      value_template: >-
        {% if states('sensor.upstairsbt_ianiphone')|float == 100 %}
          Seen
        {% else %}
          Not seen
        {% endif %}

And here’s an old automation of mine, easily amended, which assumes the presence of an input_boolean:

- id: 08_ian_trigger_presence
  alias: 08 Detect presence 
  trigger:
    - platform: homeassistant
      event: start
    - platform: state
      entity_id: upstairs_ianiphone
  action:
    service_template: >
      {% if is_state('upstairs_ianiphone', "Seen") %}
      homeassistant.turn_on
      {% elif is_state('upstairs_ianiphone', "Not seen") %}
      homeassistant.turn_off
      {% endif %}
    entity_id:
      input_boolean.ian_at_home

There are LOTS of ways of doing this, and these are just some suggestions to build on, not definitive solutions!

I’m using Raspbian Lite, what Andrew calls Jessie Stretch Lite.

I’m pretty sure the problems I’m having are related to the Zero’s connection to WiFi rather than the script. I was just connected over SSH and got booted after 45 minutes. No idea why. And I couldn’t reconnect to the Pi for a full five minutes. I would LOVE to know what logs to check to find out what went on…

Just had a look at syslog, nothing weird going on (as far as I could see) when I got booted. AAandddd I’ve just been booted again. So as far as I can make out this is a WiFi issue??

My Pi is running Raspian Jessie.

I suspect that my problem may be that HA isn’t connecting to my MQTT broker correctly. In the log I can see both the Pi and my phone connecting but I don’t see anything from HA.

I’ll need to investigate a little further.

@Texangeek I see in the thread above that your MQTT log shows disconnects from your presence/monitor system? I’ve just had to shut mine down (monitor) as when I have it running and connected to the HA MQTT add-on, I get never ending connect/disconnects which seems to flood my broker and cause other devices to fail?

1534132924: New connection from 10.0.1.52 on port 1883.
1534132924: New client connected from 10.0.1.52 as mosqpub|725-pi-presence (c1, k60, u'').
1534132924: Client mosqpub|725-pi-presence disconnected.
1534132927: New connection from 10.0.1.51 on port 1883.
1534132927: New client connected from 10.0.1.51 as mosqpub|29235-pi-presen (c1, k60, u'').
1534132927: Client mosqpub|29235-pi-presen disconnected.

I have two Sonoffs that I had running ESPEasy and it was sure it was the firmware as I would loose control of them from the broker using any client. I then upgraded both to Tasmota and re-worked by entire HA config and much to my dismay, same thing when presence is running.

I’ve now had both presence Pi’s off for a week and MQTT is rock solid. Odd thing is, if I connect the presence Pi’s to my docker broker on my Synology NAS (running the same broker version), I see no disconnects in the log there.

FYI for anyone having MQTT instability issues with the HA MQTT add-on and presence/monitor.

P.S Was doing the same with presence before I migrated to monitor.

Think I’ve cracked my problem, possibly. My power supply is only 1A rated. So brown outs I’m guessing!

Re: @xbmcnut

@AndrewJfreyer ??

Anybody know if it’s possible to create an image based on a working, nicely configured Zero to install onto other Zeros? Or if there’s some fancy rsync(?) method of achieving the same end?

Cant you just clone the sd card?

@yuran sorry - very possibly I can - I just didn’t know if that would work given that the MAC addresses be different and the SD cards might be different sizes etc. But it sounds like a great solution if that’s not going to worry the Zero.

Hi, any updates of the iTag or the NUT beacon? Do they work currently? Thanks

What should the [topic_path]/scan/depart look like? What portion is the [topic_path]? is that "location/owner/kitchen"?

OK, I installed Monitor today and everything seemed to be fine. My Pi published messages to the MQTT broker and HA received those messages and correctly indicated that I was home.

However I just noted that the sensor that I am using for Monitor now shows me as Away. I SSH’ed into the Pi and tried to run monitor as I had earlier. However I receive an error as shown below:

**pi@raspberrypi** : **~/monitor $** sudo bash monitor.sh -a

Starting monitor.sh (v. 0.1.482)...

&gt; report all scan results mode enabled

&gt; preference: delay between scans = 5

&gt; preference: periodic arrive/depart check interval = 90

&gt; preference: periodic arrive interval = 90

&gt; preference: periodic depart interval = 120

&gt; preference: database refresh interval = 35

&gt; preference: max arrival scan attempts = 2

&gt; preference: max depart scan attempts = 4

&gt; preference: random advertisement expiration = 120

&gt; stopping other instances of 'monitor.sh'

Can't get device info: No such device

Can't open device: No such device

The Monitor service seems to be running on the PI:

pi@raspberrypi:~/monitor $ sudo systemctl status monitor.service
● monitor.service - Monitor Service
   Loaded: loaded (/etc/systemd/system/monitor.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-08-28 15:47:44 CDT; 18s ago
 Main PID: 4307 (bash)
   CGroup: /system.slice/monitor.service
           ├─4307 /bin/bash /home/pi/monitor/monitor.sh &
           ├─4386 /bin/bash /home/pi/monitor/monitor.sh &
           ├─4387 /bin/bash /home/pi/monitor/monitor.sh &
           ├─4389 /bin/bash /home/pi/monitor/monitor.sh &
           ├─4390 /bin/bash /home/pi/monitor/monitor.sh &
           ├─4391 /bin/bash /home/pi/monitor/monitor.sh &
           ├─4394 sleep 90
           ├─4397 sleep 35
           ├─4398 /bin/bash /home/pi/monitor/monitor.sh &
           ├─4402 /usr/bin/mosquitto_sub -v -h xx.xx.xx.xx -p 15575 -u User -P Pass -t Home/scan/# 
           └─4464 sleep 5

Aug 28 15:47:45 raspberrypi sudo[4395]: pam_unix(sudo:session): session closed for user root
Aug 28 15:47:50 raspberrypi sudo[4420]:     root : TTY=unknown ; PWD=/home/pi/monitor ; USER=root ; COMMAND=/usr
Aug 28 15:47:50 raspberrypi sudo[4420]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 28 15:47:50 raspberrypi sudo[4420]: pam_unix(sudo:session): session closed for user root
Aug 28 15:47:55 raspberrypi sudo[4441]:     root : TTY=unknown ; PWD=/home/pi/monitor ; USER=root ; COMMAND=/usr
Aug 28 15:47:55 raspberrypi sudo[4441]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 28 15:47:55 raspberrypi sudo[4441]: pam_unix(sudo:session): session closed for user root
Aug 28 15:48:00 raspberrypi sudo[4454]:     root : TTY=unknown ; PWD=/home/pi/monitor ; USER=root ; COMMAND=/usr
Aug 28 15:48:00 raspberrypi sudo[4454]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 28 15:48:01 raspberrypi sudo[4454]: pam_unix(sudo:session): session closed for user root

Does anyone know why I can no longer run Monitor from the command line. The

Sudo bash monitor.sh -a

command was very helpful in seeing what was going on.