Zanzito GPS Filter (python script)

So this thread is a WIP but I’ve got a working configuration and example to share with the Community. I’m hoping this will help someone. It was a learning process for me and in the end, quite enjoyable (even if I’m waaay off base here) but then I really like problem solving and the satisfaction of resolving problems.

Being an old Perl guy, learning Python wasn’t too hard but I did have to Google a lot to find what I was Perl-izing in my head to work. Please be gentle :slight_smile:

Requirements:

  • Zanzito (w/o OwnTracks emulation enabled.) My test devices also include iOS and works fine with it too.
  • At least one zone named ‘home’
  • HomeAssistant 0.68.0 (probably backwards compatible to a point)

Problem:

So like everyone else, I was using OwnTracks but got tired of the app registering Zone transitions most of the time but not passing the coordinates to HomeAssistant consistently. I would get false positives (even with max_gps_accuracy) or it simply wouldn’t register sometimes.

You may be asking yourself why not using OwnTracks emulation. Couple of reasons; if I wanted OwnTracks, I’d use it and two, documentation stated it was no longer necessary to use it so why use it?

Like some of you, Home Automation started off as a curiosity but became addictive to the point where Wife Approval became important. It was working for us to the point where my Wife began to expect it to work reliably. Was a challenge for me to get this working to her approval. I tried GPSLogger but since I have Android and she has iPhone, it was a challenge finding something cross platform that worked reliably.

I had seen people talking about Zanzito and the iOS HomeAssistant application and tried it. Surprisingly, the iOS application worked pretty darn reliably and Zanzito was too. This was back (heh… talking like it was a lifetime ago…) with Home Assistant 0.67.x – As I started upgrading, I noticed I was getting a lot of false positives (or lack of zone acknowledgement altogether) and once again I found myself staring down the business-end of my Wife’s approval.

So amid promises of “I know what’s happening and how to fix it” (I didn’t), I began to investigate the problem itself (and highly motivated to figure it out) and discovered two issues, mostly with me:

  • All my device_trackers were lumped under one known_device entry (WiFi and GPS) and resolved as a binary_sensor. This in itself wasn’t an issue but I ended up separating them to help diagnose into individual trackers.
  • GPS drift with Android. I have an ASUS Zenfone 3 running Oreo 8.0.0 - while not uncommon was causing me uncommon grief. Sometimes the drift (mostly indoors and never while moving) would be only a few meters up to a kilometre or more. Definitely an issue for me and how my configuration works which is based entirely around Zones and home vs not_home.

So I pursued this issue to no end. It may sound like I solved this in day but be assured this resolution was easily a couple of weeks in effort, mostly around identifying and troubleshooting the issue.

Whew. It was a heavy lift but I believe I have a working solution which I present to you, the Home Assistant Community!

Resolution:

Below you will see reference to device_tracker.chris - This is a meta tracker, it exists in my known_devices.yaml as seen below. There is also a device_tracker.chris_ping and device_tracker.chris_gps - All three have a friendly_name of ‘Chris’ (this is important for the script to work - so keep them consistent if you use something similar.)

chris:
  hide_if_away: false
  icon: mdi:cellphone-android
  mac:
  name: Chris
  picture:
  track: true
  vendor: Meta Tracker
  • Enable Python scripts in configuration.yaml
python_script:
  • Create Zanzito MQTT device_tracker to read GPS data in from my phone:
  - platform: mqtt_json
    qos: 0
    devices:
      chris_gps: zanzito/chris/location
  • Create automation for zones. Mine are combined and maybe a bit fancy. I send notifications to FB Messenger in mine (Wife and I) as well, I configure Zanzito to use high-precision GPS when leaving a zone and low-precision GPS when I enter a zone (credit goes to @scorpio862 for this part and to @anon43302295 for helping me with action-conditions) to save battery:
#### Presence: Zone
- alias: Tracker Enter Zone
  trigger:
    - platform: zone
      event: enter
      zone: zone.home
      entity_id: device_tracker.chris, device_tracker.val
    - platform: zone
      event: enter
      zone: zone.work_chris
      entity_id: device_tracker.chris, device_tracker.val
    - platform: zone
      event: enter
      zone: zone.work_val
      entity_id: device_tracker.chris, device_tracker.val
  action:
    - service: python_script.calc_gps_coords
      data_template:
        entity_id: '{{ trigger.entity_id }}'
        meta_entity: '{{ trigger.to_state.attributes.friendly_name }}'
        zone_entity: '{{ trigger.zone.entity_id }}'
        zone_data: '{{ trigger.event }}'
    - service: notify.fbmsg
      data_template:
        message: "{{ trigger.to_state.attributes.friendly_name }} arrived at {{ trigger.zone.attributes.friendly_name }} at {{ now().strftime('%Y-%m-%d %H:%M') }}"
        target:
          - !secret fbmsg_chris
          - !secret fbmsg_val
    - condition: state
      entity_id: device_tracker.chris
      state: 'home'
    - service: mqtt.publish
      data_template:
        topic: "zanzito/chris/set_prefs"
        payload: >-
          {
            "location_high_precision": false
          }

- alias: Tracker Leave Zone
  trigger:
    - platform: zone
      event: leave
      zone: zone.home
      entity_id: device_tracker.chris, device_tracker.val
    - platform: zone
      event: leave
      zone: zone.work_chris
      entity_id: device_tracker.chris, device_tracker.val
    - platform: zone
      event: leave
      zone: zone.work_val
      entity_id: device_tracker.chris, device_tracker.val
  action:
    - service: python_script.calc_gps_coords
      data_template:
        entity_id: '{{ trigger.entity_id }}'
        meta_entity: '{{ trigger.to_state.attributes.friendly_name }}'
        zone_entity: '{{ trigger.zone.entity_id }}'
        zone_data: '{{ trigger.event }}'
    - service: notify.fbmsg
      data_template:
        message: "{{ trigger.to_state.attributes.friendly_name }} left {{ trigger.zone.attributes.friendly_name }} at {{ now().strftime('%Y-%m-%d %H:%M') }}"
        target:
          - !secret fbmsg_chris
          - !secret fbmsg_val
    - condition: state
      entity_id: device_tracker.chris
      state: 'not_home'
    - service: mqtt.publish
      data_template:
        topic: "zanzito/chris/set_prefs"
        payload: >-
          {
            "location_high_precision": true
          }
  • Create an automation that will update GPS data when received:
- alias: GPS Filter
  trigger:
    - platform: state
      entity_id: device_tracker.chris_gps, device_tracker.val
  action:
    - service: python_script.calc_gps_coords
      data_template:
        entity_id: '{{ trigger.entity_id }}'
        meta_entity: '{{ trigger.to_state.attributes.friendly_name }}'
        zone_entity: 'None'
        zone_data: 'None'
  • If it doesn’t exist already, create a directory named python_scripts in your ~/.homeassistant directory. Assumptions are made that permissions jives, etc. Copy the script to this directory (at bottom of post.)

  • Restart Home Assistant.

  • Troubleshoot (if need be,) and Profit.

What it does:

  • When a zone transition occurs, GPS data is updated via the zone Enter and Leave automations.
  • When GPS data is available, the automation “GPS Filter” will execute the script to update the meta tracker(s).

When the tracker is updated, two considerations are taken into account:

  1. Is this a zone transition? If so, lock the meta tracker to exact coorindates of this zone when entering it. Create meta data called ‘meters’ and set gps_accuracy to 1 (absolute accurate without being 0.)
  2. Is a GPS coordinates update? We will only accept differences in meters if the difference is less-than existing meter calculation. This prevents drift and as differences get smaller, it becomes harder and harder to drift away from the zones actual coordinates. So when you’re ‘home’ you stay ‘home’ until the difference in meters exceeds a pre-determined amount (200m–based off recommendations for max_gps_accuracy for OwnTracks. I believe iOS uses 500m for ‘significant movement’ measurements which is pre-defined by the OS itself – I realize the intent of max_gps_accuracy is exactly the opposite of how I use it here.)

Measurements are made using the Haversine formula. This math is beyond my understanding (beyond basic grasp of what it does) and credit here goes to an Internet author named Nathan Rooy who published the Python code of this formula on his website, here:

Additional credit goes go @oakbrad who posted about a meta_tracker python script that I came across and then proceeded to pick apart his Git in cobbling this solution together:

Without both of these authors hard work, I wouldn’t be in the position I find myself today. Many thanks to these folks who shared their work with the world (and ultimately, me.)

For more information on the logic I used for this, please read the script in detail. For a finale before the script itself, I present some logging (sanitized for privacy) to validate my claims and demonstrates GPS drift when stationary without line-of-site of satellites :slight_smile:

[homeassistant.components.python_script.calc_gps_coords.py] GPS met threshold for Chris: 371.38 meters (200) [new=x,y old=x,y acc=72 state=not_home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS zone transition: Chris (leave) [zone=Home lat=x long=y radius=25]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 186.91 meters (200) [new=x,y old=x,y acc=72 meters=1000.00 state=Home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 23.78 meters (200) [new=x,y old=x,y acc=26 meters=186.91 state=not_home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS met threshold for Chris: 365.60 meters (200) [new=x,y old=x,y acc=46 state=not_home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 145.16 meters (200) [new=x,y old=x,y acc=46 meters=365.60 state=not_home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 22.79 meters (200) [new=x,y old=x,y acc=24 meters=145.16 state=not_home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS zone transition: Chris (enter) [zone=Home lat=x long=y radius=25]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 58.63 meters (200) [new=x,y old=x,y acc=1 meters=0.50 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 52.29 meters (200) [new=x,y old=x,y acc=1 meters=0.50 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 3.08 meters (200) [new=x,y old=x,y acc=1 meters=0.50 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 2.04 meters (200) [new=x,y old=x,y acc=1 meters=0.50 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 3.17 meters (200) [new=x,y old=x,y acc=1 meters=0.50 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 0.14 meters (200) [new=x,y old=x,y acc=1 meters=0.50 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS zone transition: Val (leave) [zone=Home lat=x long=y radius=25]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 27.78 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 17.75 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 8.73 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 11.60 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 7.49 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 12.76 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 18.01 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 0.03 meters (200) [new=x,y old=x,y acc=23 meters=0.14 state=home]
[homeassistant.components.python_script.calc_gps_coords.py] GPS below threshold for Chris: 3.84 meters (200) [new=x,y old=x,y acc=22 meters=0.03 state=home]

And finally, the script:

https://pastebin.com/e78yERmF

6 Likes

Just came to my attention after updating to 0.70.0 today that the script doesn’t function as it did before. Currently troubleshooting and will update the thread with new script once worked out.

Figured out what the issue is - now to resolve it. When HASS restarts, it doesn’t update the tracker device long/lat but if manually updated, it’ll work going forward. I’m going to have to re-jig the script a bit to deal with this so the caveat is that the tracker (right now) needs to be manually updated after a restart. Hopefully I’ll have some time this week to dig into it and re-write accordingly.

new version: https://pastebin.com/eZg8sJcb

Tested with a restart of HASS; it now properly updates data - seems I had a few python gotchas to learn but so far, so good.

1 Like

With many thanks to @pnbruckner for sanitizing this (more pythonized), I present a new version of the script. I had cleaned this up a lot since originally posting this but he cleaned it up even further (donate beer to man!)

Instead of posting it to pastebin, I post it here in case it goes down:

#
# calc_gps_coords.py
#
# based on meta_device_tracker.py by oakbrad
# https://github.com/oakbrad/brad-homeassistant-config/blob/master/python_scripts/meta_device_tracker.py
#
# adapted GPS Haversine calculation from nathan rooy
# https://nathanrooy.github.io/posts/2016-09-07/haversine-with-python/
#

# Enable debugging
ENABLE_DEBUG = 0

# Suggested max_gps_accuracy for Owntracks is 200m (656ft)
# https://www.home-assistant.io/components/device_tracker.owntracks/
#
# We'll use it too but use it to determine if the new value is greater-than current
# value to indicate significant movement or if GPS accuracy is within acceptable range.
# The idea behind this is to prevent false positive zone-transitions when stationary
# without line-of-sight of satellites and GPS lock drifts. Calculation is determined
# by using the Haversine formula. If the new value is greater-than current, we allow
# the meta_tracker to update otherwise we retain previous value.
#
MAX_GPS_METERS = 200

# triggered Entity
trigName = data.get('entity_id')
if not trigName:
  logger.error('ERROR: entity_id is missing!')
else:
  trigState = hass.states.get(trigName) # newState
  trigAttr = trigState.attributes.copy() # newstateAttributes
  trigStatus = trigState.state # newStatus

# metatracker Entity
metaName = "device_tracker." + data.get('meta_entity')
if not metaName:
  logger.error('ERROR: meta_entity is missing!')
else:
  metaState = hass.states.get(metaName) # currentState
  metaAttr = metaState.attributes.copy()
  metaStatus = metaState.state

# Get zone data, if it exists
zoneTransition = data.get('zone_data')
if not zoneTransition:
  logger.error('ERROR: zone_data is missing!')
else:
  zoneEntity = data.get('zone_entity')
  zoneState = hass.states.get(zoneEntity)
  try:
    zoneAttr = zoneState.attributes.copy()
  except:
    zoneAttr = trigState.attributes.copy()
    zoneAttr['friendly_name'] = 'not_home'

# Set some initial states for our metatracker
if metaAttr.get('latitude') is None:
  metaAttr['latitude'] = trigAttr['latitude']
  metaAttr['longitude'] = trigAttr['longitude']
  metaAttr['gps_accuracy'] = trigAttr['gps_accuracy']

if metaAttr.get('meters') is None:
    metaAttr['meters'] = 100

if ENABLE_DEBUG:
  logger.info("DEBUG: metaTrackerStates are %s", metaState)
  logger.info("DEBUG: triggerEntityStates are %s", trigState)
  logger.info("DEBUG: ---")
  logger.info("DEBUG: trigName is %s", trigName)
  logger.info("DEBUG: trigAttr are %s", trigAttr)
  logger.info("DEBUG: trigStatus is %s", trigStatus)
  logger.info("DEBUG: ---")
  logger.info("DEBUG: metaName is %s", metaName)
  logger.info("DEBUG: metaAttr are %s", metaAttr)
  logger.info("DEBUG: metaStatus is %s", metaStatus)
  logger.info("DEBUG: ---")
  logger.info("DEBUG: zoneEntity is %s", zoneEntity)
  logger.info("DEBUG: ZoneState is %s", zoneState)
  logger.info("DEBUG: zoneAttr is %s", zoneAttr)

# If GPS source, calculate GPS movement in meters
if trigAttr['source_type'] == 'gps':
  metaAttr['source_type'] = trigAttr['source_type']

  # If a zone trasition is triggered, use zone data to populate device_tracker.meta_entity
  # We set the distance for the zone to 0.5m to prevent GPS drift and lock the
  # device_tracker.meta_entity to the zone unless a leave event happens in which case we
  # begin calculating distance.
  if (zoneState is not None and trigAttr['gps_accuracy'] < MAX_GPS_METERS):
    logger.info("GPS zone transition: %s (%s) [zone=%s lat=%f long=%f acc=%d radius=%d]", metaName[15:],
      zoneTransition, zoneAttr['friendly_name'], zoneAttr['latitude'],
      zoneAttr['longitude'], trigAttr['gps_accuracy'], zoneAttr['radius'])

    if zoneTransition == 'enter':
      metaAttr['latitude'] = zoneAttr['latitude']
      metaAttr['longitude'] = zoneAttr['longitude']
      metaAttr['gps_accuracy'] = 1
      metaAttr['meters'] = 0.5
    else:
      metaAttr['latitude'] = trigAttr['latitude']
      metaAttr['longitude'] = trigAttr['longitude']
      metaAttr['gps_accuracy'] = trigAttr['gps_accuracy']
      metaAttr['meters'] = 100

      # Simple case-check for zone.home - if state was home, set not_home else
      # set state to zoneName. Ensure zone entity name matches in your config.
      metaStatus = 'not_home' if trigState.lower() == 'home' else zoneAttr['friendly_name']
  else:
    # Begin Haversine formula calculation. This calculation will take
    # newCoords and compare against currentCoords and determine the distance
    # in meters. As each calculation happens, if the result is smaller than
    # the last, keep it. The idea is that when in a particular zone, it allows
    # the device_tracker.meta_entity to zero in as close as possible and
    # eliminate GPS drift.
    R = 6371000
    phi_1 = math.radians(metaAttr['latitude'])
    phi_2 = math.radians(trigAttr['latitude'])

    delta_phi = math.radians(trigAttr['longitude'] - metaAttr['longitude'])
    delta_lambda = math.radians(trigAttr['longitude'] - metaAttr['longitude'])

    a = (math.sin(delta_phi/2.0)**2+
      math.cos(phi_1)*math.cos(phi_2)*
      math.sin(delta_lambda/2.0)**2)
    c = 2*math.atan2(math.sqrt(a),math.sqrt(1-a))

    meters = round(R*c, 2)
    if meters > MAX_GPS_METERS and trigAttr['gps_accuracy'] < MAX_GPS_METERS:
      logger.info("GPS met threshold for %s: %.2f meters (%d) [new=%f,%f old=%f,%f acc=%d state=%s]", metaName[15:],
        meters, MAX_GPS_METERS, trigAttr['latitude'],
        trigAttr['longitude'], metaAttr['latitude'], metaAttr['longitude'],
        trigAttr['gps_accuracy'], trigStatus)
      metaAttr['latitude'] = trigAttr['latitude']
      metaAttr['longitude'] = trigAttr['longitude']
      metaAttr['gps_accuracy'] = trigAttr['gps_accuracy']
      metaAttr['meters'] = meters
    else:
      # keep values that are lowest in meters from previous to ensure
      # we stay within a zone measurement similar to max_gps_accuracy
      if meters != 0:
        logger.info("GPS below threshold for %s: %.2f meters (%d) [new=%f,%f old=%f,%f acc=%d meters=%.2f state=%s]", metaName[15:],
          meters, MAX_GPS_METERS, trigAttr['latitude'],
          trigAttr['longitude'], metaAttr['latitude'], metaAttr['longitude'],
          metaAttr['gps_accuracy'], metaAttr['meters'], metaStatus)
        if meters <= metaAttr['meters']:
          if trigAttr['latitude'] != metaAttr['latitude'] or trigAttr['longitude'] != metaAttr['longitude']:
            metaAttr['latitude'] = trigAttr['latitude']
            metaAttr['longitude'] = trigAttr['longitude']
            metaAttr['gps_accuracy'] = trigAttr['gps_accuracy']
            metaAttr['meters'] = meters

# Update device_tracker.meta_entity. We'll inherit all trigger states
# and overwrite with the calculated meta states where appropriate.
trigAttr.update(metaAttr)
hass.states.set(metaName, trigStatus, trigAttr)

EDIT: Updated with more error checking; ran into a issue today where zoneAttr being None caused an issue.

2 Likes

Is there a reason you wouldn’t just use zanzito along with google geocode?

Not familiar with Google geocode. I’ve been slowly moving away from Google services in my HA setup due to the changes in policies. But I will look this up, thanks.

EDIT: Just read up on it here:

Stopped reading here:

Note: Google Maps APIs Premium Plan customers may use either an API key, or a valid client ID and digital signature, in your Geocoding requests. Get more information on authentication parameters for Premium Plan customers.

I’ve moved to OpenStreetMap and removed all other Google services (other than HTML push) from my HA instance. Since they changed their Maps policy (and friends) where I need to supply a credit card even for “free” usage, I’ve stopped using their services.

Thats fair. I was mainly curious if your solution was doing something different and if the Google Geocode was something that was similar or not.

It does as it corrects (or I should say, locks) the device from GPS drift.

Planning to refactor this script due to recent changes in HASS. Will post updated code once completed and tested.

2 Likes

What happened to this script? I was using it but it doesn’t work anymore, I guess something in the recent updates broke it?

Yes, it’s an issue (from what I can tell) with restore_state. I filed a bug but I haven’t seen any traction on it. I’ve since refactored it, it “works” but it’s not as smooth as it was. Still working on it.

EDIT: Issue #19702

Unfortunately, my github issue has been closed. Will continue to refactor this until I can get a working solution with the newer HASS code. Until then, this script is probably not going to work as before.

Good news - I believe I have this fixed. I will post new code once I’m 100% positive it’s working.

Just an update - things appear to be working well. Re-writing this as a custom component has been a lot harder than I originally anticipated mostly because of my lack of understanding (in general) around the internals of HA as it relates to this.

In the interim, the workaround for having the entity constantly reset is to remove the meta tracker device from known_devices.yaml and let the script build it. From there, it should work as expected.

I’ve re-written some of the original python-based code but still testing this but once I’m satisfied, I’ll post it but still intend to re-write as a custom component device_tracker platform.

That’s good news! I’ll keep an eye out for it!