Filter out bad GPS values

Configuration
1

I have a working GPS receiver coming in via GPSD.

I get bad location updates periodically which result in my device notifying me that it is ‘leaving’ and ‘entering’ my zone via an existing automation

I can access the ‘to’ and ‘from’ latitudes in my automation. I have tested these are accessible by including them in the notification from the automation and they always populate when the automation fires.

I use
trigger.to_state.attributes.latitude
and
trigger.from_state.attributes.latitude

I would like to ignore the bad data and it is characterised by impossibly big jumps in location that I could not actually do in the time between updates.

Is there a way I can setup a condition in my automation to ensure, for example, that the automation only runs if the difference between the two latitudes is less than 1 ?

Run automation if (trigger.to_state.attributes.latitude - trigger.from_state.attributes.latitude ) < 1

I am confused if I can achieve this through a value template in a numeric state condition or if I should be doing something else?

Any guidance appreciated! Thanks

2

Does your GPS entity have an accuracy attribute?

e.g. for my phone I can do this:

- condition: numeric_state
  entity_id: device_tracker.iphone13
  attribute: gps_accuracy
  below: 30 # meters
3

No, but the bad values I want to filter out or ignore are crazy bad.

For example, I am in Sydney Australia and the bad values put me off the coast of central america for one GPS update then they jump back to ‘correct values’.

Hence if I can just filter out any updates with big ‘jumps’ I will be fine. I just don’t know the best approach to compare the two ‘from’ and ‘to’ state attribute values in my automation’s condition (trigger.to_state.attributes.latitude and trigger.from_state.attributes.latitude)

4

Are you sure there is no accuracy attribute?

It could be called something like HDOP (horizontal dilution of precision).

5

I am using a USB connected UBlox GPS receiver that sends its updates in via GPSD - the updates are all coming in okay but I get these odd/invalid ones once in a while.

I think that the reason I see these occasional bad results is that GPSD is sending over (or HA is parsing) an occasional GST message rather than sticking to the location updates (TPV time-position-velocity messages) - the bad values exactly match GST data (down to the time stamp) which is actually a noise report and not a lat/long pair (see here gpsd_json(5))

I thought the easy thing might be to filter out the impossibly big jumps in lat/lon I am seeing but I don’t really know where how to get started in the automation?

6

FYI - these two issues address the underlying problem but neither have been actioned in HA so the issue us still there and hence I need to find a work around:

7

Ok lets look at a template condition then. To do this I need:

All your current trigger configuration for the automation.

An example of the state of the trigger entities and their attributes. An obfuscated screenshot from developer tools → states will do. Just don’t obscure the location so much that I can’t see the format of the data.

8

Thanks Tom,

I am hoping this is what you need - below is an example of a update where the to_state.latitude and from_state.latitude have bad and good data. This has come from ‘changed variables’ in a trace

Also below is the definition of the automation - you can see I can access the latitude attributes in the outgoing message.

TRACE:
this:
entity_id: automation.sensor_gps_left
state: ‘on’
attributes:
id: ‘1700172067916’
last_triggered: ‘2023-12-10T11:22:07.060581+00:00’
mode: single
current: 0
friendly_name: Sensor.gps left
last_changed: ‘2023-11-27T02:52:42.390149+00:00’
last_updated: ‘2023-12-10T11:22:07.064049+00:00’
context:
id: 01HH9QD2TMEG760535GS15B5HK
parent_id: 01HH9QD2TG7PPG2SFA5PZG9MDA
user_id: null
trigger:
id: ‘0’
idx: ‘0’
alias: null
platform: zone
entity_id: sensor.gps
from_state:
entity_id: sensor.gps
state: 3D Fix
attributes:
latitude: -33.776345667
longitude: 151.210182
elevation: 6.5
gps_time: ‘2023-12-10T12:14:06.000Z’
speed: 0.031
climb: -0.1
mode: 3
friendly_name: GPS
last_changed: ‘2023-11-27T02:53:12.885717+00:00’
last_updated: ‘2023-12-10T12:14:07.198317+00:00’
context:
id: 01HH9TC9TYAC64R2NDCE8W2RB5
parent_id: null
user_id: null
to_state:
entity_id: sensor.gps
state: 3D Fix
attributes:
latitude: 1.2
longitude: 1.2
elevation: 2.4
gps_time: ‘2023-12-10T12:14:37.000Z’
speed: 0.08
climb: -0.1
mode: 3
friendly_name: GPS
last_changed: ‘2023-11-27T02:53:12.885717+00:00’
last_updated: ‘2023-12-10T12:14:37.200044+00:00’
context:
id: 01HH9TD74GJHDYMKJZ2NS5YBY7
parent_id: null
user_id: null
zone:
entity_id: zone.home
state: ‘1’
attributes:
latitude: -33.776793034650595
longitude: 151.2102842330933
radius: 100
passive: false
persons:
- person.myharbour
editable: true
icon: mdi:home
friendly_name: Home
last_changed: ‘2023-12-10T09:59:27.644688+00:00’
last_updated: ‘2023-12-10T09:59:27.644688+00:00’
context:
id: 01HH9JNQMWCB43DS3AA2ADHT3K
parent_id: null
user_id: null
event: leave
description: sensor.gps leaving Home

AUTOMATION:
alias: Sensor.gps left
description: “”
trigger:

  • platform: zone
    entity_id: sensor.gps
    zone: zone.home
    event: leave
    condition:
    action:
  • device_id: f61d9a43a69a8a26fcc799915e2489b3
    domain: mobile_app
    type: notify
    message: >-
    Left!! ToLat:{{ trigger.to_state.attributes.latitude }} - FromLat:{{
    trigger.from_state.attributes.latitude }} - ToLon:{{
    trigger.to_state.attributes.longitude }} - FromLon:{{
    trigger.from_state.attributes.longitude }}
    mode: single
9

Is the info below plus my original post enough to suggest a technique/syntax for me to filter out these bad events/triggers in my automation? Thanks!

10

As for the “gps_accuracy” value.
In some cases it may be a “good” value - but coords may be wrong anyway.

Here is a case when some phone jumped to 0 coords and sometimes reported good accuracy:

11111
111111521×933 221 KB

Could be a GPS tracking server issue, or a local position tracking issue (for instance, when GPS services are jammed).
So I ended up filtering definitely wrong coords (0 with some delta) & coords with gps_accuracy > xxxxx.

11

Absolutely, because HDOP/VDOP is not accuracy. That’s a very common misconception. The *DOP values basically tell you how ideal the current satellite positions are for trilateration. But it’s only one of many factors that affect the accuracy achieved in practice. DOP does not take into account signal reception fading and noise, multipath reflections (very common in urban environments), atmospheric absorption and refraction, etc. All these will heavily influence the accuracy, but DOP alone can’t tell you that.

I’ve been playing around with some UBlox chips for my embedded boat monitor / tracker and ran into issues like that. UBlox chips actually send a specific accuracy parameter in meters, in additional to the usual NMEA phrases. They perform an internal calculation using DOP, snr, multipath, etc. It’s part of their $PUBX,00 extended NMEA phrases (see specs, page 69). They also send a status / validity flag that tells you exactly the type of lock they have and how reliable it is (2D/3D/DR/RTK/etc).

The Hacc / Vacc params are actually very reliable from my experience. I run the chip on very low power mode with partially updating ephemeris tables and a position every 2 hours in standby mode. The accuracy is about 2.5 to 6m usually according to Hacc, which I can visually confirm as my boat is moored in a fixed position.

image
image900×447 26.2 KB

Null island usually comes up when you lose the lock due to signal issues (see validity flag above) but still have good DOP values.

12

Strange that these “zero coords” issues occurred with 2 similar Android devices (Redmi 7A/9A) from 29.11 till 11.12 almost every night ))).
Blaming a Traccar location service - although they say that just using coords reported by a device.
Also could be GPS/GLONASS jamming issues. It’s rashist Russia, you know…

13

Can you help me understand how you ‘filter’ the bad ones out - it may not work for me (GPSD based updates) but maybe it will!!

14

Depends on the GNSS chip you have. Normally these chips will not give you zero coords, they will just give you an invalid lock flag. It’s the software that reads the data from the chip and interprets it that will sometimes put 0 coords in when the lock is lost (which is common, but kinda crappy practice).

A good DOP with no lock usually means that even if the satellites are positioned OK for a lock (good H/V/PDOP), something is wrong with the signal. This can be something harmless like a partially shielded antenna or maybe a thunderstorm in the signal path, or it can be something more nefarious like jamming.

1 Like
15

I am processing changes of 1…x device_trackers:

- trigger:
    - platform: state
      entity_id: ... some source device_tracker ...
    - platform: state
      entity_id: ... some source device_tracker ...
    - platform: state
      entity_id: ... some source device_tracker ...
    ...
  sensor:
    ...
    attributes:
      latitude: >-
        ...
        {%- if macro_map_test_CHECK_ZERO_COORDS(LATITUDE,LONGITUDE) == "777" -%}
          {{ LATITUDE }}
        {%- else -%}
          {{ state_attr(this.entity_id,"latitude") }}
        {%- endif -%}
      ...

where “macro_map_test_CHECK_ZERO_COORDS” is a macro:

{% macro macro_map_test_CHECK_ZERO_COORDS(input_LATITUDE,input_LONGITUDE) -%}
  {%- set CHECK_LATITUDE = macro_map_test_CHECK_ZERO_COORD_1(input_COORD = input_LATITUDE) -%}
  {%- set CHECK_LONGITUDE = macro_map_test_CHECK_ZERO_COORD_1(input_COORD = input_LONGITUDE) -%}
  {%- if CHECK_LATITUDE == '555' and
          CHECK_LONGITUDE == '555' -%}
    555
  {%- elif CHECK_LATITUDE == '999' or
            CHECK_LONGITUDE == '999' -%}
    999
  {%- else -%}
    777
  {%- endif -%}
{%- endmacro %}

{% macro macro_map_test_CHECK_ZERO_COORD_1(input_COORD) -%}
  {%- if is_number(input_COORD) -%}
    {%- set EPS = 0.001 -%}
    {%- if input_COORD|float|abs <= EPS|float -%}
      555
    {%- else -%}
      777
    {%- endif -%}
  {%- else -%}
    999
  {%- endif -%}
{%- endmacro %}

where “999” - unknown/none, “555” - zero coord, “777” - normal value.
Not elegant, but a quick fix made when I started seeing zero coords.

16

Thanks a lot for explanations, although I understood it just partly as I am not an expert ).
Will keep monitoring coords reported by devices.
Have not observed 0 coords for more than a week.

17

Neither am I, I just read a lot of technical docs and specs around this when I was writing the firmware for low level access to the Ublox chips. It’s actually pretty fascinating how all this works :slight_smile:

Did you try other GNSS besides GPS or Glonass ? Beidou and Galileo operate on a different frequency band. Then again, if they want to jam that stuff I’d guess they would do it on all bands.

18

Hmm, these are Chinese mobile phones bought on AliExpress.
They probably have support Beidou.
I’ll will try to check it.

Imho a device which supports several positioning services (gps, glonass, …) probably should have kind of toggles for every service, i.e. you may disable GPS or any other service.
iOS devices seem to be not having these toggles (Apple policy “do not allow users …”).

19

Ublox chips support this natively. Here’s the GNSS config page on their uCenter firmware configuration software:

image
image1353×726 70.9 KB

And at least on Android, you can list the GNSS satellites that were used for a fix by constellation (but I don’t know if you can exclude a specific constellation from being used).

Oh and this is a nice app for your Android phone to see what GNSS is around your position, with signal strength, etc. I used it quite a bit for debugging and testing.

1 Like
20

It’s okay Tom - I’ve figured it out here using templates in the automation condition. Thanks for the offer of help though.

Cheers