GeoJSON Generalized Client

Fusion · August 15, 2020, 6:16am

So, kudos to @exxamalte who has been prolific with GeoJson integrations.

First by creating a base feed manager, then creating bespoke integrations on top of it. For instance: New sensor for alerts by NSW rural fire service - feedback welcome

I’ve been a tad frustrated with HomeAssistant’s default handling of geo_location platforms, with limitations such as all get_json_events share the same source attribute, making it difficult to create specific maps. The only workaround being, as @exxamalte has been doing, creating bespoke integrations.

This is frustrating because when we want to integrate something in HomeAssistant, we usually want to integrate it immediately and not wait for the proper integration. I know I’ve been a bit impatient that way

Anyway, introducing aio-geojson-qujery (hey, naming things is hard!) which is a consumer for any GeoJson feed out there. Specify what is needed to parse the feed in configuration.yaml and get going!

As with previous integrations, what I have done is create a Pypi package, then write the HomeAssistant integration as an example:

(this is where the README file you should look at is located)

Below: easy integration with Flex Table and the Map component:

msedge_g2jUsuDtJc

Here is the configuration:

configuration.yaml:

geo_location:
  - platform: aio_geojson_query
    source: myquakes
    endpoint: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson
    latitude: !secret home_latitude
    longitude: !secret home_longitude
    radius: 50
    criteria:
      - mag > 4.0
    mappings:
      id: code
      dateformat: milliseconds
      date: updated
    extra_fields:
      - mag
      - place
      - type
    entity_namespace: 'aio_earthquakes'
  - platform: aio_geojson_query
    source: myfires
    endpoint: https://www.fire.ca.gov/umbraco/api/IncidentApi/GeoJsonList?inactive=false
    latitude: !secret home_latitude
    longitude: !secret home_longitude
    radius: 50
    criteria:
      - IsActive == Y
    mappings:
      id: UniqueId
      dateformat: iso
      date: Updated
      title: Name
    extra_fields:
      - PercentContained
      - AcresBurned
    entity_namespace: 'aio_fires'

Note: I added a filter on IsActive since this source seems to now ignore the inactive parameter.

ui-lovelace.yaml:

          - type: horizontal-stack
            cards:
              - type: custom:button-card
                name: Map
                color: auto
                color_type: label-card
                tap_action:
                  action: call-service
                  service: browser_mod.command
                  service_data:
                    command: popup
                    title: Fires
                    card:
                      type: map
                      dark_mode: false
                      geo_location_sources:
                        - aio_geo_myfires
              - type: custom:button-card
                name: Map
                color: auto
                color_type: label-card
                tap_action:
                  action: call-service
                  service: browser_mod.command
                  service_data:
                    command: popup
                    title: Quakes
                    card:
                      type: map
                      dark_mode: false
                      geo_location_sources:
                        - aio_geo_myquakes

bacco007 · August 15, 2020, 7:19am

I’ve installed the custom_component and go to config a feed based on your example above, but am getting this error

Platform error geo_location.aio_geojson_query - No module named 'aio_geojson_query'

Fusion · August 15, 2020, 6:21pm

The manifest.json file contains a clear requirement, which HomeAssistant is expected to fetch when you restart it. If you could look into your logfiles/output if there is a reason why it was unable to fetch the library…?

Anyway, try installing it manually: pip install aio-geojson-query

Depending on your environment you may have to replace pip with pip3

bacco007 · August 15, 2020, 11:30pm

I had a look at the logs - I couldnt find any error, so tried installing manually as you suggested

I’m running HA OS, so I think i’ve hit a hurdle - installing manually fails trying to install pycares

    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'libffi', required by 'virtual:world', not found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'libffi', required by 'virtual:world', not found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'libffi', required by 'virtual:world', not found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'libffi', required by 'virtual:world', not found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'libffi', required by 'virtual:world', not found

Fusion · August 16, 2020, 7:44pm

Sorry about that: I am not familiar with HA-OS.
Googling it taught me that:

It is an OS where packages are found inside containers
Thus, not compatible with the concept of installing Python packages after the fact
I believe aiohttp is the problematic dependency, which is going to be an issue for many packages
What is missing is libffi-dev
This search shows HA-OS including libffi but not libffi-dev
This page is clear about libffi-dev being a hard requirement so it’s bit puzzling

Maybe I need to learn how to create a container for HA-OS. Or maybe someone who is familiar with the OS can shed some light on this topic?

IeiuniumLux · August 30, 2020, 2:21am

It filters the year fine, but does anyone know why it is ignoring the inactive parameter?

https://www.fire.ca.gov/umbraco/api/IncidentApi/GeoJsonList?year=2020&inactive=false

vatw · September 7, 2020, 6:03am

yes- it seems that the cal fire site doesn’t filter properly for inactive- (they are probably busy…) – I found an unwholesome workaround, involving a separate process that I run on a webhost I already had running on another computer on my network…

The ‘intermediate’ host modifies the json file coming down from calfire – I was able to make it work using the “jq” command in bash thus:

jq 'del(.features[] |  select(.properties.IsActive=="N"))' calFireAll.json  > calFireActive.json

(Probably overkill-- but… I don’t want a year’s worth of wildfire’s showing up). Alternative might be to use the RSS option…

I’m very interested in @Fusion 's solution here - since the default GeoJSON event integration doesn’t seem to be grabbing much data-- this will be a tomorrow project

StingBot · January 28, 2022, 4:57am

Sorry to dredge up an old post, but I can’t much info on this elsewhere.

Trying to link in the VicEmergency GeoJSON data feed.

It works fine in the standard GeoJSON integration, but with no valid attributes other than points on the map.

Is it possible the custom addon no longer works with the most recent HA.

My config is as follows:

geo_location:
  - platform: aio-geojson-query
    endpoint: https://emergency.vic.gov.au/public/osom-geojson.json
    source: VIC_Emergency
    criteria:
      status != "Complete"
    mappings:
      category1: Category1
      category2: Category2
      location: Location
      status: Status
      created: created_date
      updated: updated_date
      resources: resources
      sourceOrg: Source Organisation
      feedType: Feed Type
      entity_namespace: vic_emergency

And recieve this in the logs:

Logger: homeassistant
Source: custom_components/aio_geo_json_events/geo_location.py:165
Integration: aio_geo_json_events
First occurred: 3:52:36 PM (2 occurrences)
Last logged: 3:53:41 PM

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/aio_geo_json_events/geo_location.py", line 165, in async_update
    await self._feed_manager.update()
  File "/usr/local/lib/python3.9/site-packages/aio_geojson_client/feed_manager.py", line 74, in update
    status, feed_entries = await self._feed.update()
  File "/usr/local/lib/python3.9/site-packages/aio_geojson_client/feed.py", line 93, in update
    return await self._update_internal(
  File "/usr/local/lib/python3.9/site-packages/aio_geojson_client/feed.py", line 78, in _update_internal
    self._last_timestamp = self._extract_last_timestamp(filtered_entries)
  File "/usr/local/lib/python3.9/site-packages/aio_geojson_query/feed.py", line 84, in _extract_last_timestamp
    return dates[0]
IndexError: list index out of range

exxamalte · January 28, 2022, 7:40am

I’m not using this custom component myself, but I believe you need to define a mapping that maps a suitable date from the JSON to “date”. So something like this maybe:

mappings:
  ...
  date: updated

There are a few more mappings that you may need to add according to the documentation: GitHub - Fusion/python-aio-geojson-query

StingBot · January 28, 2022, 10:47am

thanks,

I’ve gone back to basics just trying the python on its own against that json URL, if I can get that to parse then I’ll try and extrapolate into the mappings into the HA code.

I have realised that the mappings are wrong in the above, re the date and the other fields.

Hopefully this custom component still works with the latest ones, if not hopefully I can twiddle the code to align with whatever the current requirements are.

thanks,

StingBot · January 28, 2022, 12:46pm

I’m not getting any results with Fusions code, even with the provided examples.

Your code works great, I have no idea what I am doing, so my idea is adapt the NSW RFS python example to the Vic GeoJSON file, run it as a daemon in a docker container and somehow output each valid entry into HA as a map point via MQTT or the API(if that is even possible, more research required) and have it run every 5 or 10mins(whatever the min the API allows)

edit: looks like this:

would suit to be adapted for the Vic Emergency file and your geojson feed, so I’ll give that a whirl.