RESTful sensor json help - Australian tide predictions (BOM)

[peterosullivan]

New to home assistant!

Having read the doc’s I’m trying to get tidal information from the URL below, using the rest platform. But I cant even get the current_time returned. The Developer Tool -> States for sensor.rest_tide unknown.

Here’s the returned json from http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=VIC_TP020&offset=false&tz=Australia%2FMelbourne

{ "results": { "current_time": 1574317137, "next_high": { "time": 1574322960, "height": 2.23 }, "next_low": { "time": 1574343480, "height": 0.55 } } }

I think the problem might be that the URL is returning HTML even though it looks like JSON.

sensor:
  - platform: rest
    resource: http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=VIC_TP020&offset=false&tz=Australia%2FMelbourne
    name: rest tide
    scan_interval: 30
    value_template: '{{ value_json.results.current_time }}'

[lolouk44]

something’s wrong with the URL I suspect.
I’ve copied to content of the page in a file and here are the results:

sensor:
  - platform: file
    file_path: /config/test1.json
    value_template: '{{ value_json.results.current_time }}'

  - platform: rest
    resource: http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=VIC_TP020&offset=false&tz=Australia%2FMelbourne
    name: rest tide
    scan_interval: 30
    value_template: '{{ value_json.results.current_time }}'

[peterosullivan]

Thanks for the reply, yes I think the URL is returning HTML and not JSON.

I wonder is there a way to converted using from_json or to_json shown here? https://www.home-assistant.io/docs/configuration/templating/#tofrom-json

[SteveDinn]

Came here to suggest this.

You might be able to use

(value | from_json).results.current_time

I’m thinking the page is just reporting itself as ‘text/html’ instead of ‘application/json’.

[lolouk44]

@SteveDinn

(value | from_json).results.current_time

doesn’t work. I think it should really be to_json but even with just value it fails.

So I tried with a scrape sensor and I get this:

Access Denied You don't have permission to access "http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?" on this server. Reference #18.8401d417.1574432882.b042b46

Went back to a rest sensor and added header info and “faked” a Chrome user agent and Bingo!

Full sensor declaration:

sensor:
  - platform: rest
    resource: http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=VIC_TP020&offset=false&tz=Australia%2FMelbourne
    name: rest tide_rest
    value_template: '{{ value_json.results.current_time }}'
    headers:
      User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
      Content-Type: application/json

[tom_l]

Is there a better way (including my awful regex) to do this?

- platform: rest
  resource: http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=TAS_TP003&offset=false&tz=Australia%2FHobart
  name: Hobart Tides
  value_template: '{{ value_json.results }}'
  headers:
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
    Content-Type: application/json
  scan_interval: 21600 # every six hours should be plenty fast enoough.

- platform: template
  sensors:
    hobart_tide_update_time:
      entity_id: sensor.hobart_tides
      friendly_name: 'Last Updated'
      value_template: >
        {{ states('sensor.hobart_tides')|regex_findall_index('\d+')|int|timestamp_custom("%a %H:%M", true) }}
    hobart_high_tide_time:
      entity_id: sensor.hobart_tides
      friendly_name: 'Next High Tide At'
      value_template: >
        {{ states('sensor.hobart_tides')|regex_findall_index('\d+',1)|int|timestamp_custom("%a %H:%M", true) }}
    hobart_high_tide_height:
      entity_id: sensor.hobart_tides
      friendly_name: 'High Tide Height'
      value_template: >
        {{ states('sensor.hobart_tides')|regex_findall_index('(?:[1-9]\d*|0)?(?:\.\d+)',0) }}
      unit_of_measurement: m
    hobart_low_tide_time:
      entity_id: sensor.hobart_tides
      friendly_name: 'Next Low Tide At'
      value_template: >
        {{ states('sensor.hobart_tides')|regex_findall_index('\d+',4)|int|timestamp_custom("%a %H:%M", true) }}
    hobart_low_tide_height:
      entity_id: sensor.hobart_tides
      friendly_name: 'Low Tide Height'
      value_template: >
        {{ states('sensor.hobart_tides')|regex_findall_index('(?:[1-9]\d*|0)?(?:\.\d+)',1) }}
      unit_of_measurement: m

It’s ugly but it works:

[SteveDinn]

What exactly does the state of sensor.hobart_tides look like?

[tom_l]

{'current_time': 1574442801, 'next_high': {'time': 1574449260, 'height': 1.3}, 'next_low': {'time': 1574471880, 'height': 0.63}}

But (as usual) it’s a string not JSON.

[SteveDinn]

You can use

  {{ (states('sensor.hobart_tides') | from_json).next_high.time }}

etc.

[SteveDinn]

@tom_l

574471880, 'height': 0.63}}

But (as usual) it’s a string not JSON.

Oh wait…I just got this. I think if you replace all the single quotes with double quotes, it’ll work.

[tom_l]

Ha! yeah:

Error rendering template: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

[SteveDinn]

{{ (states(‘sensor.hobart_tides’).Replace("’", ‘"’) | from_json).next_high.time }}

[SteveDinn]

ACtually, you can just put |to_json at the end of the value template in your REST sensor.

[tom_l]

I’ll give that a go.

EDIT: Yep. That did the trick. Thanks Steve.

- platform: rest
  resource: http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=TAS_TP003&offset=false&tz=Australia%2FHobart
  name: Hobart Tides
  value_template: '{{ value_json.results|to_json }}'
  headers:
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
    Content-Type: application/json
  scan_interval: 21600 # every six hours should be plenty fast enoough.

- platform: template
  sensors:
    hobart_tide_update_time:
      entity_id: sensor.hobart_tides
      friendly_name: 'Last Updated'
      value_template: >
        {{ (states('sensor.hobart_tides')|from_json).current_time|int|timestamp_custom("%a %H:%M", true) }}
    hobart_high_tide_time:
      entity_id: sensor.hobart_tides
      friendly_name: 'Next High Tide At'
      value_template: >
        {{ (states('sensor.hobart_tides')|from_json).next_high.time|int|timestamp_custom("%a %H:%M", true) }}
    hobart_high_tide_height:
      entity_id: sensor.hobart_tides
      friendly_name: 'High Tide Height'
      value_template: >
        {{ (states('sensor.hobart_tides')|from_json).next_high.height }}
      unit_of_measurement: m
    hobart_low_tide_time:
      entity_id: sensor.hobart_tides
      friendly_name: 'Next Low Tide At'
      value_template: >
        {{ (states('sensor.hobart_tides')|from_json).next_low.time|int|timestamp_custom("%a %H:%M", true) }}
    hobart_low_tide_height:
      entity_id: sensor.hobart_tides
      friendly_name: 'Low Tide Height'
      value_template: >
        {{ (states('sensor.hobart_tides')|from_json).next_low.height }}
      unit_of_measurement: m

[SteveDinn]

You may also want to consider using the JSON attributes property on that REST sensor. It’ll break all that stuff out into the sensor attributes if you want.

Just remember that if you can access fields on it, it’s not actually JSON, it’s an in-memory object that is constructed USING the JSON. The default “string” form is python’s native one, which is that weird, single-quoted JSON-like string. To get it to an ACTUAL JSON string, you need ‘to_json’.

[tom_l]

It’ll break all that stuff out into the sensor attributes if you want.

I didn’t think you could use nested json in restful sensor attributes?

[SteveDinn]

Hrm…Maybe I’m wrong in that regard, but you have to be careful that the actual state doesn’t exceed 255 characters. The attributes don’t have that limitation.

[tom_l]

Yeah I did check that. It was something like 130 characters for the current readings, so plenty of room for small changes.

[peterosullivan]

Thanks guys for all the input really appreciate it. I’m sure I’ll be able to use your suggestions above to get it working.

In the interests of completeness another approach that I think is close, is to use command_line combined with an inline python script. I’ve tried two versions;

  - platform: command_line
    command: python3 -c "import requests; print(requests.get('http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=VIC_TP020&offset=false&tz=Australia%2FMelbourne').json()['results'])"
    name: flinders high

This tries to use python to dump the JSON i.e. json.dumps(str)

  - platform: command_line
    command: python3 -c "import requests; import json; str = requests.get('http://www.bom.gov.au/australia/tides/scripts/getNextTides.php?aac=VIC_TP020&offset=false&tz=Australia%2FMelbourne').json()['results']; print(json.dumps(str))"
    name: flinders

Then use a template sensor, (which is NOT working) to display the values;

 platform: template
    sensors:
      six:
        value_template: '{{ states.sensor.flinders.next_high.time }} {{ states.sensor.flinders.next_high.height }}'

I feel this approach should work and is close, just need to work out the value_template part.

Thanks again for you help.

[peterosullivan]

Super work @tom_l and @SteveDinn thanks