Need help with Restful sensor or web scrape

balajeek1 · September 16, 2022, 2:54pm

My goal is to show last two train times (in min) on my route in the HA so I don’t have to use their app or check their website while I am walking fast to catch my train.

Luckily they have rest endpoints so I can get the json. here it is
https://path.api.razza.dev/v1/stations/newport/realtime
It lists all the train arrives at this station.

I did the restful template and was able to setup a sensor that shows like this

This is the code I used to get it:

 platform: rest
    name: path_train_wtc
    scan_interval: 30
    json_attributes:
      - upcomingTrains
      - projectedArrival
    resource: https://path.api.razza.dev/v1/stations/newport/realtime
    value_template: "{{ (value_json['upcomingTrains'][0]['projectedArrival'])}}"

  - platform: template
    sensors:
      path_train_wtc_1:
        icon_template: mdi:train
        device_class: timestamp
        friendly_name: "To HOB"
        value_template: "{{ states.sensor.path_train_wtc.attributes['upcomingTrains'][0]['projectedArrival']}}"

The issue I have is (if you look into the json) the api return top 4 or 5 trains arriving at the station, and the one I need is in one of the array and not necessarily a fixed index. Is there any trick to get the particular one?

I am also looking to see if I can scrape the website PATH Train Schedules , Maps, Fares and Station Updates, New York and New Jersey and get the particular card (they have many cards with same class ids, and I couldn’t nail them)

If any of this options works for me to get a card in my HA UI that would great.
Thanks.

CentralCommand · September 16, 2022, 3:07pm

Well I believe the answer is the selectattr filter. It picks out items in a list that have an attribute which match a test. You didn’t actually say how you find the right one in the list so I’ll write what I can and let you fill in the blanks:

value_template: >-
  {{ (value_json['upcomingTrains']
    | selectattr('<some attr>', 'eq', '<some value>')
    | first)['projectedArrival'] }}

Note that instead of equals you can use any Jinja test if your matching logic isn’t that simple. I was just guessing

But yea fill in the blanks and use that as value_template in your rest sensor

balajeek1 · September 16, 2022, 3:15pm

what does the pipe symbol mean, it is like subsequent expression or condition? wondering if I can do more than one selectattr

btw I am looking for


"lineName": "World Trade Center"
"direction": "TO_NY"
"projectedArrival": "2022-09-16T14:21:17Z"

CentralCommand · September 16, 2022, 3:26pm

Pipe symbol passes the output of the previous expression in as the first input of the next filter. Yes you can absolutely chain them.

Tbh that’s kind of Jinja (and templating languages in general) 101. I would strongly recommend at least skimming Jinja’s templating docs as its pretty core to HA:
https://jinja.palletsprojects.com/en/latest/templates/

Also here’s the doc on Home Assistant’s extensions to Jinja:

Although the latter is definitely secondary. HA basically just uses Jinja as is and added a few key functions, filters and tests.

EDIT: And yea so based on the data you shared I believe your template should be this:

value_template: >-
  {{ (value_json['upcomingTrains']
    | selectattr('lineName', 'eq', 'World Trade Center')
    | selectattr('direction', 'eq', 'TO_NY')
    | first)['projectedArrival'] }}

balajeek1 · September 16, 2022, 4:03pm

Thank you for your help.

I got confused about the updated value_template should be in the 1st section where I say “platform: rest” or it should be under “- platform: template”. So I added it to he second section and that did not work, so I added into the first section. As you can see in the second section I mention “states.sensor.path_train_wtc…” reference

basically I don’t understand why I need to put value_template under “platform: rest”
sorry I am not great at it and trying to do my best.

CentralCommand · September 20, 2022, 3:05pm

It should be in the REST sensor. I wasn’t really looking at the template sensor, I (perhaps incorrectly) assumed that was you trying things to work around the issues you were having with the normal REST sensor. Since it seemed like it was looking at the same field in the data?

balajeek1 · September 20, 2022, 3:08pm

Thank you Mike for helping me, yesterday I was able to make some changes and make it work the way I want. Surprisingly this morning it stopped working on my way to work, now looking at the log all I can see is home assistant got updated to the latest 2022.9.5 by watchtower and I get “Error while processing template”. Not sure if something is changed.

CentralCommand · September 20, 2022, 3:13pm

Can you share the config you used?

Also FYI you can test this in isolation pretty easily. Go to developer tools → template. In the box first add this:

{% set value_json = <copy and paste the raw text from https://path.api.razza.dev/v1/stations/newport/realtime > %}

Then below that add your template exactly as entered in value_template in the sensor. That way you will set the variable value_json exactly as it would be set when the sensor actually ran and then can test your template against it to see what happens.

balajeek1 · September 20, 2022, 3:28pm

working on it now …

I tried what you mentioned on developer tools-template: I get this
UndefinedError: ‘str object’ has no attribute ‘upcomingTrains’

(I shortened the json for screenshot by removing rows)

CentralCommand · September 20, 2022, 3:34pm

Remove the single quotes you wrapped the raw response in. value_json is supposed to be a dictionary not a string. Just paste in the JSON exactly as is, no added quotes.

EDIT: Removing rows is fine, doesn’t have to be exactly as is, sorry. Just meant don’t add quotes.

balajeek1 · September 20, 2022, 3:57pm

oh, removing single quote and it seem to work and gave me the result.

balajeek1 · September 20, 2022, 4:05pm

I went back to my config and removed all I had and start with one first. What I had is one platform:rest and two platform:template so I don’t have to run the rest multiple times for my templates. What I am grabbing is the first WTC train, status.

What do I have to do (as I have seen in other examples) where people grab the json into the platform:rest, and then use the rest sensor for all the templates ?

Also if there is more than one WTC train in the list, how can I grab that, do another template?

CentralCommand · September 20, 2022, 4:08pm

So I mean one thing that comes to mind is the template currently will have an error if there is no line in there with name World Trade Center and direction TO_NY. Does that happen? If so it can be handled. Something like this should work:

value_template: >-
  {% set trains = value_json['upcomingTrains']
    | selectattr('lineName', 'eq', 'World Trade Center')
    | selectattr('direction', 'eq', 'TO_NY')
    | list %}
  {{ trains[0]['projectedArrival']  if trains | count > 0 else None }}

That should make the sensor’s value go to unknown when there’s no train without an error and work normally otherwise. Perhaps that was the issue?

balajeek1 · September 20, 2022, 4:16pm

you are right, I noticed it showed ‘unknown’ when wtc train not in the list, it happen few times, good point and I copied your code with the conditions and pasted it and it seem work.

knoma · September 29, 2022, 9:21pm

@balajeek1

can you share you fully working example or write down all the steps ?

thank you

balajeek1 · September 30, 2022, 3:49pm

If you are looking to do the same, just copy the code from the first post and then replace the value_tempate part from last posted by Mike. Thats all and it should work.

Add this template into to your sensor.yaml

I used this for some days and then have made some changes, i did not like that HA pulling every few seconds and didn’t want that burden so i removed it.

Instead now i use node-red automation with geo-location, so when i am approaching the station i get notification sent to my apple watch that says “Train to WTC in 5min, then 12min”. it will keep sending me notification every 2 min as long as i am in the zone (defined in HA as my train station). Same thing for my return trip another geo-location automation. works great.

I also added a button in lovelace.ui so anytime i need i can click once and i get notification. It makes it easy that when notfication arrives i raise my hand and you see the info, don’t need to grab the phone from pocket,unlock,swipe and do all that stuff.

I will stick with this for now.

knoma · October 7, 2022, 1:13pm

thank you @balajeek1,

do you mind sharing the code?

I’m a newbie and added the part you mentioned but I see no data

I added the code from you and Mike in my /config/configuration.yaml under sensor

  - platform: rest
    name: path_train_wtc
    scan_interval: 30
    json_attributes:
      - upcomingTrains
      - projectedArrival
    resource: https://path.api.razza.dev/v1/stations/newport/realtime
    value_template: "{{ (value_json['upcomingTrains'][0]['projectedArrival'])}}"

  - platform: template
    sensors:
      path_train_wtc_1:
        icon_template: mdi:train
        device_class: timestamp
        friendly_name: "To HOB"
        value_template: >-
          {% set trains = value_json['upcomingTrains']
            | selectattr('lineName', 'eq', 'World Trade Center')
            | selectattr('direction', 'eq', 'TO_NY')
            | list %}
          {{ trains[0]['projectedArrival']  if trains | count > 0 else None }}

but this is the result

Not sure what is missing but I’m seeing this error

Logger: homeassistant.helpers.template_entity
Source: helpers/template_entity.py:356 
First occurred: 09:48:17 (1 occurrences) 
Last logged: 09:48:17

TemplateError('UndefinedError: 'value_json' is undefined') while processing template 'Template("{% set trains = value_json['upcomingTrains'] | selectattr('lineName', 'eq', 'World Trade Center') | selectattr('direction', 'eq', 'TO_NY') | list %} {{ trains[0]['projectedArrival'] if trains | count > 0 else None }}")' for attribute '_attr_native_value' in entity 'sensor.path_train_wtc_1'

Maybe indentation isn’t right

knoma · October 7, 2022, 1:13pm

I run the template tester and I can see data there

{## Imitate available variables: ##}
{% set value_json = {
 "upcomingTrains": [
  {
   "lineName": "33rd Street",
   "headsign": "33rd Street",
   "route": "JSQ_33",
   "routeDisplayName": "Journal Square - 33rd Street",
   "direction": "TO_NY",
   "lineColors": [
    "#FF9900"
   ],
   "status": "ARRIVING_NOW",
   "projectedArrival": "2022-10-07T13:10:14Z",
   "lastUpdated": "2022-10-07T13:10:14Z"
  },
  {
   "lineName": "World Trade Center",
   "headsign": "World Trade Center",
   "route": "HOB_WTC",
   "routeDisplayName": "Hoboken - World Trade Center",
   "direction": "TO_NY",
   "lineColors": [
    "#65C100"
   ],
   "status": "ON_TIME",
   "projectedArrival": "2022-10-07T13:11:43Z",
   "lastUpdated": "2022-10-07T13:10:14Z"
  },
  {
   "lineName": "Hoboken",
   "headsign": "Hoboken",
   "route": "HOB_WTC",
   "routeDisplayName": "World Trade Center - Hoboken",
   "direction": "TO_NJ",
   "lineColors": [
    "#65C100"
   ],
   "status": "ON_TIME",
   "projectedArrival": "2022-10-07T13:10:39Z",
   "lastUpdated": "2022-10-07T13:10:14Z"
  },
  {
   "lineName": "Journal Square",
   "headsign": "Journal Square",
   "route": "JSQ_33",
   "routeDisplayName": "33rd Street - Journal Square",
   "direction": "TO_NJ",
   "lineColors": [
    "#FF9900"
   ],
   "status": "ARRIVING_NOW",
   "projectedArrival": "2022-10-07T13:10:18Z",
   "lastUpdated": "2022-10-07T13:10:14Z"
  }
 ]
}
%}

value_template: >-
  {% set trains = value_json['upcomingTrains']
    | selectattr('lineName', 'eq', 'World Trade Center')
    | selectattr('direction', 'eq', 'TO_NY')
    | list %}
  {{ trains[0]['projectedArrival']  if trains | count > 0 else None }}

balajeek1 · October 7, 2022, 4:48pm

The code seem fine and not sure either why its not returning.
Try this, on the sensor path_train_wtc_1, change the valueTemplate to this:

value_template: >-
  {{ (value_json['upcomingTrains']
    | selectattr('lineName', 'eq', 'World Trade Center')
    | selectattr('direction', 'eq', 'TO_NY')
    | first)['projectedArrival'] }}

and then check what value you get from developer console for this sensor, it should be a date. Then when you add this sensor to the lovelace.ui it should automatically shows like " in 5 min"

balajeek1 · October 20, 2022, 8:09pm

@knoma, not sure if you still looking to fix this. I have a message for you

some days ago the mrazza api stopped working (which we get the json result from). I opened an issue in their github and he mentioned something wrong in the server may be and will look into it. Couple of days later it was still down and then i noticed path official site have the real-time data showing, so i wondered it has to come from some where, so i started looking it chrome developer console and ended up find this link that contains all the stations and timings in json format and it get updated real-time. So i am using this link on my node-red to accomplish the task.

https://www.panynj.gov/bin/portauthority/ridepath.json