Rest API help needed (Multiple data from one resource)

Configuration
41

I had this earlier(which was slamming my hvac box-ie, sending update request every few ms)

  - authentication: basic
    username: "xx"
    password: "xx"
    scan_interval: 5
    resource: http://192.168.39.14/app/config/0/1
    sensor:
      - name: "CP_A1"
        value_template: "{{ value_json.Result.Point.0.Value }}"
      - name: "DP_A"
        value_template: "{{ value_json.Result.Point.2.Value }}"
      - name: "SP_A"
        value_template: "{{ value_json.Result.Point.3.Value }}"
      - name: "SCT_A"
        value_template: "{{ value_json.Result.Point.4.Value }}"
      - name: "SST_A"
        value_template: "{{ value_json.Result.Point.5.Value }}"

This isnt different from what you posted above. (or am I not reading it correct)

PS - I was reading this box using webcall of Gira HSFS(a knx solution) and it had no problems in getting it web scraped every 2 seconds. I am in process of moving to HA. I tried modbus - and it did the same - slammed millions of Modbus queries in a minute. Now this Rest is also doing the same. (other than editing after Andrew solution). But then it slams itself (rate limiting the sensors). I mentioned this to remove the doubt that this box has some kind of rate limitation built in, It doesnt. On being slammed, it just times out.(I didnot know this till I switched to HA)
Also- the above code was the source of my claim that Tim wasnt correct (in old thread)

42

That scan interval is in seconds, not ms. So there’s something wrong going on if you’re getting more than 1 update in 5 seconds.

43

I think you have a system time issue. Is this a virtual box or a virtual machine?

44

Hello
After abandoning Modbus , I turned to Rest API to pull data from my own equipment(no rate limits).
I found that HA regularly goes into unavailabe state (just like Modbus!!).
After delving deeper, i found this log entry -

2022-08-29 20:15:19.048 DEBUG (MainThread) [homeassistant.helpers.ratelimit] Reached rate limit of 0:01:00 for Template("{{ state_attr('sensor.hvac_1','Result')['Point'][2]['Value'] }}") and deferred action until 2022-08-29 16:16:19.011208+00:00

I couldnot find any documentation on this. I have powerful servers and I want my sesnors to update at high frequency. Why HA has this speed bump and where?

45

Templates aren’t rate limited unless the template is fired constantly. Andrews solution above is doing the same thing if your template that references 1 sensor is hitting the rate limit. That means it’s source is constantly changing. Just an FYI. Something is messed up with timing on your system. I run plenty of rest sensors and I can guarantee that the behavior is how I described above, not to mention I debug-attached to the the code earlier to verify this.

46

Yes. But decently powered.

47

I will try this - The problem is it changes sensor names.
All these sensors also send value to knx bus - so I have to change that too. A time consuming edit.

48

Is there some limitation in in HA when using json_attributes_path because I can’t get array saved into attributes even doe it works on http://jsonpath.com.

image
image1340×972 63.9 KB

JSON

{
    "location": {
        "name": "Samobor",
        "region": "Grad Zagreb",
        "country": "Croatia",
        "lat": 45.8,
        "lon": 15.72,
        "tz_id": "Europe/Zagreb",
        "localtime_epoch": 1679912365,
        "localtime": "2023-03-27 12:19"
    },
    "current": {
        "last_updated_epoch": 1679912100,
        "last_updated": "2023-03-27 12:15",
        "temp_c": 9.0,
        "temp_f": 48.2,
        "is_day": 1,
        "condition": {
            "text": "Partly cloudy",
            "icon": "//cdn.weatherapi.com/weather/64x64/day/116.png",
            "code": 1003
        },
        "wind_mph": 9.4,
        "wind_kph": 15.1,
        "wind_degree": 260,
        "wind_dir": "W",
        "pressure_mb": 1004.0,
        "pressure_in": 29.65,
        "precip_mm": 0.3,
        "precip_in": 0.01,
        "humidity": 76,
        "cloud": 75,
        "feelslike_c": 7.9,
        "feelslike_f": 46.1,
        "vis_km": 10.0,
        "vis_miles": 6.0,
        "uv": 2.0,
        "gust_mph": 5.6,
        "gust_kph": 9.0,
        "air_quality": {
            "co": 240.3000030517578,
            "no2": 3.700000047683716,
            "o3": 85.0999984741211,
            "so2": 1.0,
            "pm2_5": 1.600000023841858,
            "pm10": 2.5,
            "us-epa-index": 1,
            "gb-defra-index": 1
        }
    },
    "forecast": {
        "forecastday": [
            {
                "date": "2023-03-27",
                "date_epoch": 1679875200,
                "day": {
                    "maxtemp_c": 11.4,
                    "maxtemp_f": 52.5,
                    "mintemp_c": 5.2,
                    "mintemp_f": 41.4,
                    "avgtemp_c": 7.8,
                    "avgtemp_f": 46.0,
                    "maxwind_mph": 14.1,
                    "maxwind_kph": 22.7,
                    "totalprecip_mm": 52.2,
                    "totalprecip_in": 2.06,
                    "totalsnow_cm": 0.0,
                    "avgvis_km": 7.4,
                    "avgvis_miles": 4.0,
                    "avghumidity": 72.0,
                    "daily_will_it_rain": 1,
                    "daily_chance_of_rain": 89,
                    "daily_will_it_snow": 0,
                    "daily_chance_of_snow": 0,
                    "condition": {
                        "text": "Heavy rain",
                        "icon": "//cdn.weatherapi.com/weather/64x64/day/308.png",
                        "code": 1195
                    },
                    "uv": 2.0,
                    "air_quality": {
                        "co": 276.5040014648437,
                        "no2": 7.775999946594238,
                        "o3": 73.63599975585937,
                        "so2": 0.6160000064969062,
                        "pm2_5": 2.9479999852180483,
                        "pm10": 4.3599999904632565,
                        "us-epa-index": 1,
                        "gb-defra-index": 1
                    }
                },
                "astro": {
                    "sunrise": "06:46 AM",
                    "sunset": "07:17 PM",
                    "moonrise": "09:36 AM",
                    "moonset": "01:15 AM",
                    "moon_phase": "Waxing Crescent",
                    "moon_illumination": "30",
                    "is_moon_up": 0,
                    "is_sun_up": 0
                }
            },
            {
                "date": "2023-03-28",
                "date_epoch": 1679961600,
                "day": {
                    "maxtemp_c": 11.1,
                    "maxtemp_f": 52.0,
                    "mintemp_c": 2.6,
                    "mintemp_f": 36.7,
                    "avgtemp_c": 6.6,
                    "avgtemp_f": 43.9,
                    "maxwind_mph": 11.2,
                    "maxwind_kph": 18.0,
                    "totalprecip_mm": 0.0,
                    "totalprecip_in": 0.0,
                    "totalsnow_cm": 0.0,
                    "avgvis_km": 10.0,
                    "avgvis_miles": 6.0,
                    "avghumidity": 46.0,
                    "daily_will_it_rain": 0,
                    "daily_chance_of_rain": 0,
                    "daily_will_it_snow": 0,
                    "daily_chance_of_snow": 0,
                    "condition": {
                        "text": "Sunny",
                        "icon": "//cdn.weatherapi.com/weather/64x64/day/113.png",
                        "code": 1000
                    },
                    "uv": 3.0,
                    "air_quality": {
                        "co": 426.1880010986328,
                        "no2": 23.51599998474121,
                        "o3": 44.03199970066547,
                        "so2": 0.576000000834465,
                        "pm2_5": 11.303999965190888,
                        "pm10": 15.640000002384186,
                        "us-epa-index": 1,
                        "gb-defra-index": 1
                    }
                },
                "astro": {
                    "sunrise": "06:44 AM",
                    "sunset": "07:19 PM",
                    "moonrise": "10:19 AM",
                    "moonset": "02:20 AM",
                    "moon_phase": "Waxing Crescent",
                    "moon_illumination": "39",
                    "is_moon_up": 1,
                    "is_sun_up": 0
                }
            },
            {
                "date": "2023-03-29",
                "date_epoch": 1680048000,
                "day": {
                    "maxtemp_c": 13.4,
                    "maxtemp_f": 56.1,
                    "mintemp_c": 1.9,
                    "mintemp_f": 35.4,
                    "avgtemp_c": 7.6,
                    "avgtemp_f": 45.8,
                    "maxwind_mph": 11.0,
                    "maxwind_kph": 17.6,
                    "totalprecip_mm": 0.0,
                    "totalprecip_in": 0.0,
                    "totalsnow_cm": 0.0,
                    "avgvis_km": 10.0,
                    "avgvis_miles": 6.0,
                    "avghumidity": 58.0,
                    "daily_will_it_rain": 0,
                    "daily_chance_of_rain": 0,
                    "daily_will_it_snow": 0,
                    "daily_chance_of_snow": 0,
                    "condition": {
                        "text": "Partly cloudy",
                        "icon": "//cdn.weatherapi.com/weather/64x64/day/116.png",
                        "code": 1003
                    },
                    "uv": 3.0,
                    "air_quality": {
                        "co": 321.37199829101564,
                        "no2": 11.343999919891358,
                        "o3": 66.14799993515015,
                        "so2": 0.300000008046627,
                        "pm2_5": 9.58399995803833,
                        "pm10": 12.631999950408936,
                        "us-epa-index": 1,
                        "gb-defra-index": 1
                    }
                },
                "astro": {
                    "sunrise": "06:43 AM",
                    "sunset": "07:20 PM",
                    "moonrise": "11:12 AM",
                    "moonset": "03:16 AM",
                    "moon_phase": "First Quarter",
                    "moon_illumination": "49",
                    "is_moon_up": 1,
                    "is_sun_up": 0
                }
            }
        ]
    }
}

Wanted output:

[
  {
    "co": 276.5040014648437,
    "no2": 7.775999946594238,
    "o3": 73.63599975585937,
    "so2": 0.6160000064969062,
    "pm2_5": 2.9479999852180483,
    "pm10": 4.3599999904632565,
    "us-epa-index": 1,
    "gb-defra-index": 1
  },
  {
    "co": 426.1880010986328,
    "no2": 23.51599998474121,
    "o3": 44.03199970066547,
    "so2": 0.576000000834465,
    "pm2_5": 11.303999965190888,
    "pm10": 15.640000002384186,
    "us-epa-index": 1,
    "gb-defra-index": 1
  },
  {
    "co": 321.37199829101564,
    "no2": 11.343999919891358,
    "o3": 66.14799993515015,
    "so2": 0.300000008046627,
    "pm2_5": 9.58399995803833,
    "pm10": 12.631999950408936,
    "us-epa-index": 1,
    "gb-defra-index": 1
  }
]

HA code:

- resource_template: >
    {% set api_key = "removed" %}
    {% set location = "removed" %}
    https://api.weatherapi.com/v1/forecast.json?key={{ api_key }}&q={{ location }}&days=3&aqi=yes&alerts=no
  scan_interval: "00:30:00"
  sensor:
    - unique_id: air_quality_index_forecast
      name: Air Quality Index Forecast
      value_template: >
        {{ value_json['forecast']['forecastday'][0]['day']['air_quality']['gb-defra-index'] }}
      state_class: measurement
      device_class: aqi
      json_attributes_path: "$.forecast.forecastday.*.day"
      json_attributes:
        - air_quality

HA state:

image
image1104×281 21.1 KB

So instead having array of 3 sets of attributes I only get 1.

49

You need to loop through the array and build that output.
As far as I know there no “column” commands.

50

There is a way to use jinja syntax inside json_attributes_path or json_attributes? Or how do I loop over items in array in this case?

51

Not sure.
But you could create a template sensor that is based on this sensor.
Not a nice solution but it will work

52

Yeah I saw that as solution but there is so much data that it would probably take a lot of space for minimal amount that is actually needed.

53

Not sure I understand this.
Why would it take a lot of space? The data will take the same amount of space where ever it is, right?

54

Full response from API has 158k characters. And it only works when I save whole forecast object to sensor attribute. Then I could create templates from it but that’s just stupid if there is option to use json_attributes_path. Only problem is that it does not have full functionality. I would rather create my API in the middle where I would prepare data for HA instead of storing bunch of junk to DB each few minutes.

55

In that case I suggest you move this to Node Red since it will be able to do these things

56

Had that addon on my TODO list for a while will give it a try. Didn’t even realize it could do stuff like that. Thanks for suggestions.

57

Here is a start, if you import this then you get a sequence that gets the weather data from a Lat/Lon (need to edit this in the second node).

The function node does some magic and then outputs it as a sensor in HA

[{"id":"deb8fd1153e98ceb","type":"debug","z":"ad594fb84fe6d294","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":690,"y":80,"wires":[]},{"id":"baaeb67025c3b042","type":"function","z":"ad594fb84fe6d294","name":"","func":"var array = {};\nvar time = \"\";\n\ntime = new Date(msg.payload.approvedTime);\nvar offset = time.getTimezoneOffset(); \ntime = new Date(time.getTime() - (offset*60*1000)); \nmsg.state = time.toISOString().replace(\"T\", \" \").substring(0,16)\n\n\nfor (var index in msg.payload.timeSeries.slice(0,67)) {\n    time = new Date(msg.payload.timeSeries[index].validTime);\n    offset = time.getTimezoneOffset();\n    time = new Date(time.getTime() - (offset * 60 * 1000)); \n\n    array[time.toISOString().replace(\"T\", \" \").substring(0, 16)] = { \"temp\": 0, \"wind\": 0, \"gust\": 0, \"nederbord\": 0, \"symbol\" : 0 };\n    for (var val in msg.payload.timeSeries[index].parameters) {\n        if (msg.payload.timeSeries[index].parameters[val][\"name\"] == \"t\") {\n            array[time.toISOString().replace(\"T\", \" \").substring(0, 16)].temp = msg.payload.timeSeries[index].parameters[val].values[0];   //  { \"temp\": Math.round(msg.payload.timeSeries[index].parameters[val].values[0]) }\n        }else if (msg.payload.timeSeries[index].parameters[val][\"name\"] == \"ws\"){\n            array[time.toISOString().replace(\"T\", \" \").substring(0,16)].wind =  msg.payload.timeSeries[index].parameters[val].values[0];  \n        }else if (msg.payload.timeSeries[index].parameters[val][\"name\"] == \"gust\"){\n            array[time.toISOString().replace(\"T\", \" \").substring(0,16)].gust =  msg.payload.timeSeries[index].parameters[val].values[0];  \n        }else if (msg.payload.timeSeries[index].parameters[val][\"name\"] == \"pmax\") {\n            array[time.toISOString().replace(\"T\", \" \").substring(0, 16)].nederbord = msg.payload.timeSeries[index].parameters[val].values[0];  \n        } else if (msg.payload.timeSeries[index].parameters[val][\"name\"] == \"Wsymb2\") {\n            array[time.toISOString().replace(\"T\", \" \").substring(0, 16)].symbol = msg.payload.timeSeries[index].parameters[val].values[0];\n        }\n    }\n}\nmsg.offset = offset;\nmsg.arr = array;\nreturn msg;\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":480,"y":160,"wires":[["deb8fd1153e98ceb","aeb70bcf74c928d9"]]},{"id":"aeb70bcf74c928d9","type":"ha-entity","z":"ad594fb84fe6d294","name":"smhi hourly","server":"4bbca37b.1700ec","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"smhi hourly"},{"property":"device_class","value":""},{"property":"icon","value":""},{"property":"unit_of_measurement","value":""},{"property":"state_class","value":""},{"property":"last_reset","value":""}],"state":"state","stateType":"msg","attributes":[{"property":"hourly","value":"arr","valueType":"msg"}],"resend":true,"outputLocation":"payload","outputLocationType":"none","inputOverride":"allow","outputOnStateChange":false,"outputPayload":"$entity().state ? \"on\": \"off\"","outputPayloadType":"jsonata","x":670,"y":180,"wires":[[]]},{"id":"994a4699a32ef4ec","type":"inject","z":"ad594fb84fe6d294","name":"","props":[{"p":"payload"}],"repeat":"7200","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"str","x":150,"y":120,"wires":[["8584052db0255346"]]},{"id":"8584052db0255346","type":"http request","z":"ad594fb84fe6d294","name":"","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://opendata-download-metfcst.smhi.se/api/category/pmp3g/version/2/geotype/point/lon/<<<LON>>>/lat/<<<LAT>>>/data.json","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"credentials":{},"x":310,"y":120,"wires":[["baaeb67025c3b042"]]},{"id":"4bbca37b.1700ec","type":"server","name":"Home Assistant","version":2,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":30}]