Reading JSON array with REST sensor

robotek · March 17, 2021, 9:35am

Hi all!

I have a list of dates from our electricity provider when electricity will be out due to maintenance from an REST API:
https://emspletnestoritve.azurewebsites.net/Izklopi/SlovenskaBistrica.json

I would like to get all of the values and displayed in on my dashboard (with some filters).
I just could not figure out how to get JSON values - the complete array stored into HA to create some templates. This is my current config:

   - platform: rest
     resource: https://emspletnestoritve.azurewebsites.net/Izklopi/SlovenskaBistrica.json
     name: Izpadi elektrike
     json_attributes_path: "$.Izklopi.Izklop[0]"
     json_attributes:
      - RadioObjava
      - DatumOd
      - UraOd
      - UraDo
     value_template: 'OK'

Any ideas what I’m doing wrong? Thanks!

m0wlheld · March 17, 2021, 10:05am

a) using wrong formatting for code sequences (it’s the </> icon)
b) properly describing the problem (or challenge). It the configuration invalid, does the sensor not receive any values or are you unsure about how to display data in the UI.

The value_template looks strange, too. By content and indentation. Maybe better to the presence and length of the Izkopi array as value_template.

But start with a) and b), please.

robotek · March 17, 2021, 10:14am

I fixed the code. Thanks.

I am just not able to get any values into the sensor. I did set value_template to OK since the JSON is too big to be set as a value.

By default, the sensor state would be set to the full JSON — here, that would exceed the 255-character maximum allowed length for the state, so we override that default by using value_template to set a static value of OK.

m0wlheld · March 17, 2021, 10:32am

So there should be at least a sensor.izpadi_elektrike with a state of ‘OK’. Do you see that in the Developer Tools?

pnbruckner · March 17, 2021, 1:46pm

Hmm, this is interesting. I see nothing wrong with your config. I tried it myself, and by enabling debug for homeassistant.components.rest I see this:

2021-03-17 08:42:20 WARNING (MainThread) [homeassistant.components.rest.sensor] REST result could not be parsed as JSON
2021-03-17 08:42:20 DEBUG (MainThread) [homeassistant.components.rest.sensor] Erroneous JSON: {
"Izklopi":
{
  "Izklop": [
    {
      "ImeTP": "T-272 GLADOMES",
      "SifraTP": "4000497",
      "ImeIzvoda": "T-272 GLADOMES",
      "Enota": "Slovenska Bistrica",
      "Opomba": "REVIZIJA TP",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "08:00",
      "UraDo": "10:00",
      "RadioObjava": "TP GLADOMES(1979)",
      "Preklican": false,
      "Lat": "46.39019988",
      "Lng": "15.49752966"
    },
    {
      "ImeTP": "T-316 STRUG",
      "SifraTP": "4000589",
      "ImeIzvoda": "T-316 STRUG",
      "Enota": "Slovenska Bistrica",
      "Opomba": "REKONSTRUKCIJA NNO",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "08:00",
      "UraDo": "14:00",
      "RadioObjava": "TP STRUG",
      "Preklican": false,
      "Lat": "46.33677619",
      "Lng": "15.69445733"
    },
    {
      "ImeTP": "T-376 PETELINJEK",
      "SifraTP": "4000609",
      "ImeIzvoda": "T-376 PETELINJEK",
      "Enota": "Slovenska Bistrica",
      "Opomba": "PREDELAVA NN IZVODOV",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "08:00",
      "UraDo": "12:00",
      "RadioObjava": "TP PETELINJEK",
      "Preklican": false,
      "Lat": "46.33479397",
      "Lng": "15.51205191"
    },
    {
      "ImeTP": "T-147 BREZJE PRI LOČAH",
      "SifraTP": "4000185",
      "ImeIzvoda": "T-147 BREZJE PRI LOČAH",
      "Enota": "Slovenska Bistrica",
      "Opomba": "PREDELAVA NN IZVODOV",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "08:00",
      "UraDo": "12:00",
      "RadioObjava": "TP BREZJE PRI LOČAH",
      "Preklican": false,
      "Lat": "46.33597005",
      "Lng": "15.50164316"
    },
    {
      "ImeTP": "T-559 LESKOVEC 3",
      "SifraTP": "4099538",
      "ImeIzvoda": "I-01 LESKOVEC CENTER",
      "Enota": "Slovenska Bistrica",
      "Opomba": "Okvara - Defekt",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "09:16",
      "UraDo": "10:15",
      "RadioObjava": "",
      "Preklican": false,
      "Lat": "46.38697661",
      "Lng": "15.62133786"
    },
    {
      "ImeTP": "T-338 TEPANJE 4",
      "SifraTP": "4000243",
      "ImeIzvoda": "I-06 OMARICA 4",
      "Enota": "Slovenska Bistrica",
      "Opomba": "Okvara - Defekt",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "10:12",
      "UraDo": "10:45",
      "RadioObjava": "TP TEPANJE 4, nizkonapetostni izvod: I-06 OMARICA 4",
      "Preklican": false,
      "Lat": "46.34374194",
      "Lng": "15.46986619"
    },
    {
      "ImeTP": "T-342 KORPLJE",
      "SifraTP": "4000452",
      "ImeIzvoda": "T-342 KORPLJE",
      "Enota": "Slovenska Bistrica",
      "Opomba": "REVIZIJA TP",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "10:30",
      "UraDo": "12:00",
      "RadioObjava": "TP KORPLJE",
      "Preklican": false,
      "Lat": "46.38053558",
      "Lng": "15.50602738"
    },
    {
      "ImeTP": "T-499 STRAŽA 2",
      "SifraTP": "4000489",
      "ImeIzvoda": "T-499 STRAŽA 2",
      "Enota": "Slovenska Bistrica",
      "Opomba": "REVIZIJA TP",
      "DatumOd": "17.03.2021",
      "DatumOdInt": "20210317",
      "DatumDo": "17.03.2021",
      "DatumDoInt": "20210317",
      "UraOd": "12:30",
      "UraDo": "14:00",
      "RadioObjava": "TP STRAŽA 2",
      "Preklican": false,
      "Lat": "46.38338288",
      "Lng": "15.47340691"
    },
    {
      "ImeTP": "T-127 BUKOVA GORA 1",
      "SifraTP": "4000364",
      "ImeIzvoda": "T-127 BUKOVA GORA 1",
      "Enota": "Slovenska Bistrica",
      "Opomba": "POSTAVITEV NOVE TP",
      "DatumOd": "18.03.2021",
      "DatumOdInt": "20210318",
      "DatumDo": "18.03.2021",
      "DatumDoInt": "20210318",
      "UraOd": "08:00",
      "UraDo": "14:00",
      "RadioObjava": "TP BUKOVA GORA 1",
      "Preklican": false,
      "Lat": "46.389418",
      "Lng": "15.34411894"
    },
    {
      "ImeTP": "T-443 BUKOVA GORA 2",
      "SifraTP": "4000400",
      "ImeIzvoda": "T-443 BUKOVA GORA 2",
      "Enota": "Slovenska Bistrica",
      "Opomba": "POSTAVITEV NOVE TP",
      "DatumOd": "18.03.2021",
      "DatumOdInt": "20210318",
      "DatumDo": "18.03.2021",
      "DatumDoInt": "20210318",
      "UraOd": "08:00",
      "UraDo": "14:00",
      "RadioObjava": "TP BUKOVA GORA 2(1987)",
      "Preklican": false,
      "Lat": "46.40043513",
      "Lng": "15.33643091"
    },
    {
      "ImeTP": "T-461 LJUBNICA",
      "SifraTP": "4000403",
      "ImeIzvoda": "T-461 LJUBNICA",
      "Enota": "Slovenska Bistrica",
      "Opomba": "POSTAVITEV NOVE TP",
      "DatumOd": "18.03.2021",
      "DatumOdInt": "20210318",
      "DatumDo": "18.03.2021",
      "DatumDoInt": "20210318",
      "UraOd": "08:00",
      "UraDo": "14:00",
      "RadioObjava": "TP LJUBNICA",
      "Preklican": false,
      "Lat": "46.38945308",
      "Lng": "15.33186647"
    },
    {
      "ImeTP": "T-667 BUKOVA GORA 3",
      "SifraTP": "4154304",
      "ImeIzvoda": "T-667 BUKOVA GORA 3",
      "Enota": "Slovenska Bistrica",
      "Opomba": "POSTAVITEV NOVE TP",
      "DatumOd": "18.03.2021",
      "DatumOdInt": "20210318",
      "DatumDo": "18.03.2021",
      "DatumDoInt": "20210318",
      "UraOd": "08:00",
      "UraDo": "14:00",
      "RadioObjava": "TP BUKOVA GORA 3",
      "Preklican": false,
      "Lat": "46.39828983",
      "Lng": "15.34603244"
    },
    {
      "ImeTP": "T-145 PRAGERSKO TEHTNICA",
      "SifraTP": "4000203",
      "ImeIzvoda": "I-04 BLOK",
      "Enota": "Slovenska Bistrica",
      "Opomba": "IZDELAVA PRIKLJUČKA",
      "DatumOd": "19.03.2021",
      "DatumOdInt": "20210319",
      "DatumDo": "19.03.2021",
      "DatumDoInt": "20210319",
      "UraOd": "08:00",
      "UraDo": "09:00",
      "RadioObjava": "TP PRAGERSKO TEHTNICA, nizkonapetostni izvod: I-04 BLOK",
      "Preklican": false,
      "Lat": "46.39316236",
      "Lng": "15.662094"
    },
    {
      "ImeTP": "T-077 ZAFOŠT 1",
      "SifraTP": "4000420",
      "ImeIzvoda": "T-077 ZAFOŠT 1",
      "Enota": "Slovenska Bistrica",
      "Opomba": "MONTAŽA DLC",
      "DatumOd": "19.03.2021",
      "DatumOdInt": "20210319",
      "DatumDo": "19.03.2021",
      "DatumDoInt": "20210319",
      "UraOd": "09:00",
      "UraDo": "11:00",
      "RadioObjava": "TP ZAFOŠT 1(1959)",
      "Preklican": false,
      "Lat": "46.38076608",
      "Lng": "15.54147655"
    }
  ]
}
}

But I don’t see why it’s complaining. I tried it via curl & jq and it seems to work fine:

$ curl -s https://emspletnestoritve.azurewebsites.net/Izklopi/SlovenskaBistrica.json | jq .Izklopi.Izklop[0]
{
  "ImeTP": "T-272 GLADOMES",
  "SifraTP": "4000497",
  "ImeIzvoda": "T-272 GLADOMES",
  "Enota": "Slovenska Bistrica",
  "Opomba": "REVIZIJA TP",
  "DatumOd": "17.03.2021",
  "DatumOdInt": "20210317",
  "DatumDo": "17.03.2021",
  "DatumDoInt": "20210317",
  "UraOd": "08:00",
  "UraDo": "10:00",
  "RadioObjava": "TP GLADOMES(1979)",
  "Preklican": false,
  "Lat": "46.39019988",
  "Lng": "15.49752966"
}
$ curl -s https://emspletnestoritve.azurewebsites.net/Izklopi/SlovenskaBistrica.json | jq .Izklopi.Izklop[0].RadioObjava
"TP GLADOMES(1979)"

Seems like a bug.

pnbruckner · March 17, 2021, 2:36pm

Ok, I figured out what the problem is. Not sure the best way to fix it though.

By changing the code slightly I was able to figure out what the root problem is. Here’s the exception:

2021-03-17 09:03:04 ERROR (MainThread) [homeassistant.components.sensor] Error adding entities for domain sensor with platform rest
Traceback (most recent call last):
  File "/home/phil/repos/core/homeassistant/helpers/entity_platform.py", line 317, in async_add_entities
    await asyncio.gather(*tasks)
  File "/home/phil/repos/core/homeassistant/helpers/entity_platform.py", line 509, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/home/phil/repos/core/homeassistant/helpers/entity.py", line 529, in add_to_platform_finish
    await self.async_added_to_hass()
  File "/home/phil/repos/core/homeassistant/components/rest/entity.py", line 64, in async_added_to_hass
    self._update_from_rest_data()
  File "/home/phil/repos/core/homeassistant/components/rest/sensor.py", line 152, in _update_from_rest_data
    json_dict = json.loads(value)
  File "/usr/lib/python3.8/json/__init__.py", line 337, in loads
    raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

It seems the output from the website includes a BOM (Byte Order Mark) at the beginning (which I’m guessing is not very standard.) When the data is decoded by the httpx package that HA uses it’s not properly handling that BOM. Hence it is effectively getting forwarded to json.loads(), which spits out that exception.

The solution, as the error message suggests, is to decode the response using utf-8-sig (i.e., utf-8 with signature) instead of the default utf-8.

I tested that by modifying this line of code:

github.com

home-assistant/core/blob/9011a54e7f0c8b3a10fc648b9a2bdd8660efb42b/homeassistant/components/rest/data.py#L64


try:
    response = await self._async_client.request(
        self._method,
        self._resource,
        headers=self._headers,
        params=self._params,
        auth=self._auth,
        data=self._request_data,
        timeout=self._timeout,
    )
    self.data = response.text
    self.headers = response.headers
except httpx.RequestError as ex:
    _LOGGER.error("Error fetching data: %s failed with %s", self._resource, ex)
    self.data = None
    self.headers = None

to this:

    self.data = response.content.decode('utf-8-sig')

That solved the problem.

Now the big question is, what is the best way to solve this in the “production” code. I’ll go ahead and open an issue. Maybe the person who maintains the REST integration will have a good idea of how to handle this.

pnbruckner · March 17, 2021, 2:56pm

See Issue #48032.

robotek · March 18, 2021, 7:10am

Thank you Phil!

I really hoped that there is something wrong with my config, but I guess I will have to wait for a bugfix

robotek · March 18, 2021, 11:31am

So I created a simple Flask API hosted on Heroku to decode JSON and forward it to HA and now I’m able to see data as a sensor’s attributes.

But since there is an array of items only attributes of the first one are shown. Of course that’s because I am pointing to the first item with this config:

json_attributes_path: "$.Izklopi.Izklop[0]"

How can I obtain data if I change it so that it points to the entire array?

json_attributes_path: "$.Izklopi.Izklop"

Thanks!