Programing Python sensor platform. Need help finalizing

Tomahawk · September 18, 2018, 2:21pm

I like to think I am 80% there, but I need someone with the knowhow to finalize the code.

The URL/API gets a Json dataset, table. And for every row in the dataset, I want to create a sensor, and update the state and attributes at the selected time interval.

So as a N00B, trying to do something way over my skill level, - which is still on the “Hello world” level, did set up Windows 10, with Visual Studio and Python. Managed to get the data set trough the URL/API. Python code for getting dataset:

import requests
import json

url = 'https://api.nilu.no/aq/utd.json?areas=Lillehammer;Oslo&stations=Bankplassen;Manglerud'

myData = json.loads(requests.get(url).text)

for i in range(len(myData)):
    zone         = myData[i]['zone']
    municipality = myData[i]['municipality']
    station      = myData[i]['station']
    component    = myData[i]['component'].replace(".","")
    fromTime     = myData[i]['fromTime']
    toTime       = myData[i]['toTime']
    value        = myData[i]['value']
    unit         = myData[i]['unit']
    index        = myData[i]['index']
    color        = myData[i]['color']
    latitude     = myData[i]['latitude']
    longitude    = myData[i]['longitude']

    
    sensor_name = ("sensor.nilu_" + municipality + "_" + station + "_" + component).lower()
    print(sensor_name,":", value)

    print("ATTRIBUTES:")
    print("zone:",zone)
    print("municipality:",municipality)
    print("station:",station)
    print("unit:",unit)
    print("index:",index)
    print("color:",color)
    print("latitude:",latitude)
    print("longitude:",longitude)
    print()

Json dataset result:

[{
	"zone": "Stor-Oslo",
	"municipality": "Oslo",
	"area": "Oslo",
	"station": "Manglerud",
	"eoi": "NO0071A",
	"component": "PM10",
	"fromTime": "2018-09-18T13:00:00+01:00",
	"toTime": "2018-09-18T14:00:00+01:00",
	"value": 28.38,
	"unit": "µg/m³",
	"index": 1,
	"color": "6ee86e",
	"latitude": 59.898690,
	"longitude": 10.814950
}, {
	"zone": "Øst og Sørlandet",
	"municipality": "Lillehammer",
	"area": "Lillehammer",
	"station": "Bankplassen",
	"eoi": "NO0074A",
	"component": "PM10",
	"fromTime": "2018-09-18T13:00:00+01:00",
	"toTime": "2018-09-18T14:00:00+01:00",
	"value": 21.34,
	"unit": "µg/m³",
	"index": 1,
	"color": "6ee86e",
	"latitude": 61.112880,
	"longitude": 10.464970
},etc.

Running the code, the print out, is what I am trying to do in HA. There is no manipulation of the dataset, so getting this to work, will make a great template for others, who want to develop sensors for HA.

The result:

The data set is air quality measurements, from measuring stations, scattered around Norway. If I can get this sensor working, I´ll try to make sensors for other countries too.

sensor:
  - platform: nilu
    locations:
      - Lillehammer,Bankplassen 
      - Oslo, Manglerud
    scan_interval:
      minutes: 30

If someone out there has the time to finalize my 80% code, I would be a happy camper.

80% NILU air quality code:

Cheers.

w1ll1am23 · September 19, 2018, 12:57am

So I don’t have a lot of time at the moment to work on this, but something could be put together to work pretty easily.

I put together a very minimal “library” that this should be type of logic should be put in. https://github.com/w1ll1am23/pynilu

Take a look at the README and/or the test.py in the src directory. Basically everything in the test.py would be the part that gets added to Home Assistant and the library would get published to pypi and pulled in to Home Assistant.

If you look at the following line in the test.py you will notice that the Station and the Area can be passed in to the creation of the API interface which simulates pulling in those values from Home Assistants .yaml for the sensor setup.

nilu = NiluApiInterface(["Bankplassen"], ["Lillehammer"])

This line could also be

nilu = NiluApiInterface(["Bankplassen", "Manglerud"], ["Lillehammer", "Oslo"])

or whatever stations and areas are supported meaning this would work for all supported countries?

I attempted to look at their site https://www.nilu.no/ but I don’t speak Norwegian? So I can’t find any documentation on the API which should also be read through so proper error handling can happen.

clyra · September 19, 2018, 2:29am

Hi,

I just wrote my first sensor a few days ago which basically do the same: connect to something, scrape and present that as sensor. I don’t pretend to know much about how to write a sensor, and to be true, I copied a lot of chunks from others :-). Anyway… take a look at this.

I guess you can copy a lot of parts of it too. The def setup_platform is where the sensors are created. I wrote a class to hold the sensor values, and another one with the methods to log/download/process the values. Write a class like the UnifiSensorData with your code.The update method doing most of the work. Change the UnifiSensor class to represent your sensor. If I got it right, “value” should be your “state” and the rest the attributes of your sensor.

Tomahawk · September 29, 2018, 11:05am

@w1ll1am23 @clyra
Thanks for helping. I must admit that what I am trying to accomplish is way over my skill level. Too much intricate coding going on,since I am trying to iterate a list and creating a sensor for every record. I have to put this to rest…

clyra · September 29, 2018, 9:51pm

We should make a camper happy. Right now my code is ugly and fragile, but it seem to work :-). Put this in your custom_component folder and configure like this:

- platform: nilu
  region: Lillehammer,Bankplassen

- platform: nilu
  region: Oslo,Manglerud

One entry per station. The code will find which component each station monitors an will create the sensors for each one.

Now the bad things: Scan interval is hardcoded and on update the code is requesting the same thing 3 times (update function should test if the data is fresh or not before request it again). I’m not checking for errors and so on… So, this is the 0.0.1 version :-).

clyra · September 29, 2018, 9:56pm

@Tomahawk

Looks like this now:

Tomahawk · September 30, 2018, 9:01am

@clyra Thank you so much! Exactly what I was hoping for.

Nilu updates the station data once an hour. Hardcoding the data fetch should maybe be every 15 or 30 minutes(?)

clyra · September 30, 2018, 11:04am

I left the scan interval at 5 minutes. You may change that, but it will not affect nilu nor your system if you use a smaller interval. Remember that if you use 30 minutes you may run in a situation like this: HA grabs the information at, say, 10h59m then the site updates at 11h00m, and HA will only update again at 11h29m, so about half the time you have a old value.

Tomahawk · October 1, 2018, 9:36am

Installed the Nilu code, in my HassOS,0.79.1 system and getting errors. Any ideas?

Here is my error log

clyra · October 1, 2018, 12:21pm

Your log has a lot of errors not related to the nilu. The relevant one is only this:
INFO:homeassistant.util.package:Attempting install of colorlog==3.1.4 Testing configuration at /config Failed config General Errors: - Platform not found: sensor.nilu Successful config (partial)
11:29 components/hassio/init.py (ERROR)
Unable to find component sensor.nilu

Is the nilu.py at ~/.homeassistant/custom_components/sensor/nilu.py ?

Tomahawk · October 2, 2018, 11:24am

Updated to Hassio 0.79.2, and your Nilu code works.

All smiles!

I am trying to add some functionality without luck.

The friendly name with space instead of "_".
Format the value with 2 decimals.
Add an extra sensor for every location called "sensor.XXX_YYY_status", which have the highest index value of all the sensors for the location. Think my code is almost there, except it fails. I have commented out my code so its easy to see where I am messing about. Off course my code works when testing in Visual Studio, but not when adapting to HA.
Any hints?

clyra · October 2, 2018, 12:29pm

Sorry… the code is unformatted and it’s difficult to read on the forum. You may attach it to the forum but changing the extension to yaml or pastebin it.

Tomahawk · October 2, 2018, 1:08pm

Updated my previous post with link to my file.

clyra · October 2, 2018, 5:25pm

Your function is wrong… when you use it inside a class the first argument should be “self”, so it should look like as “find_row_status(self, json)”. But even this will not work as you expect. The sensors are created on the the line add_devices([NiluSensor(hass, data, k, name)], True). I will check the friendly name issue.

clyra · October 4, 2018, 2:37am

1, 2 done. 3 soon :-). code on the git now.

clyra · October 4, 2018, 12:24pm

3 done. Check the GitHub plz.

Tomahawk · October 5, 2018, 5:48pm

We are almost there. The *_state sensor do pick up the wrong values. It should pick up values from the sensor with the highest index value. Now it takes the lowest.

The state sensor is for having a badge on the front page, showing the pollution sensor with the most damaging state.

I tried to wrap my head around the *_state value set code, but got a brain freeze.

for i in range(len(myData)):
    component = myData[i]['component'].replace(".","")
    self.attrs[component] = myData[i]
    if len(state_sensor) < 1:
        self.attrs['state'] = myData[i]
    elif myData[i]['value'] > self.attrs['state']['value']:
        self.attrs['state'] = myData[i]

nilu

clyra · October 5, 2018, 6:04pm

hm… I did it based on the one with the highest value (on your image 84.28 > 17.25). Should we use index then?
Changing:
elif myData[i][‘value’] > self.attrs[‘state’][‘value’]:

to:
elif myData[i][‘index’] < self.attrs[‘state’][‘index’]:

Should do it.

Tomahawk · October 8, 2018, 9:22am

Looks like it doesn’t work. I have tried both < and >.

This i unsersatnd,myData[i][‘index’], -gives the value off det records index field.
I dont understand whats happening with:
self.attrs[‘state’] = myData[i] and
self.attrs[‘state’][‘index’]

The sensor.nilu_xxx_yyy_state (maybe the name “sensor.nilu_xxx_yyy_max” is more correct?) should be a copy of the sensor with the highest index attribute. - The index is a scale of the pollution severity. The different sensors PM10/PM25/NO2/… has hits separate scale for severity, the Index display the individual sensor place in this scale.

Index/Explanation/color
1 - Low health risk - #6ee86e
2 - Moderate health risk - #ff9900
3 - Significant health risk - #ff0000
4 - Serious health risk - #990099

I am also trying to add “#” to the color code.
Tried in the method update(self) with this code.
I suspect the “self._state =” should be something else?

color = self._data.attrs[self._component]['color']
if color is None:
    color = STATE_UNKNOWN
    self._attributes = {}
else:
    self._state = "#{0}".format(color)
    self._attributes = self._data.attrs[self._component]

clyra · October 9, 2018, 10:40pm

hm… ok, I was ordering by index and getting the first one. I guess I fixed it, but since it seems that the air quality in Norway is good everywhere I couldn’t find a sensor with a index > 1 . There was a little change in the URL, which I also fixed. New version on github.
State is the value that appears on the sensor, so it doenst make sense to attribute the color to the sensor. There should be a way to use lovelace to do what you want.