My custom sensor won't fetch the arguments from configuration.yaml

hulkhaugen · March 6, 2019, 5:14am

First of all, I don’t know much python, but found this to be an interesting project for me. I’m scraping finance data from Morningstar.no, and I got the sensor working. However, I wan’t to be able to customize the sensor data sources so that I can use the component several times for different stocks/funds. I can’t seem to get the component read the configuration.yaml, it just uses the default data from within the component. Here is what I got so far:

"""Data from Morningstar.no"""
import asyncio
from datetime import timedelta
import logging

import aiohttp
import async_timeout
import voluptuous as vol

from homeassistant.components.sensor import PLATFORM_SCHEMA
from homeassistant.const import ATTR_ATTRIBUTION, CONF_NAME, CONF_RESOURCE
from homeassistant.helpers.aiohttp_client import async_get_clientsession
import homeassistant.helpers.config_validation as cv
from homeassistant.helpers.entity import Entity

REQUIREMENTS = ['beautifulsoup4==4.6.3']

_LOGGER = logging.getLogger(__name__)

CONF_ATTRIBUTION = "Data provided by Morningstar.no"

DEFAULT_NAME = 'Morningstar'

DEFAULT_RESOURCE = 'http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000ZFG1&tab=1'

SCAN_INTERVAL = timedelta(minutes=5)

PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
    vol.Required(CONF_NAME): cv.string,
    vol.Required(CONF_RESOURCE): cv.string,
})


async def async_setup_platform(hass, config, async_add_entities, discovery_info=None):
    """Set up the Morningstar.no sensor."""
    name = config.get(CONF_NAME)
    session = async_get_clientsession(hass)
    resource = config.get(CONF_RESOURCE)
    async_add_entities([
        MorningstarSensor(session, name, resource)], True)


class MorningstarSensor(Entity):
    """Representation of Morningstar.no sensor."""

    def __init__(self, config, session, name, resource):
        """Initialize Morningstar.no sensor."""
        self._name = config[CONF_NAME]
		self._resource = config[CONF_RESOURCE]
        self._state = None
        self._session = session
        self._unit_of_measurement = '%'
        self._attrs = {ATTR_ATTRIBUTION: CONF_ATTRIBUTION}

    @property
    def name(self):
        """Return the name of the sensor."""
        return self._name

    @property
    def resource(self):
        """Return the url of the sensor."""
        return self._resource
		


    @property
    def unit_of_measurement(self):
        """Return the unit the value is expressed in."""
        return self._unit_of_measurement

    @property
    def state(self):
        """Return the state of the device."""
        return self._state

    @property
    def device_state_attributes(self):
        """Return the device state attributes."""
        return self._attrs

    async def async_update(self):
        """Get the latest data from the source and updates the state."""
        from bs4 import BeautifulSoup

        try:
            with async_timeout.timeout(10, loop=self.hass.loop):
                response = await self._session.get(CONF_RESOURCE)

            _LOGGER.debug(
                "Response from Morningstar.no: %s", response.status)
            data = await response.text()
            _LOGGER.debug(data)
        except (asyncio.TimeoutError, aiohttp.ClientError):
            _LOGGER.error("Can not load data from Morningstar.no")
            return

        raw_data = BeautifulSoup(data, 'html.parser')

        try:
            value = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(8)")[0].text
            self._attrs['1 dag'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(8)")[0].text
            self._attrs['1 uke'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(12)")[0].text
            self._attrs['1 måned'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(16)")[0].text
            self._attrs['3 måneder'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(20)")[0].text
            self._attrs['6 måneder'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(24)")[0].text
            self._attrs['I år'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(28)")[0].text
            self._attrs['1 år'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(32)")[0].text
            self._attrs['3 år annualisert'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(36)")[0].text
            self._attrs['5 år annualisert'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(40)")[0].text
            self._attrs['10 år annualisert'] = raw_data.select(
                "#returnsTrailingDiv td:nth-of-type(44)")[0].text
        except IndexError:
            _LOGGER.error("Unable to extract data from Morningstar.no")
            return

        self._state = value

So what I want first of all is for the sensor to rename sensor.morningstar to whatever I type in as name in configuration.yaml. And I also wan’t to be able to specify the url (using resources above) to the stock or fund I’m tracking.
I haven’t started on this yet, but I wan’t to get data from 2 separate urls, but the difference is only the additional &tab=1 at the end, so that could be hardcoded in the component and left out of the configuration.yaml. Don’t know where to start here. It also includes stripping down the currency NOK ahead of the value I’m intending to scrape.
I wan’t an template icon using mdi:trending-up/down/neutral, depending on the last days result. And I wan’t to add a %-symbol after all the attribute data values.

olloe · April 11, 2019, 10:44am

Hey! Did you ever make this work? I have some Norwegian funds i would like to add to HA.

hulkhaugen · April 11, 2019, 3:11pm

Hi! I haven’t had the time to work on it yet, but I intend to do during Easter holidays, and I’m positive I will make it work eventually, shouldn’t be to hard really.

olloe · April 12, 2019, 6:44am

Sweet! Would appreciate it if you share it
I really just want todays price (and maybe the change) .

hulkhaugen · April 12, 2019, 7:54am

Yes, I’ll share it when it’s done. But what you want, you can easily do with the Scrape sensor:

sensor: 
  - platform: scrape
    name: 'DNB Norge Indeks 1 dag'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORR&tab=1
    select: '#returnsTrailingDiv tr:nth-of-type(3) td:nth-of-type(2)'
    unit_of_measurement: "%"

  - platform: scrape
    name: 'DNB Norge Indeks kurs'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORR
    select: '#overviewQuickstatsDiv tr:nth-of-type(2) td:nth-of-type(3)'

olloe · April 14, 2019, 3:07pm

Awesome! Thanks man! Didn’t know about that, will check it out!

hulkhaugen · April 15, 2019, 6:18pm

I fine tuned the setup a little and added more examples, as I’m changing from command_line sensor to scrape (for now):

sensor: 
  - platform: scrape
    name: 'DNB Global Indeks endring'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORS
    select: '#overviewQuickstatsDiv tr:nth-of-type(3) td:nth-of-type(3)'
    value_template: '{{ ((value.split("%")[0]) | replace (",", ".")) }}'
    unit_of_measurement: "%"

  - platform: scrape
    name: 'DNB Health Care R endring'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000ZFFV
    select: '#overviewQuickstatsDiv tr:nth-of-type(3) td:nth-of-type(3)'
    value_template: '{{ ((value.split("%")[0]) | replace (",", ".")) }}'
    unit_of_measurement: "%"

  - platform: scrape
    name: 'DNB Norge Indeks endring'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORR
    select: '#overviewQuickstatsDiv tr:nth-of-type(3) td:nth-of-type(3)'
    value_template: '{{ ((value.split("%")[0]) | replace (",", ".")) }}'
    unit_of_measurement: "%"

  - platform: scrape
    name: 'DNB Teknologi R endring'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000ZFG1
    select: '#overviewQuickstatsDiv tr:nth-of-type(3) td:nth-of-type(3)'
    value_template: '{{ ((value.split("%")[0]) | replace (",", ".")) }}'
    unit_of_measurement: "%"

  - platform: scrape
    name: 'DNB Global Indeks kurs'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORS
    select: '#overviewQuickstatsDiv tr:nth-of-type(2) td:nth-of-type(3)'
    value_template: '{{ ((value.split("\xa0")[1]) | replace (",", ".")) }}'
    unit_of_measurement: "kr"

  - platform: scrape
    name: 'DNB Health Care R kurs'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000ZFFV
    select: '#overviewQuickstatsDiv tr:nth-of-type(2) td:nth-of-type(3)'
    value_template: '{{ ((value.split("\xa0")[1]) | replace (",", ".")) }}'
    unit_of_measurement: "kr"

  - platform: scrape
    name: 'DNB Norge Indeks kurs'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORR
    select: '#overviewQuickstatsDiv tr:nth-of-type(2) td:nth-of-type(3)'
    value_template: '{{ ((value.split("\xa0")[1]) | replace (",", ".")) }}'
    unit_of_measurement: "kr"

  - platform: scrape
    name: 'DNB Teknologi R kurs'
    resource: http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000ZFG1
    select: '#overviewQuickstatsDiv tr:nth-of-type(2) td:nth-of-type(3)'
    value_template: '{{ ((value.split("\xa0")[1]) | replace (",", ".")) }}'
    unit_of_measurement: "kr"

olloe · April 20, 2019, 8:44pm

Nice! Thanks!
I was using template sensors to make the value templates work.

Have a question tho… Trying to use this (xxx = shares):
value_template: '{{ (((value.split("\xa0")[1]) | replace (",", ".")) | float) * XXX | round }}'

I don’t think I know how to use “round”. I want to round and remove the decimals after the multiplication is done. It worked then I used the extra value template, but within the scrape it doesn’t.

Do you have an example for stocks too? I’ve made it work, but I need an extra value template.

hulkhaugen · April 20, 2019, 8:57pm

It will probably work if you remove the replace function. That makes it non-numeric. You can put the replace after calculation and/or round function…

olloe · February 13, 2020, 11:55am

Did you ever finish the python project?

hulkhaugen · February 14, 2020, 11:37am

No, now i’m using Node-RED to scrape the data i want and get multiple attributes for the entities, very happy with that.

olloe · February 15, 2020, 5:44pm

Didn’t know you could do that! Would you mind giving me some startup help?

I’ve now installed the node-red addon and the Companion Component for node-red-contrib-home-assistant-websocket.

I use a http request with GET and url https://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORS

The I have a HTML node and don’t know what to use as a selector. I have tried a few without any result. With scrape I used ‘#overviewQuickstatsDiv tr:nth-of-type(2) td:nth-of-type(3)’

At last I have the home assistant sensor with state msg.payload.

In HA it updates the name of the sensor, but no state.

What am I doing wrong? How do I see what data the html node actually get?

hulkhaugen · February 15, 2020, 11:58pm

I’m no guru with this, and I got some help from @fame89 over at Discord getting this to work. However, it stopped working just last week, I haven’t had the time to figure out why yet, probably some small changes to the website. Here is the code i use in Node-RED:

The last one, pulling data from lt.morningstar.com (Pension fund) still works. Initially, i used the same flow for multiple funds, and had multiple inject nodes into the same flow sequence. That’t why the unique part of the url’s are in separate inject nodes before the http request nodes. Unfortunately, that ended up getting the data mixed, even with several tweak attempts. In the end, I ended up copy/paste for each sensor, at least that worked reliably for a month or two. If you use this and find the error(s), please share

olloe · February 19, 2020, 6:40pm

Thanks! I’m having trouble getting the selectors to work. Will have to look more into it.

I get the "TypeError: Cannot read property '0' of null"
from node: string/float.

I guess it’s because it’s empty…

olloe · February 19, 2020, 6:51pm

It seems like the response from http://www.morningstar.no/no/funds/snapshot/snapshot.aspx?id=F00000JORS
is:

statusCode: 500
responseUrl: "http://www.morningstar.no/errors/defaulterror.aspx?aspxerrorpath=/virtual/snapshot/snapshot.aspx"

Probably not that weird that selector doesn’t work…

olloe · February 19, 2020, 9:48pm

Got it to work. changed the urls so it matches your pension fund… Did some changes with the format too so you know. You should be able to just use another url and change the names. Thanks for the help!

[{"id":"429a64a4.782a3c","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"de463362.b4b63","type":"http request","z":"429a64a4.782a3c","name":"dnb_global_indeks_a","method":"GET","ret":"txt","paytoqs":false,"url":"https://lt.morningstar.com/vnq5t3n66n/snapshot/snapshot.aspx?SecurityToken=F00000JORS","tls":"","persist":false,"proxy":"","authType":"","x":320,"y":280,"wires":[["40e7a5b9.36909c","46b509b0.e2eb48"]]},{"id":"40e7a5b9.36909c","type":"html","z":"429a64a4.782a3c","name":"dato","property":"payload","outproperty":"payload","tag":"#KeyStatsLatestNav th span","ret":"text","as":"single","x":510,"y":260,"wires":[["14efee26.659232"]]},{"id":"46b509b0.e2eb48","type":"html","z":"429a64a4.782a3c","name":"kurs","property":"payload","outproperty":"payload","tag":"#KeyStatsLatestNav td","ret":"text","as":"single","x":510,"y":300,"wires":[["d0998d05.b5ede"]]},{"id":"14efee26.659232","type":"change","z":"429a64a4.782a3c","name":"date","rules":[{"t":"set","p":"topic","pt":"msg","to":"date","tot":"str"},{"t":"move","p":"payload.0","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":260,"wires":[["a5684cb7.e9262"]]},{"id":"d0998d05.b5ede","type":"change","z":"429a64a4.782a3c","name":"price","rules":[{"t":"set","p":"topic","pt":"msg","to":"price","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":300,"wires":[["2e36f9d5.53baf6"]]},{"id":"2e36f9d5.53baf6","type":"function","z":"429a64a4.782a3c","name":"string/float","func":"msg.payload = /(?:[^A-z]+)/.exec(msg.payload)[0].replace(\",\",\".\").replace(/\\s+/g, \"\");\nmsg.payload = parseFloat(msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":770,"y":300,"wires":[["a5684cb7.e9262"]]},{"id":"a5684cb7.e9262","type":"join","z":"429a64a4.782a3c","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"100","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"num","reduceFixup":"","x":930,"y":260,"wires":[["db083c5c.71787"]]},{"id":"db083c5c.71787","type":"function","z":"429a64a4.782a3c","name":"format","func":"msg.payload = {\n \"data\": {\n \"state\": parseFloat(msg.payload.price * 10).toFixed(0),\n \"attributes\": {\n \"Kurs\": msg.payload.price,\n \"Dato\": msg.payload.date,\n \"Andeler\": 10,\n \"Kjøpt for\": 100,\n \"Totalavgift\": 0.21,\n \"friendly_name\": \"DNB Global Indeks A\",\n \"unit_of_measurement\": \"kr\"\n }\n }\n};\n\nreturn msg;","outputs":1,"noerr":0,"x":1050,"y":260,"wires":[["1c620d9a.f025a2"]]},{"id":"1c620d9a.f025a2","type":"ha-api","z":"429a64a4.782a3c","name":"sensor","server":"f680d360.53645","debugenabled":false,"protocol":"http","method":"post","path":"/api/states/sensor.dnb_global_indeks_a","data":"","dataType":"json","location":"payload","locationType":"msg","responseType":"json","x":1170,"y":260,"wires":[[]]},{"id":"6a5e4d5a.fde004","type":"inject","z":"429a64a4.782a3c","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":280,"wires":[["de463362.b4b63"]]},{"id":"f680d360.53645","type":"server","z":"","name":"Home Assistant","legacy":false,"hassio":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":false,"cacheJson":true}]

hulkhaugen · February 22, 2020, 4:11pm

That’s awesome! I wonder what happed to the good old url, if there’s some sort of scraper block function? With this new url, we’re missing out on 1-day change %. I gave a quick shoot at using oslobors.no instead of morningstar.no, but i got the same reply. I haven’t had time to dig into it as I’m very busy these days, and my Node-RED skills are on the n00b end of the scale.

olloe · February 22, 2020, 7:53pm

Oh, didnt know there was a delay…
I have no idea how to fix that, but will try to look i to it when I have the time. Appreciate it if you tell me if you find a better way

hulkhaugen · February 26, 2020, 1:50pm

I’ve changed nothing, but today the old sensor started working again. Weird, but i’m ok with that. Don’t know if it was a Node-RED issue or Morningstar issue, but for as long as it’s working, i’m happy.

hulkhaugen · March 13, 2021, 10:27pm

Not sure if you’re still interested, but I finally finished this custom component which uses the Oslo Børs API instead of scraping: https://github.com/hulkhaugen/hass_custom_components/tree/main/ob_fond
At the time of writing, I’ve started a more ambitious project which will use the same data as well as manual input to calculate personal accounts.