Scrape sensor for SMA Sunny Portal

[Marius82 (Marius)]

Hello,
I want to read the data value from this html code:
<span class="mainValueAmount" data-peak="19980" data-value="123" data-timestamp="2016-11-23T17:00:00">123</span>

The result should be 123.
The website is:
https://www.sunnyportal.com/Templates/PublicPage.aspx?page=7431e23c-a310-4fdf-99ec-e31db735d65f

what should I type in for select??
This is my try:

  - platform: scrape
    resource: https://www.sunnyportal.com/Templates/PublicPage.aspx?page=7431e23c-a310-4fdf-99ec-e31db735d65f
    name: PV Anlage
    select: ".mainValueAmount"

But I get only as a result a “-”

Can someone help me?

Thanks, Marius

[hborawski (Harris)]

The 123 value looks to be loaded asynchronously, the dash is a placeholder until the fetch returns the real value, so the sensor parses it before the real data gets loaded.

[Marius82 (Marius)]

Hmmm, so it is not possible to get the value in Home Assistant?

[Marius82 (Marius)]

Hi,
and is it also not possible to get the “2482” value out of this html code from the same website:
<span id="ctl00_ContentPlaceHolder1_PublicPagePlaceholder1_PageUserControl_ctl00_PublicPageLoadFixPage_energyYieldWidget_energyYieldValue" class="mainValueAmount simpleTextFit">2482</span>

Thank you

[Felix]

@Marius82> did you get this to work Marius? Would in be very interested in your solution. Thanks!

[Marius82 (Marius)]

Here my solution:

  - platform: scrape
    resource: "https://www.sunnyportal.com/Templates/PublicPage.aspx?page=7431e23c-a310-4fdf-99ec-e31db735d65f"
    name: pv_einheit
    select: '#ctl00_ContentPlaceHolder1_PublicPagePlaceholder1_PageUserControl_ctl00_PublicPageLoadFixPage_energyYieldWidget_energyYieldUnit'
  
  
  - platform: scrape
    resource: "https://www.sunnyportal.com/Templates/PublicPage.aspx?page=7431e23c-a310-4fdf-99ec-e31db735d65f"
    name: pv_periode
    select: '#ctl00_ContentPlaceHolder1_PublicPagePlaceholder1_PageUserControl_ctl00_PublicPageLoadFixPage_energyYieldWidget_energyYieldPeriodTitle'
      
  

  - platform: scrape
    resource: "https://www.sunnyportal.com/Templates/PublicPage.aspx?page=7431e23c-a310-4fdf-99ec-e31db735d65f"
    name: pv_wert
    select: '#ctl00_ContentPlaceHolder1_PublicPagePlaceholder1_PageUserControl_ctl00_PublicPageLoadFixPage_energyYieldWidget_energyYieldValue'
    value_template: '{% if is_state("sensor.pv_einheit", "Wh") %}{{ value | float / 1000 }}{% else %}{{ value | float }}{% endif %}'
    #unit_of_measurement: 'kW/h'
  
  - platform: template
    sensors:
      pv_energie:
        value_template: '{%- if is_state("sensor.pv_periode", "Today") and (states.sensor.pv_wert.state | float) < 150 -%}{{ states.sensor.pv_wert.state | round(1) }}{% else %}0{% endif %}'
        friendly_name: 'Photovoltaik'
        unit_of_measurement: 'kW/h'

[Felix]

perfect! thanks!

[Faecon (Jo)]

Thanks @Marius82

[monkey-house (Ced)]

Great stuff here. Thanks for sharing.

My Sunny Portal page cannot currently be made publicly available. Does anyone know what the code should look like for a non-publicly available page of the Sunny Portal site. I tried the following with no luck:

- platform: scrape
  resource: "https://www.sunnyportal.com/FixedPages/Dashboard.aspx"
  username: "user"
  password: "pass"
  name: sma yield
  select: "#ctl00_ContentPlaceHolder1_UserControlShowDashboard1_energyYieldWidget_energyYieldValue"
  unit_of_measurement: kWh

Here is the error message in HA:

scrape: Error on device update!
Traceback (most recent call last):
  File "/usr/src/app/homeassistant/helpers/entity_platform.py", line 244, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/app/homeassistant/helpers/entity.py", line 327, in async_device_update
    yield from self.hass.async_add_job(self.update)
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/app/homeassistant/components/sensor/scrape.py", line 120, in update
    value = raw_data.select(self._select)[0].text
IndexError: list index out of range

Help would be much appreciated.

[Felix]

Hi There, did you ever get this to work?

[Freshhat]

@Felix this is my config in sensors.yaml (just replace the ID from your sunnyportal.com public page)

# Webscrapping for SMA Inverter values
- platform: scrape
  resource: "https://www.sunnyportal.com/Templates/PublicPage.aspx?page=>>PublicID<<"
  name: pv_einheit
  select: '#ctl00_ContentPlaceHolder1_PublicPagePlaceholder1_PageUserControl_ctl00_PublicPageLoadFixPage_energyYieldWidget_energyYieldUnit'

- platform: scrape
  resource: "https://www.sunnyportal.com/Templates/PublicPage.aspx?page=>>PublicID<<"
  name: pv_periode
  select: '#ctl00_ContentPlaceHolder1_PublicPagePlaceholder1_PageUserControl_ctl00_PublicPageLoadFixPage_energyYieldWidget_energyYieldPeriodTitle'

- platform: scrape
  resource: "https://www.sunnyportal.com/Templates/PublicPage.aspx?page=>>PublicID<<"
  name: pv_wert
  select: '#ctl00_ContentPlaceHolder1_PublicPagePlaceholder1_PageUserControl_ctl00_PublicPageLoadFixPage_energyYieldWidget_energyYieldValue'
  value_template: '{% if is_state("sensor.pv_einheit", "Wh") %}{{ value | float / 1000 }}{% else %}{{ value | float }}{% endif %}'
#unit_of_measurement: 'kW/h'

- platform: template
  sensors:
pv_energie:
  value_template: '{%- if is_state("sensor.pv_periode", "Today") and (states.sensor.pv_wert.state | float) < 150 -%}{{ states.sensor.pv_wert.state | round(1) }}{% else %}0{% endif %}'
  friendly_name: 'SMA'
  unit_of_measurement: 'kW/h'

Maybe you need to adjust the formatting. But this one works for me.

[MYeager1967 (Michael Yeager)]

@Marius82

PV MK PV System Overview

How do I find the “public page” address for my instance? This looks like a solution to my desire to extract certain information about my solar production but I don’t know the public page address. Can this be used with the regular log in site as well???

[Marius82 (Marius)]

After you registered your pv system to Sunnyportal.com you will find the public page config in the configuration menu-> PV System Presentation.

[MYeager1967 (Michael Yeager)]

I didn’t register my plant. The installer did…

[poggialb]

I tried this same path, but the quality of the data I could retrieve wasn’t great.
See here for alternative approach using SBFspot

Integration with SMA Solar STP 10000TL-20 via SBFspot

I have spent few days trying to integrate my SMA inverter STP 10000TL-20 with HA and finally today I managed to pull all the data I needed. The possible solutions I have found looking online didn’t quite work with my inverter, until I found this fantastic tool https://github.com/SBFspot/sbfspot-config Let me walk you through the options I found in the forum, and the approach + configuration I used. 1: Direct integration using the SMA integration. In my case this doesn’t work, as the inverter…

[psl3 (Paul)]

I have a 3600TL-21 with no webconnect and cannot get modbus working so have tried the ‘scrape’ method - with parts of Freshhats code I have got the total PV kWh out. But I really need the realtime Watts value that his code does not seem to provide me.

The value seems to be in the following webpage code (current value is ‘0’). I would be really grateful if anyone could help me get the Home Assistant scrape parameters right to get this value out?
Thanks in advance.

[psl3 (Paul)]

Found same issue as Marius82 - the webpage has a “-” in place before it updates with the real number and the scrape does not have any delay capability so only reads the “-” – hence does not work

[tropfen]

Hello,

i do know that this tread is pretty old. But may be someone can help. I do use the following confitguration in the configuration.yaml:

  - platform: scrape 
    resource: "https://sunnyportal.com/FixedPages/Dashboard.aspx" 
    username: !secret SMA_username
    password: !secret SMA_passwort
    name: sma yield
    select: "#ctl00_ContentPlaceHolder1_UserControlShowDashboard1_energyYieldWidget_energyYieldValue"
    unit_of_measurement: kWh    

All i get is:

2021-09-21 18:05:21 ERROR (MainThread) [homeassistant.components.scrape.sensor] Unable to extract data from HTML for sma_yield

Does anyone has a working solution for this?

Thanks in advance
Frank

---

i have a HA OS based installation on a raspi 4

[tropfen]

Hello,
have solved the problem. The web address was wrong. It does not go the general web address. In sunnyportal the specific public page must be activated for the user. See above.

Starting with: https://www.sunnyportal.com/Templates/PublicPage.aspx?page=7431e23c…

Frank

[thw (Tom)]

Hi, thanks for the very helpful configuration examples.
I managed to scrape the sensors described by @Marius82
Now I wonder if it’s possible to get the sensor of “.mainValueAmount” somehow.
Any ideas?