How to scrape data from an xml?

petro · November 22, 2023, 2:20pm

- resource: http://10.0.0.13/measurements.xml
  scan_interval: 10
  sensor:
    name: "AC Power"
    unit_of_measurement: "W"
    json_attributes_path: $.root.Device.Measurements.Measurement[2]
    json_attributes:
      - @Value
    value_template: "{{ value_json.root.Device.Measurements.Measurement[2]['@Value'] }}"

MichaelNeys · November 22, 2023, 2:31pm

Thanks a lot!
Had to remove the @ but it finally works, now I can start adding it to my production

petro · November 22, 2023, 2:32pm

just make sure you add state_class and device_class if you plan to use the energy tab

MichaelNeys · November 22, 2023, 3:19pm

So now I have this:

- resource: http://10.0.0.13/measurements.xml
  scan_interval: 2
  sensor:
    name: "AC Power"
    unit_of_measurement: "W"
    icon: 'mdi:solar-power'
    json_attributes_path: $.root.Device.Measurements.Measurement[2]
    json_attributes:
      - Value
    value_template: "{{ value_json.root.Device.Measurements.Measurement[2]['@Value'] }}"
    device_class: energy
    state_class: measurement

it won’t show up in de energy dashboard, don’t know what I’m doing wrong…

petro · November 22, 2023, 3:28pm

refresh the page and it should be available.

Just make sure the state is a number, if not, you might need to remove the @ in the value_template

MichaelNeys · November 22, 2023, 3:37pm

How do i make sure it’s a number because when I remove the @ in value_template the sensor just doesn’t appear, even in Developer tools → states

petro · November 22, 2023, 3:40pm

just look at the state of the entity, if it’s a number then it’s a number

MichaelNeys · November 22, 2023, 3:48pm

It’s showing a number but still won’t show up in the solar production

petro · November 22, 2023, 3:50pm

energy units are not W. Energy is going to be Wh or kWh

MichaelNeys · November 22, 2023, 4:22pm

It finally showed up and added it, unfortunately the sun is gone now but the Wh did it.
Huge thanks!

NaitsirhcNL · February 23, 2024, 7:46pm

Hi,

First of all my apologies, to hijack this topic.
This is my first post over here and I’m struggling with yaml.
My goal is to get input from a steca.
I have made an stecagrid.yaml (also included it in configurations.yaml) and copied the solution of MichaelNeys.
Developer tools showed me a configuration warning:

Integration error: stecagrid - Integration ‘stecagrid’ not found.

All right,
lets copy it into sensors.yaml.
Then developer tools showed me:

Invalid config for ‘sensor’ at sensors.yaml, line 1: required key ‘platform’ not provided.

Could anyone be so kind to tell me what I’m doing wrong?
Many thanks.

I’m sure that a solution will help me to create much more sensors😁

Probably good to tell, is the fact I’m running
Home Assistant Core Update 2024.2.2

MichaelNeys · February 23, 2024, 8:51pm

Hi,

Are you able to see the data of the inverter when visiting its IP with /measurements.xml behind it? (eg. http://10.0.0.13/measurements.xml)

In my config file I have rest: !include rest.yaml and this is inside my rest.yaml file:

- resource: http://10.0.0.13/measurements.xml
  scan_interval: 2
  sensor:
    - name: "Solar AC Voltage"
      unique_id: solar_ac_voltage
      unit_of_measurement: "V"
      json_attributes_path: $.root.Device.Measurements.Measurement[0]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[0]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar AC Current"
      unique_id: solar_ac_current
      unit_of_measurement: "A"
      json_attributes_path: $.root.Device.Measurements.Measurement[1]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[1]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar AC Power"
      unique_id: solar_ac_power
      unit_of_measurement: "W"
      icon: 'mdi:solar-power'
      json_attributes_path: $.root.Device.Measurements.Measurement[2]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[2]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar AC Power fast"
      unique_id: solar_ac_power_fast
      unit_of_measurement: "W"
      icon: 'mdi:solar-power'
      json_attributes_path: $.root.Device.Measurements.Measurement[3]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[3]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar AC Frequency"
      unique_id: solar_ac_frequency
      unit_of_measurement: "Hz"
      json_attributes_path: $.root.Device.Measurements.Measurement[4]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[4]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar DC Voltage1"
      unique_id: solar_dc_voltage1
      unit_of_measurement: "V"
      json_attributes_path: $.root.Device.Measurements.Measurement[5]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[5]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar DC Voltage2"
      unique_id: solar_dc_voltage2
      unit_of_measurement: "V"
      json_attributes_path: $.root.Device.Measurements.Measurement[6]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[6]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar DC Current1"
      unique_id: solar_dc_current1
      unit_of_measurement: "A"
      json_attributes_path: $.root.Device.Measurements.Measurement[7]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[7]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Solar DC Current2"
      unique_id: solar_dc_current2
      unit_of_measurement: "A"
      json_attributes_path: $.root.Device.Measurements.Measurement[8]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[8]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "LINK Voltage"
      unique_id: solar_link_voltage
      unit_of_measurement: "V"
      json_attributes_path: $.root.Device.Measurements.Measurement[9]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[9]['@Value'] }}"
      device_class: energy
      state_class: measurement
    - name: "Steca Omvormer Derating"
      unique_id: omvormer_derating
      unit_of_measurement: "%"
      icon: 'mdi:solar-panel'
      json_attributes_path: $.root.Device.Measurements.Measurement[14]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[14]['@Value'] }}"
      device_class: energy
      state_class: measurement

It’s possible the json_attrivutes_path will be different for you if you have a different model. You can see my XML in my first post on this thread.

My scan_interval is very frequently but you can edit this if you want.
I was struggling with this a very long time too because when I begun with Home-Assistant I didn’t work with rest and hadn’t much coding experience. Well next year I’ll be studying Informatics at uni just because I one day decided to start with this simple things lol.

Make some changes to the rest.yaml so it fits for your sensors. Like to hear when it doesn’t work for you!

MichaelNeys · February 23, 2024, 8:53pm

Do you have any suggestions on how I could make this value into a Wh for the production in HA because now It’s just W and the values are wrong with what I use… Plus the energy dashboard also needed a total increasing or am I wrong?

petro · February 23, 2024, 9:12pm

use the reiman sum integration to turn it into Wh or kWh

MichaelNeys · February 23, 2024, 9:30pm

Saw that somewhere else to but it’s still weird, should I maybe use left or right?

petro · February 23, 2024, 9:32pm

use whatever works best for you

NaitsirhcNL · March 24, 2024, 8:26pm

Hi,

Back again. I have now a sensor which will give me the output of my device. Unfortunatly the sensor has to be reloaded every morning, since the output is 0 during the night. To automate such a thing I have created a sensor (ESPHome) with a photoresistor which will trigger the reloading when the lux output reaches a chosen value for over 5 minutes.
So far so good.
To see the output of the day, I create a helper with a Riemann sum, but thats not very reliable. The xml output will also not return the total output of a day.
I have now done some research and have seen that the total output of the day can be seen in the following link
http://steca.ip/gen.yield.day.chart.js (there is also a month, a year and a total )

Is it possible to scrape the right info to create some sensors or charts or something like that?

My coding skills are minimal, so I don’t have a clue…

MichaelNeys · April 5, 2024, 4:09am

The link you provided did not work for me. I just use the data from the XML and register it every 2 seconds, it wil not be exactly right but 2 seconds is already pretty pricise and create a Riemannsum with a helper.

It’s a very clever solution with the lux-sensor although I don’t understand why it wouldn’t work after the value was 0. I personally don’t know how HA handles it but I think it maybe could have something to do with device class of the sensor?

NaitsirhcNL · April 11, 2024, 6:02pm

Hi Michael

Been very busy so excuse me for the late reply. I don’t know either why my sensor isn’t starting after 0 output, but that work around works
Perhaps we have different inverters i’ve got a StecaGrid 5503
http://10.0.0.xxx/gen.yield.day.chart.js is working for me and creates:

var chartData =
{
"labels":
[
"00:00", "", "", "", "", "",
"01:00", "", "", "", "", "",
"02:00", "", "", "", "", "",
"03:00", "", "", "", "", "",
"04:00", "", "", "", "", "",
"05:00", "", "", "", "", "",
"06:00", "", "", "", "", "",
"07:00", "", "", "", "", "",
"08:00", "", "", "", "", "",
"09:00", "", "", "", "", "",
"10:00", "", "", "", "", "",
"11:00", "", "", "", "", "",
"12:00", "", "", "", "", "",
"13:00", "", "", "", "", "",
"14:00", "", "", "", "", "",
"15:00", "", "", "", "", "",
"16:00", "", "", "", "", "",
"17:00", "", "", "", "", "",
"18:00", "", "", "", "", "",
"19:00", "", "", "", "", "",
"20:00", "", "", "", "", "",
"21:00", "", "", "", "", "",
"22:00", "", "", "", "", "",
"23:00", "", "", "", "", ""
],
"datasets":
[
{
"strokeColor": style_strokecolor.color,
"data": [
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,12,
36,30,42,66,60,66,102,186,162,210,258,282,384,558,534,
408,384,306,600,618,750,1086,1530,1500,1716,1788,1428,2298,2478,1008,
732,456,444,444,438,468,576,900,732,630,582,732,930,726,684,
876,1170,1218,1134,624,426,378,462,606,480,396,330,312,300,354,
288,252,366,300,222,216,174,222,186,132,102,42,18,6,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0]
}
]
}
var max = 5750;
var steps = 23;
var input = document.getElementById("inputId");
input.setAttribute("min",   "2024-03-12");
input.setAttribute("max",   "2024-04-11");
input.setAttribute("value", "2024-04-11");
document.getElementById("labelValueId").innerHTML = "   7.030kWh 11.04.2024";
document.getElementById("buttonPrevId").disabled  = false;
document.getElementById("buttonNextId").disabled  = true;
var myLine = new Chart(document.getElementById("canvasId")
.getContext("2d"))
.Line(chartData,
{
bla bla

so scraping blabla.inner HTML would be nice…

MichaelNeys · April 11, 2024, 10:02pm

What do you want to scrape in here? Is it to replace your workaround?
This is all that I have to scrape my wattage out of the XML, I did not need any more info from my inverter:

    - name: "Solar AC Power"
      unique_id: solar_ac_power
      unit_of_measurement: "W"
      icon: 'mdi:solar-power'
      json_attributes_path: $.root.Device.Measurements.Measurement[2]
      json_attributes:
        - Value
      value_template: "{{ value_json.root.Device.Measurements.Measurement[2]['@Value'] }}"
      device_class: energy
      state_class: measurement

If there is no data provided at night by my inverter the sensor in Home-Assistant just shows 0, I did not have to configure anything for that. The ‘0’ is also 1 measurement so not every 2 seconds, when the inverter starts when there is sun again the scrapper automatically begins. So sorry, I don’t have a clue for that.

I have a different inverter btw, the StecaGrid 3611.