Need help with Multiscrape to get values from a WELSERVER in the house

wpaul.johnson · February 23, 2023, 2:24pm

Need some help – something is just not clicking for me. I have WELSERVER in the house monitoring the Heat/AC temperatures. It will make a XML file that is updated with the values every minute or so. I am trying to use multiscrape to get the values into HA, but I can’t wrap my head around the select setting.
I get a “Could not find a tag for given selector” error with this configuration:
I just don’t understand the select portion of the configuration. All the examples that I find don’t explain how they know what to put where; I feel that there is something simple that escapes me.

This is from my configuration.yaml:

- name: WEL scraper
    resource: http://10.10.0.44:5150/data.xml
    scan_interval: 300
    sensor:
      - unique_id: 1st_supply_temp
        name: Supply Temp for 1st floor
        select: "#folder0 > div.opened > div:nth-child(8) > span > span:nth-child(2) > span.html-attribute-value"
``````````````````````````````````````````````````````````````````````````````````

This is the page from that url: 

````````````````````````````````````````````````````````````````````````````````

<devices>

<device>

<name>date</name>

<value>02/22/2023</value>

</device>

<device>

<name>time</name>

<value>15:53:57</value>

</device>

<device>

<name>Basemnt_Temp</name>

<value>73.174995</value>

</device>

<device>

<name>1st_Supply_Air</name>

<value>74.074989</value>

</device>

<device>

<name>1st_Return_Air</name>

<value>73.849998</value>

</device>

<device>

<name>Attic_Temp</name>

<value>80.150001</value>

</device>

<device>

<name>2nd_Supply_Air</name>

<value>54.049999</value>

</device>

<device>

<name>2nd_Return_Air</name>

<value>70.924995</value>

</device>

<device>

<name>deltaT1</name>

<value>-0.218506</value>

</device>

<device>

<name>deltaT2</name>

<value>16.861397</value>

</device>

</devices>

Troon · February 23, 2023, 2:31pm

Welcome .

Scrape is not the right answer if you have XML available. I suspect you were trying to work out a select statement based off your browser’s HTML rendering of the XML file, which is doomed to failure.

Use a REST sensor instead.

Here’s your XML nicely formatted:

<?xml version="1.0"?>
<devices>
  <device>
    <name>date</name>
    <value>02/22/2023</value>
  </device>
  <device>
    <name>time</name>
    <value>15:53:57</value>
  </device>
  <device>
    <name>Basemnt_Temp</name>
    <value>73.174995</value>
  </device>
  <device>
    <name>1st_Supply_Air</name>
    <value>74.074989</value>
  </device>
  <device>
    <name>1st_Return_Air</name>
    <value>73.849998</value>
  </device>
  <device>
    <name>Attic_Temp</name>
    <value>80.150001</value>
  </device>
  <device>
    <name>2nd_Supply_Air</name>
    <value>54.049999</value>
  </device>
  <device>
    <name>2nd_Return_Air</name>
    <value>70.924995</value>
  </device>
  <device>
    <name>deltaT1</name>
    <value>-0.218506</value>
  </device>
  <device>
    <name>deltaT2</name>
    <value>16.861397</value>
  </device>
</devices>

…and here’s how to pull out Attic_Temp and Basemnt_Temp — the others should be obvious (but do ask if not):

rest:
  - resource: http://10.10.0.44:5150/data.xml
    scan_interval: 300
    sensor:
      - name: Attic temperature
        unit_of_measurement: '°F'
        device_class: temperature
        value_template: >
          {{ value_json['devices']['device']
             |selectattr('name','eq','Attic_Temp')
             |map(attribute='value')
             |first }}
      - name: Basement temperature
        unit_of_measurement: '°F'
        device_class: temperature
        value_template: >
          {{ value_json['devices']['device']
             |selectattr('name','eq','Basemnt_Temp')
             |map(attribute='value')
             |first }}

wpaul.johnson · February 24, 2023, 2:51am

Awesome! That did exactly what I was looking for!!

My last programming for Clarion for DOS. Where do you learn the magic of these commands used inside the configuration file?

Troon · February 24, 2023, 7:18am

Most of the work is understanding the data structures (dictionaries, lists, strings etc) and how to manipulate them. There’s a very good template editor within HA, under Developer Tools / Template — and that links to a couple of useful references on the Jinja2 template language.

The docs are very good too, although they can take a bit of navigating and it helps if you know what you’re looking for.

In this case, your resource was providing an XML document. I knew that the REST sensor can read XML, converting it to JSON (ref). Your XML was being turned into this:

{
  "devices": {
    "device": [
      {
        "name": "date",
        "value": "02/22/2023"
      },
      {
        "name": "time",
        "value": "15:53:57"
      },
      {
        "name": "Basemnt_Temp",
        "value": "73.174995"
      },
      {
        "name": "1st_Supply_Air",
        "value": "74.074989"
      },
      {
        "name": "1st_Return_Air",
        "value": "73.849998"
      },
      {
        "name": "Attic_Temp",
        "value": "80.150001"
      },
      {
        "name": "2nd_Supply_Air",
        "value": "54.049999"
      },
      {
        "name": "2nd_Return_Air",
        "value": "70.924995"
      },
      {
        "name": "deltaT1",
        "value": "-0.218506"
      },
      {
        "name": "deltaT2",
        "value": "16.861397"
      }
    ]
  }
}

You can paste that into the template editor and play with it:

wpaul.johnson · February 24, 2023, 1:10pm

Thank you. That’s helpful.

kaykor · March 19, 2024, 2:54pm

Like to make a clarification for welserver. There is two xml’s available and this example show the file name as data.xml (not working) when in fact it is wel.xml. Working.

And a big thank you for the simple example and clarifications.

data.xml

<Devices>
<Device Name="Date" Value="03/19/2024"/>
<Device Name="Time" Value="09:05:44"/>
<Device Name="server" Value="62.824993"/>
<Device Name="util_room" Value="86.787498"/>
<Device Name="3_supply" Value="93.762489"/>
<Device Name="3_return" Value="87.912498"/>
<Device Name="2_supply" Value="158.674972"/>
<Device Name="2_return" Value="141.800003"/>
<Device Name="1_supply" Value="155.187484"/>
<Device Name="1_return" Value="77.562492"/>
<Device Name="back_return_A4" Value="132.124984"/>
<Device Name="back_supply_A4" Value="97.362495"/>
<Device Name="back_supply_B4" Value="84.987495"/>
<Device Name="back_return_B4" Value="88.587501"/>
<Device Name="dh_boiler_sup" Value="160.587478"/>
<Device Name="dh_code_supply" Value="132.124984"/>
<Device Name="dh_cold_supply" Value="55.399997"/>
<Device Name="dh_recirc" Value="129.537490"/>
<Device Name="dh_boiler_ret" Value="153.162490"/>
<Device Name="dh_tank_out" Value="138.987503"/>
<Device Name="dh_tank_in" Value="57.312496"/>
<Device Name="sys_supply" Value="136.062484"/>
<Device Name="sys_return" Value="102.537498"/>
<Device Name="apt_7_zone_1" Value="0.000000"/>
<Device Name="apt_7_zone_2" Value="1.000000"/>
<Device Name="apt_7_zone_3" Value="0.000000"/>
<Device Name="apt_7_zone_4" Value="0.000000"/>
<Device Name="zone_control_5" Value="0.000000"/>
<Device Name="zone_control_6" Value="0.000000"/>
<Device Name="zone_control_7" Value="0.000000"/>
<Device Name="zone_control_8" Value="0.000000"/>
<Device Name="ac_drain_water" Value="65.412498"/>
<Device Name="basement_supply" Value="87.462501"/>
<Device Name="basement_return" Value="89.374992"/>
<Device Name="ac_air_return" Value="62.037494"/>
<Device Name="ac_air_supply" Value="62.712497"/>
<Device Name="apt_frt_hw" Value="81.275001"/>
<Device Name="apt_frt_ac_cold" Value="50.224994"/>
<Device Name="ht-mani-pos-7" Value="64.287498"/>
<Device Name="ht-mani-pos-6" Value="65.299995"/>
<Device Name="ht-mani-pos-5" Value="65.974998"/>
<Device Name="ht-mani-pos-4" Value="66.424995"/>
<Device Name="ht-mani-pos-3" Value="67.212501"/>
<Device Name="ht-mani-pos-2" Value="68.787498"/>
<Device Name="ht-mani-pos-1" Value="70.025001"/>
</Devices>

wel.xml

<devices>
<device>
<name>date</name>
<value>03/19/2024</value>
</device>
<device>
<name>time</name>
<value>09:47:45</value>
</device>
<device>
<name>server</name>
<value>62.374996</value>
</device>
<device>
<name>util_room</name>
<value>84.424995</value>
</device>
<device>
<name>3_supply</name>
<value>105.462486</value>
</device>
<device>
<name>3_return</name>
<value>89.599998</value>
</device>
<device>
<name>2_supply</name>
<value>126.162483</value>
</device>
<device>
<name>2_return</name>
<value>103.549995</value>
</device>
<device>
<name>1_supply</name>
<value>141.012496</value>
</device>
<device>
<name>1_return</name>
<value>72.499992</value>
</device>
<device>
<name>back_return_A4</name>
<value>113.449989</value>
</device>
<device>
<name>back_supply_A4</name>
<value>110.974998</value>
</device>
<device>
<name>back_supply_B4</name>
<value>82.849998</value>
</device>
<device>
<name>back_return_B4</name>
<value>88.699989</value>
</device>
<device>
<name>dh_boiler_sup</name>
<value>132.349990</value>
</device>
<device>
<name>dh_code_supply</name>
<value>121.999992</value>
</device>
<device>
<name>dh_cold_supply</name>
<value>64.737495</value>
</device>
<device>
<name>dh_recirc</name>
<value>108.949989</value>
</device>
<device>
<name>dh_boiler_ret</name>
<value>131.899978</value>
</device>
<device>
<name>dh_tank_out</name>
<value>134.599990</value>
</device>
<device>
<name>dh_tank_in</name>
<value>80.937492</value>
</device>
<device>
<name>sys_supply</name>
<value>113.337486</value>
</device>
<device>
<name>sys_return</name>
<value>110.299995</value>
</device>
<device>
<name>apt_7_zone_1</name>
<value>0.000000</value>
</device>
<device>
<name>apt_7_zone_2</name>
<value>1.000000</value>
</device>
<device>
<name>apt_7_zone_3</name>
<value>0.000000</value>
</device>
<device>
<name>apt_7_zone_4</name>
<value>0.000000</value>
</device>
<device>
<name>zone_control_5</name>
<value>0.000000</value>
</device>
<device>
<name>zone_control_6</name>
<value>0.000000</value>
</device>
<device>
<name>zone_control_7</name>
<value>0.000000</value>
</device>
<device>
<name>zone_control_8</name>
<value>0.000000</value>
</device>
<device>
<name>ac_drain_water</name>
<value>67.212501</value>
</device>
<device>
<name>basement_supply</name>
<value>109.962486</value>
</device>
<device>
<name>basement_return</name>
<value>109.062492</value>
</device>
<device>
<name>ac_air_return</name>
<value>61.587497</value>
</device>
<device>
<name>ac_air_supply</name>
<value>62.599994</value>
</device>
<device>
<name>apt_frt_hw</name>
<value>75.650001</value>
</device>
<device>
<name>apt_frt_ac_cold</name>
<value>56.637500</value>
</device>
<device>
<name>ht-mani-pos-7</name>
<value>62.149997</value>
</device>
<device>
<name>ht-mani-pos-6</name>
<value>62.487499</value>
</device>
<device>
<name>ht-mani-pos-5</name>
<value>62.712497</value>
</device>
<device>
<name>ht-mani-pos-4</name>
<value>62.937496</value>
</device>
<device>
<name>ht-mani-pos-3</name>
<value>63.274997</value>
</device>
<device>
<name>ht-mani-pos-2</name>
<value>64.062492</value>
</device>
<device>
<name>ht-mani-pos-1</name>
<value>64.962501</value>
</device>
</devices>

duane-nepa · June 7, 2024, 4:46am

Extremely Helpful. Worked for me to get data from WeeWx.

snmrvoge · August 24, 2024, 9:02am

Hi together, im struggling with my sensor, its a Smartfox xml page.

I tried a lot of, also these two
rest:

resource: http://10.xx.xx.xx/values.xml
scan_interval: 300
sensor:
- name: ToGridNeu
  unit_of_measurement: ‘kW’
  device_class: energy
  value_template: >
  {{ value_json[‘values’]
  |selectattr(‘value id’,‘eq’,‘toGridValue’)
  |map(attribute=‘value’)
  |first }}

sensor:

platform: rest
name: Grid_Neu
resource: http://10.xx.xx.xx/values.xml
value_template: “{{ value_xml.values.value[4] }}”
scan_interval: 300
headers:
Content-Type: application/xml

but no success:-(
The file looks like this:
This XML file does not appear to have any style information associated with it. The document tree is shown below.

<values>

<value id="hidWebinterfaceVersion">9435ac2084cfe0aaaa15</value>

<value id="macAddress">6827198D4842</value>

<value id="ipAddress">010.xx.xx.xx</value>

<value id="version">EM2 00.01.07.03</value>

<value id="toGridValue">-2.59 kW</value>

<value id="lang">de</value>

<value id="dateValue">2024-08-24</value>

<value id="timeValue">09:28:18 Uhr</value>

<value id="hidApiKey">c6258081339fe35770b066e9ae9931bf</value>

and so on....
</values>

Where is the vault?

Troon · August 27, 2024, 6:42am

Please link to the documentation that covers value_xml. I suspect either you’ve invented it or you’ve made the mistake of asking AI.

From here:

Try this:

sensor:
  - platform: rest
    name: Grid_Neu
    resource: http://10.xx.xx.xx/values.xml
    scan_interval: 300
    value_template: >
      {{ (value_json['values']['value']
          |selectattr('@id','==','toGridValue')
          |first)
         ['#text']
         |select('in','-0123456789.')
         |join }}
    unit_of_measurement: 'kW'
    device_class: power

Rather than relying on its position in the list, this looks up the toGridValue element and returns its text, with the units removed to allow it to be treated as a number.

Please format your configs correctly for the forum; and note there’s no need to hide private IP addresses.