Issue with Multiscrape / Scrape

Robi07 · December 17, 2021, 10:11am

Thomas…many thanks for this tip. Changed my scan interval - and until now no crash
Amazing how much time you invest to get such a banal sensor running

Found also the value template for hot water:

      - name: Elco Set Water Temp
        value_template: "{{ value_json.dhwComfortTemp.value }}"
        unit_of_measurement: "°C"

If interested in the value “desiredRoomTemperature” - adjust value template as follows:

      - name: Elco Room Temperature
        value_template: "{{ value_json.zone.desiredRoomTemp }}"
        unit_of_measurement: "°C"

You can copy your URL-output to the following webpage to get it formatted: https://jsonformatter.curiousconcept.com
Let me know if you need some more value that you can’t get yet.

Elco

Robi07 · December 17, 2021, 4:00pm

Hi Daniel. As you might have seen, Thomas and I found a solution by combining your Multiscrape integration with a normal rest-sensor (normal rest-sensor is not working stand-alone i.e. we suspect that the form_submit makes the difference in combining both) . I do understand that multiscraping is not working with dynamic content as explained by you, but I’m wondering why the multiscrape-integration can’t extract from the second link with json-values only like the rest-sensor. Are we missing something within the value template? The following config gives no value for multiscrape sensor but the correct value for the rest-sensor.

multiscrape:
  - resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    scan_interval: 3600
    form_submit:
      submit_once: True
      resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
      select: "#login-form"
      input:
        Email: !secret elco_username
        Password: !secret elco_password
        extra: field
    sensor:
      - name: Elco Outside Temperature
        value_template: '{{ value_json.outsideTemp }}'

rest:
  - authentication: basic
    scan_interval: 3600
    resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    username: !secret elco_username
    password: !secret elco_password
    sensor:
      - name: ElcoOutsideTemperature
        value_template: "{{ value_json.outsideTemp }}"

The values from the json-value-only-link look as follows. As you see you hardly find a css-selector (only “body > pre”) - is that the problem with the multiscrape integration?

danieldotnl · January 2, 2022, 8:08pm

Hi Robi and Thomas, hats off to you! You found a very creative solution. This works because connections to the same endpoint are reused by HA, no matter which component initiated the connection.

@Robi07 Your post made me realize there is much simpler solution to support json, than what I initially had in mind! So as a reward I implemented parsing json through the value_templates.
This basically means that Multiscrape is now not just an improved Scrape component but an improved Rest component as well!!

@Thomas_Gregg : sorry, it took a while…

Now available in v5.7.0!

Thomas_Gregg · January 2, 2022, 8:44pm

will give it a try these days.

Robi07 · January 3, 2022, 2:57pm

Hi Daniel. This is genious! Works great! Many thanks for this!!! I even moved some normal rest-sensors to the multiscrape platform given the unique_id functionality of Multiscrape and subsequent area-assignment (now hoping that the new Lovelace Area Card will be improved in Q1 22 i.e. manual configuration possibilities). Best, Robert

Robi07 · January 3, 2022, 2:58pm

Hi Thomas. The following works for me:

multiscrape:
  - resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    scan_interval: 900
    form_submit:
      submit_once: True
      resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
      select: "#login-form"
      input:
        Email: !secret elco_username
        Password: !secret elco_password
        extra: field
    sensor:
      - name: Elco Aussentemperatur
        unique_id: Elco Aussentemperatur
        value_template: '{{ value_json.outsideTemp }}'
        unit_of_measurement: "°C" 
      - name: Elco Raumtemperatur
        unique_id: Elco Raumtemperatur
        value_template: '{{ value_json.zone.desiredRoomTemp }}'
        unit_of_measurement: "°C" 
      - name: Elco Wassertemperatur Ist
        unique_id: Elco Wassertemperatur Ist
        value_template: '{{ value_json.dhwStorageTemp }}'
        unit_of_measurement: "°C" 
      - name: Elco Wassertemperatur Soll
        unique_id: Elco Wassertemperatur Soll
        value_template: '{{ value_json.dhwComfortTemp.value }}'
        unit_of_measurement: "°C"

Thomas_Gregg · January 4, 2022, 9:47am

I can say that it also works perfectly fine for me. Thanks a lot @Robi07 and @danieldotnl .

Thomas_Gregg · July 14, 2022, 10:05pm

@Robi07 It seems they changed the web interface. Could you get it working with the new interface?

Thomas_Gregg · July 17, 2022, 10:06am

I gave it a try here: Help with JSON value template - #7 by Hellis81 but no success yet.

Robi07 · August 14, 2022, 4:14pm

Hi Thomas. I was on holiday and just realized the issue right now. I also had a look on your other post. Looks like the plant datas are not accessible anymore from external sources i.e. the link “https://www.remocon-net.remotethermo.com/R2/PlantHomeBsb/GetData/F…” generates a 404 error if you try to access via explorer. Unless there is another link to access json-values directly from the explorer, I don’t see any chance to get the values to HA. Best, Robert

MKumar · August 18, 2022, 6:40am

Hi,
I am trying to scrape a website that requires login.

My Configuration:

multiscrape:
- resource: 'https://xxxxx.com/studportal/Student-Home.aspx'
  scan_interval: 3600
  form_submit:
    submit_once: True
    resource: 'https://xxxxx.com/studportal/frmLogin.aspx'
    select: "#aspnetForm"
    input:
      ctl00_ContentPlaceHolder1_txtUserID_Raw: 'user'
      ctl00$ContentPlaceHolder1$txtUserID: 'user'
      ctl00_ContentPlaceHolder1_txtPassword_Raw: 'passwd'
      ctl00$ContentPlaceHolder1$txtPassword: 'passwd'
      ctl00$ContentPlaceHolder1$BnLogin: Login
      #ctl00$ContentPlaceHolder1$BnForgotPassword: Login
  sensor:
    - select: '#studInfoPanelDetailTBL > tr > td'
      name: studentname
  log_response: true

From the log response I see that form is submitted for ‘Forget Password’ instead of ‘Login’.
Can anybody suggest how do I configure to login.

fodevaux · August 20, 2022, 3:16pm

There has been an update in the html code of the login page (login-form replaced by loginForm).
You should replace

...
resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
select: "#login-form"
...

with

...
resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
select: "#loginForm"
...

fodevaux · August 31, 2022, 6:47am

Hi there,
Based on the suggestions above, i created a code to read the temperature of my Ariston Nuos Plus Wifi boiler in Home Assistant. I guess many suppliers use the same underlying resources to manage digital interactions with their boilers.
Anyway, here is the YAML I use

  - resource: https://www.ariston-net.remotethermo.com/R2/PlantHomeSlp/GetData/34AB9582FD2C?fetchSettings=true&fetchTimeProg=true&rnd=1657775300275
    scan_interval: 300
    log_response: true
    form_submit:
      submit_once: True
      resource: https://www.ariston-net.remotethermo.com/R2/Account/Login
      select: "#loginForm"
      input:
        Email: !secret ariston_username
        Password: !secret ariston_password
        extra: field
    sensor:
      - unique_id: boiler_temperature
        name: Boiler temperature
        value_template: '{{ value_json.data.plantData.waterTemp }}'
        unit_of_measurement: '°C'

Robi07 · September 2, 2022, 9:08am

Hi Francois. Many thanks for your response. I tried to adapt your YAML code to my device - but I’m getting HTTP 500 errors through Home Assistant Log.

The login through
https://www.ariston-net.remotethermo.com/R2/Account/Login
works through a web browser but when I try to open the suggested final link (with replaced device number) then I’m getting an error message (see image below). May I therefore quickly ask:

If you open your suggested link (

https://www.ariston-net.remotethermo.com/R2/PlantHomeSlp/GetData/34AB9582FD2C?fetchSettings=true&fetchTimeProg=true&rnd=1657775300275

through the web browser (Safari, Chrome etc.), do you see the respective values? I only get the following error message

What device number are you using. I tried both (appliance serial and gateway serial) but none was working

Many thanks in advance for your quick response. Best, Robert

S1M8N · September 17, 2022, 11:41am

Hello, I have the same issue for elco link

I search the probleme but if you have any suggested I will take it

Many thanks in advance for your help

AnthonieV · October 8, 2022, 10:21am

Hi,

I tested this and it woks perfectly. I was wondering, is there a way to switch the Ariston Nuos Plus Wifi from “time program” to “manual” using HA?
Thanks a lot!

Best regards,
Anthonie

fodevaux · October 28, 2022, 4:37pm

Hi,
I haven’t really tried to control the Nuos from HA, just to read data from it. Interested in the answer, if someone has it !

Cheers,

François

AnthonieV · October 28, 2022, 7:29pm

Hi,

Well I searched a bit further and used the app ‘web proxy tool’ to sniffer the messages send by my ‘AquaAristonNET’ app.
Changing the Boiler from ‘Manual mode’ to ‘Automatic Mode’ gives me some data

POST.200 OK.
https://www.ariston-net.remotethermo.com/api/v2/velis/slpPlantData/C82B96BD387C/plantMode?appId=com.remotethermo.velis
BODY
{“new”:1,“old”:2} → it means going from Manual:2 to Automatic:1
HEADERS
Expect:
100-continue
ar.authToken:
xxxxxxxxxxxxxxx…
Connection:
Keep-alive

And so on…(ask me when you need it or try it by yourself using the app)

I took over all the data using rest command:

rest_command:
  aristonnuos_man:
    url: https://www.ariston-net.remotethermo.com/api/v2/velis/slpPlantData/xxxxxxxxxxxx/plantMode? 
    appId=com.remotethermo.velis
    method: POST
    headers:
      content-type: 'application/json'
      expect: '100-continue'
      accept-encoding: 'gzip, deflate'
      accept: "application/json, text/json, text/x-json, text/javascript, application/xml, text/xml"
      host: 'www.ariston-net.remotethermo.com'
      content-length: '17'
      ar.authtoken: 'blablabla'
      user-agent: 'RestSharp/106.0.0.0'
    payload: '{"new":2, "old":1}'
    content_type: 'application/json'

And it works…for a couple of days because then the ar.authtoken was changed… I tried with a basic login in my POST, but didn’t work.

I also saw in the link the refer to velis. So I tried the HA integration of aqua remotethermo, but I didn’t work.

tgi · August 28, 2023, 5:21pm

Hi to all,
I am experiencing a problem with retrieving data from a web page.
Specifically I want to get the data from :
body > table:nth-child(1) > tbody:nth-child(1) > tr:nth-child(9) > td:nth-child(2) > font:nth-child(1) > strong:nth-child(1) > font:nth-child(1) > small:nth-child(1) retrieved from http://perachora-davis.meteoclub.gr/
I’ve tried everything but it doesn’t bring me any data.
I have tried with other fields and the result is the same.

Can you help me? What can go wrong?

Thanks

AnthonieV · October 7, 2023, 3:08pm

With this integration you can read and control Ariston Nuos Split Heatpump Boilers