Issue with Multiscrape / Scrape

Robi07 · December 13, 2021, 9:54am

Thomas,

I found a solution:

log-in via multiscrape-function (produces a sensor without function - but you can deactivate)
create separate restsensor

The rest sensor alone is not working, but works together with the multiscrape. I entered the following to my configuration.yaml:

multiscrape:
  - resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    scan_interval: 5
    form_submit:
      submit_once: True
      resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
      select: "#login-form"
      input:
        Email: !secret elco_username
        Password: !secret elco_password
        extra: field
    sensor:
      - name: Elco Login
        value_template: '{{ value_json.outsideTemp }}'
sensor:
  - platform: rest
    resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    username: !secret elco_username
    password: !secret elco_password
    authentication: basic
    name: Elco Outside Temperature
    value_template: "{{ value_json.outsideTemp }}"
    unit_of_measurement: "°C"

Does it also work on your side?

Best

Thomas_Gregg · December 13, 2021, 10:22am

Wow… I will give it a try this evening and will let you know. Thanks for sharing so far.

Robi07 · December 13, 2021, 2:01pm

Unfortunately, the first joy has passed. After some hours with no problems I added two more sensors…it worked for about 1 minute. Than it stopped working i.e. the called url responded with a gateway error message and the app has stopped working (the dashboard of the app/online-log-in responds with communication error / the “user menu” however still works i.e. shows all values . Looks like the url is not intended for scraping. I opened now a ticket to get the app working again.

Robi07 · December 13, 2021, 4:38pm

Thomas, My App is up and running again. I had someone from Elco on the line - the system re-starts automatically after a certain time (in my case some hours). He has to check internally why/if my scrape command caused this incidence. Well keep you posted.

Best

Thomas_Gregg · December 13, 2021, 5:10pm

I tried it and ran into the same problem… Will not open a ticket and hoping it will fix itself. But I can confirm that it worked well for a while. Please keep me posted.

Thomas_Gregg · December 16, 2021, 7:34pm

@Robi07 I played around a little bit more. It now works well for me without getting locked out. I changed a couple of things so not 100% sure what makes the difference but I assume it is most likely the Scan Interval which I changed to 1h. This causes not to load the values when you restart HA and you will be seeing them only after 1h. It also takes 1h to get an update but that’s good enough for me.

My code looks like this:

### - - - ELCO HEATING - - - ###
 
multiscrape:
  - resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E0B7C60?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    scan_interval: 3600
    form_submit:
      submit_once: True
      resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
      select: "#login-form"
      input:
        Email: !secret elco_username
        Password: !secret elco_password
        extra: field
    sensor:
      - name: ElcoLogin
#        value_template: '{{ value_json.outsideTemp }}'

rest:
  - authentication: basic
    scan_interval: 3600
    resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E0B7C60?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    username: !secret elco_username
    password: !secret elco_password
    sensor:
      - name: ElcoOutsideTemperature
        value_template: "{{ value_json.outsideTemp }}"
        unit_of_measurement: "°C" 
      - name: ElcoHeatPumpOn
        value_template: "{{ value_json.heatPumpOn }}"
      - name: ElcodhwStorageTemp
        value_template: "{{ value_json.dhwStorageTemp }}"  
      - name: ElcoHotWaterTemp
        value_template: "{{ value_json.value }}"

ElcoHotWaterTemp is not working yet. I have to find out the right value template for it. In case you know, please let me know.

Thomas_Gregg · December 17, 2021, 7:58am

Ok I got it working for me now (with the limitations mentioned above)

### - - - ELCO HEATING - - - ###
 
multiscrape:
  - resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E0B7C60?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    scan_interval: 3600
    form_submit:
      submit_once: True
      resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
      select: "#login-form"
      input:
        Email: !secret elco_username
        Password: !secret elco_password
        extra: field
    sensor:
      - name: ElcoLogin
#        value_template: '{{ value_json.outsideTemp }}'

rest:
  - authentication: basic
    scan_interval: 3600
    resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E0B7C60?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    username: !secret elco_username
    password: !secret elco_password
    sensor:
      - name: ElcoOutsideTemperature
        value_template: "{{ value_json.outsideTemp }}"
        unit_of_measurement: "°C" 
      - name: ElcoHeatPumpOn
        value_template: "{{ value_json.heatPumpOn }}"
      - name: ElcodhwStorageTemp
        value_template: "{{ value_json.dhwStorageTemp }}"  
        unit_of_measurement: "°C" 
      - name: ElcoHotWaterTemp
        value_template: "{{ value_json.dhwComfortTemp.value }}"  
        unit_of_measurement: "°C"

Robi07 · December 17, 2021, 10:11am

Thomas…many thanks for this tip. Changed my scan interval - and until now no crash
Amazing how much time you invest to get such a banal sensor running

Found also the value template for hot water:

      - name: Elco Set Water Temp
        value_template: "{{ value_json.dhwComfortTemp.value }}"
        unit_of_measurement: "°C"

If interested in the value “desiredRoomTemperature” - adjust value template as follows:

      - name: Elco Room Temperature
        value_template: "{{ value_json.zone.desiredRoomTemp }}"
        unit_of_measurement: "°C"

You can copy your URL-output to the following webpage to get it formatted: https://jsonformatter.curiousconcept.com
Let me know if you need some more value that you can’t get yet.

Elco

Robi07 · December 17, 2021, 4:00pm

Hi Daniel. As you might have seen, Thomas and I found a solution by combining your Multiscrape integration with a normal rest-sensor (normal rest-sensor is not working stand-alone i.e. we suspect that the form_submit makes the difference in combining both) . I do understand that multiscraping is not working with dynamic content as explained by you, but I’m wondering why the multiscrape-integration can’t extract from the second link with json-values only like the rest-sensor. Are we missing something within the value template? The following config gives no value for multiscrape sensor but the correct value for the rest-sensor.

multiscrape:
  - resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    scan_interval: 3600
    form_submit:
      submit_once: True
      resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
      select: "#login-form"
      input:
        Email: !secret elco_username
        Password: !secret elco_password
        extra: field
    sensor:
      - name: Elco Outside Temperature
        value_template: '{{ value_json.outsideTemp }}'

rest:
  - authentication: basic
    scan_interval: 3600
    resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    username: !secret elco_username
    password: !secret elco_password
    sensor:
      - name: ElcoOutsideTemperature
        value_template: "{{ value_json.outsideTemp }}"

The values from the json-value-only-link look as follows. As you see you hardly find a css-selector (only “body > pre”) - is that the problem with the multiscrape integration?

danieldotnl · January 2, 2022, 8:08pm

Hi Robi and Thomas, hats off to you! You found a very creative solution. This works because connections to the same endpoint are reused by HA, no matter which component initiated the connection.

@Robi07 Your post made me realize there is much simpler solution to support json, than what I initially had in mind! So as a reward I implemented parsing json through the value_templates.
This basically means that Multiscrape is now not just an improved Scrape component but an improved Rest component as well!!

@Thomas_Gregg : sorry, it took a while…

Now available in v5.7.0!

Thomas_Gregg · January 2, 2022, 8:44pm

will give it a try these days.

Robi07 · January 3, 2022, 2:57pm

Hi Daniel. This is genious! Works great! Many thanks for this!!! I even moved some normal rest-sensors to the multiscrape platform given the unique_id functionality of Multiscrape and subsequent area-assignment (now hoping that the new Lovelace Area Card will be improved in Q1 22 i.e. manual configuration possibilities). Best, Robert

Robi07 · January 3, 2022, 2:58pm

Hi Thomas. The following works for me:

multiscrape:
  - resource: https://www.remocon-net.remotethermo.com/BsbPlantDashboard/GetPlantData/F0AD4E109C20?zoneNum=1&isFirstRoundTrip=true&rnd=1629792069874&_=1629792068973
    scan_interval: 900
    form_submit:
      submit_once: True
      resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
      select: "#login-form"
      input:
        Email: !secret elco_username
        Password: !secret elco_password
        extra: field
    sensor:
      - name: Elco Aussentemperatur
        unique_id: Elco Aussentemperatur
        value_template: '{{ value_json.outsideTemp }}'
        unit_of_measurement: "°C" 
      - name: Elco Raumtemperatur
        unique_id: Elco Raumtemperatur
        value_template: '{{ value_json.zone.desiredRoomTemp }}'
        unit_of_measurement: "°C" 
      - name: Elco Wassertemperatur Ist
        unique_id: Elco Wassertemperatur Ist
        value_template: '{{ value_json.dhwStorageTemp }}'
        unit_of_measurement: "°C" 
      - name: Elco Wassertemperatur Soll
        unique_id: Elco Wassertemperatur Soll
        value_template: '{{ value_json.dhwComfortTemp.value }}'
        unit_of_measurement: "°C"

Thomas_Gregg · January 4, 2022, 9:47am

I can say that it also works perfectly fine for me. Thanks a lot @Robi07 and @danieldotnl .

Thomas_Gregg · July 14, 2022, 10:05pm

@Robi07 It seems they changed the web interface. Could you get it working with the new interface?

Thomas_Gregg · July 17, 2022, 10:06am

I gave it a try here: Help with JSON value template - #7 by Hellis81 but no success yet.

Robi07 · August 14, 2022, 4:14pm

Hi Thomas. I was on holiday and just realized the issue right now. I also had a look on your other post. Looks like the plant datas are not accessible anymore from external sources i.e. the link “https://www.remocon-net.remotethermo.com/R2/PlantHomeBsb/GetData/F…” generates a 404 error if you try to access via explorer. Unless there is another link to access json-values directly from the explorer, I don’t see any chance to get the values to HA. Best, Robert

MKumar · August 18, 2022, 6:40am

Hi,
I am trying to scrape a website that requires login.

My Configuration:

multiscrape:
- resource: 'https://xxxxx.com/studportal/Student-Home.aspx'
  scan_interval: 3600
  form_submit:
    submit_once: True
    resource: 'https://xxxxx.com/studportal/frmLogin.aspx'
    select: "#aspnetForm"
    input:
      ctl00_ContentPlaceHolder1_txtUserID_Raw: 'user'
      ctl00$ContentPlaceHolder1$txtUserID: 'user'
      ctl00_ContentPlaceHolder1_txtPassword_Raw: 'passwd'
      ctl00$ContentPlaceHolder1$txtPassword: 'passwd'
      ctl00$ContentPlaceHolder1$BnLogin: Login
      #ctl00$ContentPlaceHolder1$BnForgotPassword: Login
  sensor:
    - select: '#studInfoPanelDetailTBL > tr > td'
      name: studentname
  log_response: true

From the log response I see that form is submitted for ‘Forget Password’ instead of ‘Login’.
Can anybody suggest how do I configure to login.

fodevaux · August 20, 2022, 3:16pm

There has been an update in the html code of the login page (login-form replaced by loginForm).
You should replace

...
resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
select: "#login-form"
...

with

...
resource: 'https://www.remocon-net.remotethermo.com/Account/Login'
select: "#loginForm"
...

fodevaux · August 31, 2022, 6:47am

Hi there,
Based on the suggestions above, i created a code to read the temperature of my Ariston Nuos Plus Wifi boiler in Home Assistant. I guess many suppliers use the same underlying resources to manage digital interactions with their boilers.
Anyway, here is the YAML I use

  - resource: https://www.ariston-net.remotethermo.com/R2/PlantHomeSlp/GetData/34AB9582FD2C?fetchSettings=true&fetchTimeProg=true&rnd=1657775300275
    scan_interval: 300
    log_response: true
    form_submit:
      submit_once: True
      resource: https://www.ariston-net.remotethermo.com/R2/Account/Login
      select: "#loginForm"
      input:
        Email: !secret ariston_username
        Password: !secret ariston_password
        extra: field
    sensor:
      - unique_id: boiler_temperature
        name: Boiler temperature
        value_template: '{{ value_json.data.plantData.waterTemp }}'
        unit_of_measurement: '°C'