Multiscrape - please help with scraping Bayrol site

iway · September 22, 2023, 2:25pm

Hi there! I am having major problems submitting the login form for https://www.bayrol-poolaccess.de

My config:

resource: ‘https://www.bayrol-poolaccess.de/webview/p/plants.php’
scan_interval: 30
log_response: true
form_submit:
submit_once: True
#resource: ‘https://www.bayrol-poolaccess.de’

input:
username: ‘xxxxxxxxxx’
password: ‘rxxxxxxxx’
autologin: “false”
sensor:
- select: ‘div > div:nth-child(2)’
  name: pH_pool_muc

Could someone please help me here?

Log:

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] # Start processing config from configuration.yaml

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] # Found no name for scraper, generated a unique name: Scraper_noname_0

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Setting up multiscrape with config:

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Log responses enabled, creating logging folder: /config/multiscrape/scraper_noname_0/

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Initializing http wrapper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Initializing form submitter

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Initializing scraper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Initializing scraper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Initializing http wrapper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Initializing coordinator

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Scan interval is 0:00:30

2023-09-22 16:18:07.083 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # New run: start (re)loading data from resource

2023-09-22 16:18:07.083 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Deleting logging files from previous run

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Rendered resource template into: https://www.bayrol-poolaccess.de/webview/p/plants.php

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Starting with form-submit

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Skip scraping form, assuming all input is given in config.

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Merged input fields with input data in config. Result: {‘username’: ‘[email protected]’, ‘password’: ‘rybce9-kenxyh-tivTax’, ‘autologin’: ‘false’}

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Determined the url to submit the form to: https://www.bayrol-poolaccess.de/webview/p/plants.php

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Submitting the form

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Executing form_submit-request with a POST to url: https://www.bayrol-poolaccess.de/webview/p/plants.php.

2023-09-22 16:18:07.086 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_headers written to file: form_submit_request_headers.txt

2023-09-22 16:18:07.087 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_body written to file: form_submit_request_body.txt

2023-09-22 16:18:07.128 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Response status code received: 200

2023-09-22 16:18:07.129 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_headers written to file: form_submit_response_headers.txt

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_body written to file: form_submit_response_body.txt

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Form seems to be submitted succesfully (to be sure, use log_response and check file). Now continuing to retrieve target page.

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Using response from form-submit as data. Now ready to be scraped by sensors.

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Loading the content in BeautifulSoup.

2023-09-22 16:18:07.146 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # page_soup written to file: page_soup.txt

2023-09-22 16:18:07.146 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.064 seconds (success: True)

2023-09-22 16:18:07.147 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Setting up sensor

2023-09-22 16:18:07.152 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Start scraping to update sensor

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Tag selected:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Selector result:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Final selector value:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Selected:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_0 # pH_pool_muc # Updated sensor and attributes, now adding to HA

Page_soup.txt still shows login page:

BAYROL Webportal div.field:nth-child(0){display:none;} #T5c01230d75 {width: 220px;}

BAYROL Webportal

Konto erstellen Passwort vergessen?Download-Center

DE|FR|ES|EN

Anmeldung

Zeit abgelaufen, bitte Seite neu laden

Bitte melden Sie sich mit Ihrer E-Mail-Adresse (=Benutzername) und Passwort an

E-Mail-Adresse (=Benutzername)

Passwort Passwort anzeigen/verstecken

Anmeldedaten merken (nicht auf öffentlichen Computern verwenden!)

Impressum

Nutzungsbedingungen

Silvester5 · October 26, 2023, 8:20pm

You have to find the DeviceID after login. In my Case cid=19933.
Then this Code works:

multiscrape:
  - name: Bayrol
    resource: https://www.bayrol-poolaccess.de/webview/getdata.php?cid=19933
    log_response: true
    scan_interval: 30
    form_submit:
      submit_once: true
      resource: https://www.bayrol-poolaccess.de/webview/p/login.php
      select: '#form_login'
      input:
        username: # Mailadress
        password: # Password
        autologin: None
    sensor:
      - unique_id: Pool PH Wert
        name: Pool PH Wert
        select: 'body > div > div:nth-child(2) > h1'
      - unique_id: Redoxwert
        name: Redoxwert
        unit_of_measurement: "mV"
        select: 'body > div > div:nth-child(3) > h1'
      - unique_id: Pooltemperatur
        name: Pooltemperatur
        unit_of_measurement: "°C"
        select: 'body > div > div:nth-child(4) > h1'

Tomthebest1 · November 19, 2023, 6:30pm

It’s working fine! Thanks

Tomthebest1 · December 10, 2023, 10:14pm

Hi
Stop working after one day. Do you have any solution for this?
Thanks

Silvester5 · January 2, 2024, 1:56pm

They changed something on their homepage, now this works for me:

multiscrape:
  - name: Bayrol
    resource: https://www.bayrol-poolaccess.de/webview/getdata.php?cid=19933
    log_response: true
    scan_interval: 30
    form_submit:
      submit_once: true
      resource: https://www.bayrol-poolaccess.de/webview/p/login.php
      select: '#form_login'
      input:
        username: # Mailadress
        password: # Password
        autologin: None
    sensor:
      - unique_id: Pool PH Wert
        name: Pool PH Wert
        select: 'body > div > div.tab_data_link > div:nth-child(2) > h1'
      - unique_id: Redoxwert
        name: Redoxwert
        unit_of_measurement: "mV"
        select: 'body > div > div.tab_data_link > div:nth-child(3) > h1'
      - unique_id: Pooltemperatur
        name: Pooltemperatur
        unit_of_measurement: "°C"
        select: 'body > div > div.tab_data_link > div:nth-child(4) > h1'

ChicktheChicken · January 27, 2024, 10:55am

Thanks for your fantastic help! Unfortunately my sensors are unknown.

Is it still working for you?

Tomthebest1 · February 22, 2024, 4:15pm

I changed the recource line to

resource: https://www.bayrol-poolaccess.de/webview/p/device.php?c=XXXXX

The XXXXX is your device.

Now its working!
Thanks

Tomthebest1 · February 22, 2024, 4:26pm

Sorry. Worked only for some minutes.
Any suggestion?

ChicktheChicken · February 25, 2024, 2:45pm

I suspect that the paths on the website have changed. But I can’t copy them because they close too quickly.

harb70 · April 22, 2024, 4:03pm

Hi, I’m using MQTT to scrape data with Nodered.then post to Home Assistant MQTT broker.

PierreJDot · May 5, 2024, 6:40am

How exactly did you implement this with mqtt? I am looking for a way to query the status of the pump or the flow switch. In a knx forum this was described as a module which can read and set all states.

harb70 · May 12, 2024, 1:04pm

Sorry for the delay.
You can connect Bayrol Mqtt server.
When you open the web app, you can find which user/password is used.(dev tools)
wss://www.bayrol-poolaccess.de:8083/
With these informations, I configured a mqtt in node reading messages, then I write to mosquito broker in HA.

For now I just worked on 4 metrics… it could be great to be several to reverse engineer all the commands

harb70 · May 12, 2024, 2:21pm

connect to the web portal
clic on “Direct Access”
on dev tools, clic network, search for the websocket connexion wss://www.bayrol-poolaccess.de:8083/
Clic on Messages and find the 1st one
you will find the username to use.

You can test with a MQTT tool like MQTT Explorer

Dev tool screenshot (Red line is the username to copy)

MQTT Explorer connexion setup

d02//v/4.82 redox
d02//v/4.98 temperature
d02//v/4.100 salt
d02//v/4.182 pH

Duntch144 · May 18, 2024, 9:38am

Hi, I’m trying to find a way to connect Bayrol server. With procedure explained I’m connected to MQTT Bayrol server via MQTT Explorer. Nothing is displayed in MQTT Explorer. How to retrieve data from the server ?