Multiscrape - please help with scraping Bayrol site

Hi there! I am having major problems submitting the login form for https://www.bayrol-poolaccess.de

My config:

Could someone please help me here?

Log:

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] # Start processing config from configuration.yaml

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] # Found no name for scraper, generated a unique name: Scraper_noname_0

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Setting up multiscrape with config:

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Log responses enabled, creating logging folder: /config/multiscrape/scraper_noname_0/

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Initializing http wrapper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Initializing form submitter

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Initializing scraper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Initializing scraper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Initializing http wrapper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Initializing coordinator

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Scan interval is 0:00:30

2023-09-22 16:18:07.083 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # New run: start (re)loading data from resource

2023-09-22 16:18:07.083 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Deleting logging files from previous run

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Rendered resource template into: https://www.bayrol-poolaccess.de/webview/p/plants.php

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Starting with form-submit

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Skip scraping form, assuming all input is given in config.

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Merged input fields with input data in config. Result: {‘username’: ‘[email protected]’, ‘password’: ‘rybce9-kenxyh-tivTax’, ‘autologin’: ‘false’}

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Determined the url to submit the form to: https://www.bayrol-poolaccess.de/webview/p/plants.php

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Submitting the form

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Executing form_submit-request with a POST to url: https://www.bayrol-poolaccess.de/webview/p/plants.php.

2023-09-22 16:18:07.086 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_headers written to file: form_submit_request_headers.txt

2023-09-22 16:18:07.087 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_body written to file: form_submit_request_body.txt

2023-09-22 16:18:07.128 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Response status code received: 200

2023-09-22 16:18:07.129 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_headers written to file: form_submit_response_headers.txt

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_body written to file: form_submit_response_body.txt

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Form seems to be submitted succesfully (to be sure, use log_response and check file). Now continuing to retrieve target page.

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Using response from form-submit as data. Now ready to be scraped by sensors.

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Loading the content in BeautifulSoup.

2023-09-22 16:18:07.146 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # page_soup written to file: page_soup.txt

2023-09-22 16:18:07.146 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.064 seconds (success: True)

2023-09-22 16:18:07.147 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Setting up sensor

2023-09-22 16:18:07.152 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Start scraping to update sensor

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Tag selected:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Selector result:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Final selector value:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Selected:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_0 # pH_pool_muc # Updated sensor and attributes, now adding to HA

Page_soup.txt still shows login page:

BAYROL Webportal div.field:nth-child(0){display:none;} #T5c01230d75 {width: 220px;} bg
Anmeldung
Zeit abgelaufen, bitte Seite neu laden

Bitte melden Sie sich mit Ihrer E-Mail-Adresse (=Benutzername) und Passwort an


E-Mail-Adresse (=Benutzername)
Passwort Passwort anzeigen/verstecken
Anmeldedaten merken (nicht auf öffentlichen Computern verwenden!)

You have to find the DeviceID after login. In my Case cid=19933.
Then this Code works:

multiscrape:
  - name: Bayrol
    resource: https://www.bayrol-poolaccess.de/webview/getdata.php?cid=19933
    log_response: true
    scan_interval: 30
    form_submit:
      submit_once: true
      resource: https://www.bayrol-poolaccess.de/webview/p/login.php
      select: '#form_login'
      input:
        username: # Mailadress
        password: # Password
        autologin: None
    sensor:
      - unique_id: Pool PH Wert
        name: Pool PH Wert
        select: 'body > div > div:nth-child(2) > h1'
      - unique_id: Redoxwert
        name: Redoxwert
        unit_of_measurement: "mV"
        select: 'body > div > div:nth-child(3) > h1'
      - unique_id: Pooltemperatur
        name: Pooltemperatur
        unit_of_measurement: "°C"
        select: 'body > div > div:nth-child(4) > h1'
1 Like

It’s working fine! Thanks

Hi
Stop working after one day. Do you have any solution for this?
Thanks

They changed something on their homepage, now this works for me:

multiscrape:
  - name: Bayrol
    resource: https://www.bayrol-poolaccess.de/webview/getdata.php?cid=19933
    log_response: true
    scan_interval: 30
    form_submit:
      submit_once: true
      resource: https://www.bayrol-poolaccess.de/webview/p/login.php
      select: '#form_login'
      input:
        username: # Mailadress
        password: # Password
        autologin: None
    sensor:
      - unique_id: Pool PH Wert
        name: Pool PH Wert
        select: 'body > div > div.tab_data_link > div:nth-child(2) > h1'
      - unique_id: Redoxwert
        name: Redoxwert
        unit_of_measurement: "mV"
        select: 'body > div > div.tab_data_link > div:nth-child(3) > h1'
      - unique_id: Pooltemperatur
        name: Pooltemperatur
        unit_of_measurement: "°C"
        select: 'body > div > div.tab_data_link > div:nth-child(4) > h1'

Thanks for your fantastic help! Unfortunately my sensors are unknown.

Is it still working for you?

I changed the recource line to

resource: https://www.bayrol-poolaccess.de/webview/p/device.php?c=XXXXX

The XXXXX is your device.

Now its working!
Thanks

Sorry. Worked only for some minutes.
Any suggestion?

I suspect that the paths on the website have changed. But I can’t copy them because they close too quickly.