Multiscrape - please help with scraping Bayrol site

iway · September 22, 2023, 2:25pm

Hi there! I am having major problems submitting the login form for https://www.bayrol-poolaccess.de

My config:

resource: ‘https://www.bayrol-poolaccess.de/webview/p/plants.php’
scan_interval: 30
log_response: true
form_submit:
submit_once: True
#resource: ‘https://www.bayrol-poolaccess.de’

input:
username: ‘xxxxxxxxxx’
password: ‘rxxxxxxxx’
autologin: “false”
sensor:
- select: ‘div > div:nth-child(2)’
  name: pH_pool_muc

Could someone please help me here?

Log:

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] # Start processing config from configuration.yaml

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] # Found no name for scraper, generated a unique name: Scraper_noname_0

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Setting up multiscrape with config:

2023-09-22 16:18:07.079 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Log responses enabled, creating logging folder: /config/multiscrape/scraper_noname_0/

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Initializing http wrapper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Initializing form submitter

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Initializing scraper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Initializing scraper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Initializing http wrapper

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape] Scraper_noname_0 # Initializing coordinator

2023-09-22 16:18:07.081 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Scan interval is 0:00:30

2023-09-22 16:18:07.083 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # New run: start (re)loading data from resource

2023-09-22 16:18:07.083 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Deleting logging files from previous run

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Rendered resource template into: https://www.bayrol-poolaccess.de/webview/p/plants.php

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Starting with form-submit

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Skip scraping form, assuming all input is given in config.

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Merged input fields with input data in config. Result: {‘username’: ‘[email protected]’, ‘password’: ‘rybce9-kenxyh-tivTax’, ‘autologin’: ‘false’}

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Determined the url to submit the form to: https://www.bayrol-poolaccess.de/webview/p/plants.php

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Submitting the form

2023-09-22 16:18:07.085 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Executing form_submit-request with a POST to url: https://www.bayrol-poolaccess.de/webview/p/plants.php.

2023-09-22 16:18:07.086 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_headers written to file: form_submit_request_headers.txt

2023-09-22 16:18:07.087 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_body written to file: form_submit_request_body.txt

2023-09-22 16:18:07.128 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Response status code received: 200

2023-09-22 16:18:07.129 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_headers written to file: form_submit_response_headers.txt

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_body written to file: form_submit_response_body.txt

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Form seems to be submitted succesfully (to be sure, use log_response and check file). Now continuing to retrieve target page.

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Using response from form-submit as data. Now ready to be scraped by sensors.

2023-09-22 16:18:07.131 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Loading the content in BeautifulSoup.

2023-09-22 16:18:07.146 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # page_soup written to file: page_soup.txt

2023-09-22 16:18:07.146 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.064 seconds (success: True)

2023-09-22 16:18:07.147 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Setting up sensor

2023-09-22 16:18:07.152 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Start scraping to update sensor

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Tag selected:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Selector result:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # pH_pool_muc # Final selector value:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # pH_pool_muc # Selected:

2023-09-22 16:18:07.153 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_0 # pH_pool_muc # Updated sensor and attributes, now adding to HA

Page_soup.txt still shows login page:

BAYROL Webportal div.field:nth-child(0){display:none;} #T5c01230d75 {width: 220px;}

BAYROL Webportal

Konto erstellen Passwort vergessen?Download-Center

DE|FR|ES|EN

Anmeldung

Zeit abgelaufen, bitte Seite neu laden

Bitte melden Sie sich mit Ihrer E-Mail-Adresse (=Benutzername) und Passwort an

E-Mail-Adresse (=Benutzername)

Passwort Passwort anzeigen/verstecken

Anmeldedaten merken (nicht auf öffentlichen Computern verwenden!)

Impressum

Nutzungsbedingungen

Silvester5 · October 26, 2023, 8:20pm

You have to find the DeviceID after login. In my Case cid=19933.
Then this Code works:

multiscrape:
  - name: Bayrol
    resource: https://www.bayrol-poolaccess.de/webview/getdata.php?cid=19933
    log_response: true
    scan_interval: 30
    form_submit:
      submit_once: true
      resource: https://www.bayrol-poolaccess.de/webview/p/login.php
      select: '#form_login'
      input:
        username: # Mailadress
        password: # Password
        autologin: None
    sensor:
      - unique_id: Pool PH Wert
        name: Pool PH Wert
        select: 'body > div > div:nth-child(2) > h1'
      - unique_id: Redoxwert
        name: Redoxwert
        unit_of_measurement: "mV"
        select: 'body > div > div:nth-child(3) > h1'
      - unique_id: Pooltemperatur
        name: Pooltemperatur
        unit_of_measurement: "°C"
        select: 'body > div > div:nth-child(4) > h1'

Tomthebest1 · November 19, 2023, 6:30pm

It’s working fine! Thanks

Tomthebest1 · December 10, 2023, 10:14pm

Hi
Stop working after one day. Do you have any solution for this?
Thanks

Silvester5 · January 2, 2024, 1:56pm

They changed something on their homepage, now this works for me:

multiscrape:
  - name: Bayrol
    resource: https://www.bayrol-poolaccess.de/webview/getdata.php?cid=19933
    log_response: true
    scan_interval: 30
    form_submit:
      submit_once: true
      resource: https://www.bayrol-poolaccess.de/webview/p/login.php
      select: '#form_login'
      input:
        username: # Mailadress
        password: # Password
        autologin: None
    sensor:
      - unique_id: Pool PH Wert
        name: Pool PH Wert
        select: 'body > div > div.tab_data_link > div:nth-child(2) > h1'
      - unique_id: Redoxwert
        name: Redoxwert
        unit_of_measurement: "mV"
        select: 'body > div > div.tab_data_link > div:nth-child(3) > h1'
      - unique_id: Pooltemperatur
        name: Pooltemperatur
        unit_of_measurement: "°C"
        select: 'body > div > div.tab_data_link > div:nth-child(4) > h1'

ChicktheChicken · January 27, 2024, 10:55am

Thanks for your fantastic help! Unfortunately my sensors are unknown.

Is it still working for you?

Tomthebest1 · February 22, 2024, 4:15pm

I changed the recource line to

resource: https://www.bayrol-poolaccess.de/webview/p/device.php?c=XXXXX

The XXXXX is your device.

Now its working!
Thanks

Tomthebest1 · February 22, 2024, 4:26pm

Sorry. Worked only for some minutes.
Any suggestion?

ChicktheChicken · February 25, 2024, 2:45pm

I suspect that the paths on the website have changed.

harb70 · April 22, 2024, 4:03pm

Hi, I’m using MQTT to scrape data with Nodered.then post to Home Assistant MQTT broker.

PierreJDot · May 5, 2024, 6:40am

How exactly did you implement this with mqtt? I am looking for a way to query the status of the pump or the flow switch. In a knx forum this was described as a module which can read and set all states.

harb70 · May 12, 2024, 1:04pm

Sorry for the delay.
You can connect Bayrol Mqtt server.
When you open the web app, you can find which user/password is used.(dev tools)
wss://www.bayrol-poolaccess.de:8083/
With these informations, I configured a mqtt in node reading messages, then I write to mosquito broker in HA.

For now I just worked on 4 metrics… it could be great to be several to reverse engineer all the commands

harb70 · May 12, 2024, 2:21pm

connect to the web portal
clic on “Direct Access”
on dev tools, clic network, search for the websocket connexion wss://www.bayrol-poolaccess.de:8083/
Clic on Messages and find the 1st one
you will find the username to use.

You can test with a MQTT tool like MQTT Explorer

Dev tool screenshot (Red line is the username to copy)

MQTT Explorer connexion setup

d02//v/4.82 redox
d02//v/4.98 temperature
d02//v/4.100 salt
d02//v/4.182 pH

Duntch144 · May 18, 2024, 9:38am

Hi, I’m trying to find a way to connect Bayrol server. With procedure explained I’m connected to MQTT Bayrol server via MQTT Explorer. Nothing is displayed in MQTT Explorer. How to retrieve data from the server ?

harb70 · May 20, 2024, 9:40am

You need to subscribe to d02/« ID »/v/#

Duntch144 · May 20, 2024, 6:01pm

Great it works now !

PierreJDot · May 20, 2024, 6:23pm

Thanks for your reply. I found a very similar way to implement the bayrol server. I created diffrent ha sensors in node red.

11.31 Flow Sensor
11.33 Poolpump Sensor
4.78 pH (for me, maybe because its the bayrol automatic cl/ph and not salt)

harb70 · May 22, 2024, 3:02pm

Great !,
I’m trying to reverse engineer the code of the application.
And i’m about to find all functions !! (i hope)

exemple :

const KIND = {
subscribe: ‘v’,
publish: ‘s’,
request: ‘g’,
values: ‘v’
};

That means

d02/DeviceID/v/4.82
and to activate a setting it will be a message like that
/d02/DeviceID/s/xxxx

mad1812 · May 28, 2024, 1:59pm

Hi!
Can you please provide your switch-configuration.(The Node-RED-Switch from your Screenshot)
I dont get the trick.

Duntch144 · May 29, 2024, 6:25pm

Hi, I’m using an other method to process Bayrol MQTT data with a Python script and Paho as a MQTT bridge to transfer data to my local Mosquitto broker.

Here is the file. Just need to modify :
Bayrol Websockets site login (websockets server login with @harb70 procedure
Id Bayrol Equipment to subscribe to the correct topic
Local Broker IP Adress / Login and Password to connect to your Broker (Mosquitto for me). Port to be adjusted if necessary

Need to install Paho MQTT

All data received from Bayrol are automatically transfered.to the local broker.
Restrictions : This scripts works only in 1 way from Bayrol to your broker.

import paho.mqtt.client as mqtt
import time

# Callback function when connecting to the source broker
def on_connect_source(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to the source broker with result code " + str(rc))
        client.subscribe("d02/Id Bayrol Equipment/#")
    else:
        print(f"Failed to connect to the source broker with result code {rc}")

# Callback function when a message is received from the source broker
def on_message(client, userdata, msg):
    print("Message received: " + msg.topic + " " + str(msg.payload))
    forward_message(msg)

# Callback function when connecting to the destination broker
def on_connect_destination(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to the destination broker")
    else:
        print(f"Failed to connect to the destination broker with result code {rc}")

# Callback function for disconnecting from the destination broker
def on_disconnect_destination(client, userdata, rc):
    if rc != 0:
        print(f"Unexpected disconnection from the destination broker with result code {rc}")
    else:
        print("Disconnected from the destination broker")

# Forward the message to the destination broker
def forward_message(msg):
    forward_client.publish("Bayrol/" + msg.topic, msg.payload)
    print("Message sent: " + "Bayrol/" + msg.topic + " " + str(msg.payload))

# Initialize the destination MQTT client
forward_client = mqtt.Client()
forward_client.username_pw_set("Local Broker login", "Local Broker password")
forward_client.on_connect = on_connect_destination
forward_client.on_disconnect = on_disconnect_destination

# Connect to the destination broker
forward_client.connect("Local Broker IP", 1883, 60)
forward_client.loop_start()

# Initialize the source MQTT client with WebSockets
source_client = mqtt.Client(transport='websockets')
source_client.username_pw_set("Bayrol Websockets site login", "pass1")

source_client.tls_set()  # Use the system's default CA certificates
source_client.on_connect = on_connect_source
source_client.on_message = on_message

# Connect to the source MQTT broker via WebSockets
source_client.connect("www.bayrol-poolaccess.de", 8083, 60)
source_client.loop_forever()

# Stop the loop and disconnect properly when the script is interrupted
try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    source_client.loop_stop()
    source_client.disconnect()
    forward_client.loop_stop()
    forward_client.disconnect()

It would be great to share a file with V decoding where everyone can add. its own findings