Using Flask and Playwright to do more advanced scraping

Sigogglin · December 11, 2022, 12:30am

Not sure where I should post this, please move if this is the wrong place.

I have a Heatmaster outdoor wood boiler. It runs on a LOGO8 PLC and has a built in web server. I wanted to be able to monitor the status of the stove from HA. I tried the scrape integration, but the page is dynamically updated via javascript so it didn’t work.

So I made docker container, that contains flask (a web server) and playwright, and E2E web testing app, to scrape the web page and return JSON data on a GET request. This allowed me to use the RESTful integration to monitor my data.

If there is some way to do this without the need for a docker I’m all ears. Essentially I need the ability to say wait until param_1 is visible. I can do that with playwright.

So in case anyone else has a need to integrate a web into HA, and you need something more capable than scraper, you can use my docker as a base line. It would be very easy to clone and customize for your own webserver needs.

dpeart/heatmaster: expose Heatmaster wood boiler interface to REST (github.com)

Here is the configuration.yml for it:

rest:
  - scan_interval: 30
    resource: http://192.168.10.246:5000/
    sensor:
      - name: "Heatmaster Boiler Status"
        value_template: "{{ value_json.Heatmaster.status }}"
      - name: "Heatmaster Temperature"
        value_template: "{{ value_json.Heatmaster.temp }}"
        device_class: temperature
        unit_of_measurement: '°F'      
      - name: "Heatmaster O2"
        value_template: "{{ value_json.Heatmaster.O2 }}"
        unit_of_measurement: "%"
      - name: "Heatmaster Top Air"
        value_template: "{{ value_json.Heatmaster.Top_Air }}"
        unit_of_measurement: "%"
      - name: "Heatmaster Bottom Air"
        value_template: "{{ value_json.Heatmaster.Bottom_Air }}"
        unit_of_measurement: "%"

thanks,
david

superduck · December 26, 2022, 3:52am

Hey David, I have a working code snippet that talkes to the LOGO console directly and avoids screen grabbing. It hijacks the AJAX connection the UI uses. Ive been trying to create an integration but i dont have much available time.

I can send you the code if your interested.

Sigogglin · December 27, 2022, 9:26pm

That would be great. I wanted to do that, but just couldn’t figure out how.

thanks,

superduck · December 28, 2022, 2:57pm

Here is the module that does the “Work”:

github.com

LelandP/heatmaster_homeassistant/blob/a858d4f473849612cf42732ed9a497f212d7ecd8/custom_components/heatmaster_hassio/HeatmasterAjax.py

import binascii
import logging
from xml.dom.minidom import parseString
import requests


class HeatmasterAjax():
    """
    Heatmaster class uses the AJAX url to communicate and fetch data from the simens LOGO8
    """
    auth_cookie = None

    def __init__(self,
                 ip_address: str,
                 username: str = "Web User",
                 password: str = "heatmaster",
                 no_login: bool = False) -> None:
        """
        Creates a new HeatmasterAjax object.

This file has been truncated. show original

I have been working over the Xmas break trying to get this working as a homeassistant integration using HACS. Its my first time creating an integration so i’m working through the process :).

superduck · December 29, 2022, 1:56am

I managed to create a working integration if you want to check it out. Its very much in a beta stage but it does manage to create 5 entities for status, water temp, o2, top damper and bottom damper. You can check it out here.

Sigogglin · December 31, 2022, 11:04pm

unfortunately I run home assistant core, so I can load hacs. I’ve been able to manually load custom components before but I couldn’t get this to load.

I cloned to my container, then put the custom_container contents in my existing custom_container, but it didn’t like it.

thanks

Sigogglin · February 21, 2023, 2:56pm

I got HACS installed and got this installed as well. It is now up and running.

Do you mind adding bypass and timer as two more status settings?

I’ll shutdown my flask based solution.

Thanks!