Ha-multiscrape - Query data from a table

Configuration
1

Good Morning together,

I have been running my Home Assistant installation for more than a year and would now like to include my wallbox. To do this, I would like to use the
ha-multiscrape integration to read the current meter reading.

The configuration for this is as follows:

multiscrape:
  - name: Wallbox
    resource: https://192.168.2.36/cgi_s_ldp1.session
    scan_interval: 60
    verify_ssl: false
    form_submit:
      submit_once: True
      resource: https://192.168.2.36/
      input:
        username: admin
        password: "I will not publish that here ;)"
    sensor:
      - unique_id: Zaehlerstand
        name: Wallbox Zählerstand
        select: "body > div.sheet > div.main > div:nth-child(2) > table > tbody > tr:nth-child(20) > td:nth-child(2)"

The website looks like this:

2024-02-09 10_55_44-Window
2024-02-09 10_55_44-Window1746Ă—626 51.1 KB

Page: <!DOCTYPE html><html><head><title>ECU v2.3.0</title><meta http-equiv="co - Pastebin.com

I get the following error message:

2024-02-09 10:50:28.130 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Wallbox # New run: start (re)loading data from resource
2024-02-09 10:50:28.130 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Wallbox # Rendered resource template into: https://192.168.2.36/cgi_s_ldp1.session
2024-02-09 10:50:28.130 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Starting with form-submit
2024-02-09 10:50:28.130 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Skip scraping form, assuming all input is given in config.
2024-02-09 10:50:28.130 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Merged input fields with input data in config. Result: {'username': 'admin', 'password': 'I will not publish that here ;)'}
2024-02-09 10:50:28.131 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Determined the url to submit the form to: https://192.168.2.36/
2024-02-09 10:50:28.131 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Submitting the form
2024-02-09 10:50:28.131 DEBUG (MainThread) [custom_components.multiscrape.http] Wallbox # Executing form_submit-request with a POST to url: https://192.168.2.36/ with headers: {}
2024-02-09 10:50:28.219 DEBUG (MainThread) [custom_components.multiscrape.http] Wallbox # Response status code received: 200
2024-02-09 10:50:28.219 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Form seems to be submitted succesfully (to be sure, use log_response and check file). Now continuing to retrieve target page.
2024-02-09 10:50:28.219 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Wallbox # Request data from https://192.168.2.36/cgi_s_ldp1.session
2024-02-09 10:50:28.219 DEBUG (MainThread) [custom_components.multiscrape.http] Wallbox # Executing page-request with a get to url: https://192.168.2.36/cgi_s_ldp1.session with headers: {}
2024-02-09 10:50:28.238 DEBUG (MainThread) [custom_components.multiscrape.http] Wallbox # Response status code received: 200
2024-02-09 10:50:28.238 DEBUG (MainThread) [custom_components.multiscrape.scraper] Wallbox # Loading the content in BeautifulSoup.
2024-02-09 10:50:28.244 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Wallbox # Data succesfully refreshed. Sensors will now start scraping to update.
2024-02-09 10:50:28.244 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.114 seconds (success: True)
2024-02-09 10:50:28.244 DEBUG (MainThread) [custom_components.multiscrape.sensor] Wallbox # Wallbox Zählerstand # Start scraping to update sensor
2024-02-09 10:50:28.245 DEBUG (MainThread) [custom_components.multiscrape.scraper] Wallbox # Wallbox Zählerstand # Tag selected: None
2024-02-09 10:50:28.245 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Exception occurred while scraping, will try to resubmit the form next interval.
2024-02-09 10:50:28.245 ERROR (MainThread) [custom_components.multiscrape.sensor] Wallbox # Wallbox Zählerstand # Unable to scrape data: Could not find a tag for given selector 
Consider using debug logging and log_response for further investigation.
2024-02-09 10:50:28.245 DEBUG (MainThread) [custom_components.multiscrape.sensor] Wallbox # Wallbox Zählerstand # On-error, set value to None
2024-02-09 10:50:28.245 DEBUG (MainThread) [custom_components.multiscrape.entity] Wallbox # Wallbox Zählerstand # Sensor updated and state written to HA

Do you have any ideas on how I should adapt my selector? I don’t know what to do after I’ve been tinkering with it for 2 days now and have browsed through various forum entries.

Thank you for your support

Best regards,
Michael

2

There’s no tbody in that HTML. The DOM structure on the right-hand side of your screenshot is lying to you.

Also, there’s no need to be so specific. You’ll probably find that:

select: "tr:nth-child(20) > td:nth-child(2)"

will work. Just look at the HTML and work out how you’d describe the wanted element to someone: it’s the second cell (<td>) of the 20th row (<tr>).

3

Hi @Troon,

thank you very much for your fast reply :slight_smile:.
Unfortunately, the error persists even with your good advice.

2024-02-09 14:44:15.872 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Wallbox # New run: start (re)loading data from resource
2024-02-09 14:44:15.872 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Wallbox # Rendered resource template into: https://192.168.2.36/cgi_s_ldp1.session
2024-02-09 14:44:15.872 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Starting with form-submit
2024-02-09 14:44:15.872 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Skip scraping form, assuming all input is given in config.
2024-02-09 14:44:15.872 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Merged input fields with input data in config. Result: {'username': 'admin', 'password': 'xxxxxx'}
2024-02-09 14:44:15.873 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Determined the url to submit the form to: https://192.168.2.36/cgi_s_ldp1.session
2024-02-09 14:44:15.873 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Submitting the form
2024-02-09 14:44:15.873 DEBUG (MainThread) [custom_components.multiscrape.http] Wallbox # Executing form_submit-request with a POST to url: https://192.168.2.36/cgi_s_ldp1.session with headers: {}
2024-02-09 14:44:15.959 DEBUG (MainThread) [custom_components.multiscrape.http] Wallbox # Response status code received: 200
2024-02-09 14:44:15.959 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Form seems to be submitted succesfully (to be sure, use log_response and check file). Now continuing to retrieve target page.
2024-02-09 14:44:15.959 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Wallbox # Using response from form-submit as data. Now ready to be scraped by sensors.
2024-02-09 14:44:15.959 DEBUG (MainThread) [custom_components.multiscrape.scraper] Wallbox # Loading the content in BeautifulSoup.
2024-02-09 14:44:15.965 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.093 seconds (success: True)
2024-02-09 14:44:15.965 DEBUG (MainThread) [custom_components.multiscrape.sensor] Wallbox # Wallbox Zählerstand # Start scraping to update sensor
2024-02-09 14:44:15.966 DEBUG (MainThread) [custom_components.multiscrape.scraper] Wallbox # Wallbox Zählerstand # Tag selected: None
2024-02-09 14:44:15.966 DEBUG (MainThread) [custom_components.multiscrape.form] Wallbox # Exception occurred while scraping, will try to resubmit the form next interval.
2024-02-09 14:44:15.966 ERROR (MainThread) [custom_components.multiscrape.sensor] Wallbox # Wallbox Zählerstand # Unable to scrape data: Could not find a tag for given selector 
Consider using debug logging and log_response for further investigation.
2024-02-09 14:44:15.966 DEBUG (MainThread) [custom_components.multiscrape.sensor] Wallbox # Wallbox Zählerstand # On-error, set value to None
2024-02-09 14:44:15.966 DEBUG (MainThread) [custom_components.multiscrape.entity] Wallbox # Wallbox Zählerstand # Sensor updated and state written to HA
4

Just try select: "body" and see if you get the same error.

If that doesn’t help, do what the error message suggests:

# Form seems to be submitted succesfully (to be sure, use log_response and check file).

5

Thank you for your approaches @Troon
Sometimes it’s helpful just to write to someone. I have just been able to solve the problem - it was probably a problem with the authentication.

multiscrape:
  - name: Wallbox
    resource: https://192.168.2.36/cgi_s_ldp1.session
    scan_interval: 60
    verify_ssl: false
    form_submit:
      submit_once: True
      resource: https://192.168.2.36/
      select: "form"
      input:
        username: admin
        password: "xxxxxx"
    sensor:
      - unique_id: Zaehlerstand
        name: Wallbox Zählerstand
        select: "tr:nth-child(20) > td:nth-child(2)"

Here are a few more keywords in case someone else has the same problem and would like to include their Compleo / Innogy Wallbox professional Pro in Home Assistant over the LAN connected webinterface.

1 Like
6

(just looked up the German: Quietscheentschen is quite a mouthful!)

1 Like