Multiscrape - Help to recover values from my heating plant webportal

FuL75 · March 25, 2024, 1:53am

Hello to everybody. Is some months now that I try to scrape the values of my domestic heating/cooling system from the related web-portal to integrate the information in home assistant, unluckily the system that has its own plc is not integrable in Home Assistant.
This is the yaml file:

- name: IDROSISTEMI
  resource: "https://supervisione.idrosistemi.it/station/TGREG_1200#tab-zone"
  scan_interval: 60
  log_response: true
  headers:
    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
    referer: "https://supervisione.idrosistemi.it/"
    origin: "https://supervisione.idrosistemi.it"
    host: "supervisione.idrosistemi.it"
  form_submit:
    submit_once: true
    resource: "https://supervisione.idrosistemi.it/login"
    select: "#app > div > div.row > div > div > div.panel-body > form"
    input:
      email: "[email protected]"
      password: "xxxxxxxxx"
  sensor:
    - unique_id: t_soggiorno
      name: T soggiorno RADIANTE
      device_class: temperature
      unit_of_measurement: °C
      select: "#bt_z1 > a > span.badge.badge-light.ows-bool-remover > span"

Here below you can see the login page:

And this is the page were I want to extract the values from:

Log gives a positive feedback, but the sensor is “not available”

2024-03-25 02:37:42.091 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # The page with the form parsed by BeautifulSoup has been written to file: form_page_soup.txt
2024-03-25 02:37:42.091 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Try to find form with selector #app > div > div.row > div > div > div.panel-body > form
2024-03-25 02:37:42.093 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Form looks like this: 
<form action="https://supervisione.idrosistemi.it/login" class="form-horizontal" method="POST" role="form">
<input name="_token" type="hidden" value="qEl0SII7V3dL4BgLxpFyW2zzCnS63ZXLuHpr9wid"/>
<div class="form-group">
<label class="col-md-4 control-label" for="email">Indirizzo e-mail</label>
<div class="col-md-6">
<input autofocus="" class="form-control" id="email" name="email" required="" type="text" value=""/>
</div>
</div>
<div class="form-group">
<label class="col-md-4 control-label" for="password">Password</label>
<div class="col-md-6">
<input class="form-control" id="password" name="password" required="" type="password"/>
</div>
</div>
<div class="form-group">
<div class="col-md-6 col-md-offset-4">
<div class="checkbox">
<label>
<input name="remember" type="checkbox"/> Ricordami
                                    </label>
</div>
</div>
</div>
<div class="form-group">
<div class="col-md-8 col-md-offset-4">
<button class="btn btn-primary" type="submit">
                                    Login
                                </button>
<a class="btn btn-link" href="https://supervisione.idrosistemi.it/password/reset">
                                    Hai dimenticato la tua password?
                                </a>
</div>
</div>
</form>
2024-03-25 02:37:42.095 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Finding all input fields in form
2024-03-25 02:37:42.095 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Found the following input fields: {'_token': 'qEl0SII7V3dL4BgLxpFyW2zzCnS63ZXLuHpr9wid', 'email': '', 'password': None, 'remember': None}
2024-03-25 02:37:42.095 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Found form action https://supervisione.idrosistemi.it/login and method POST
2024-03-25 02:37:42.095 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Merged input fields with input data in config. Result: {'_token': 'qEl0SII7V3dL4BgLxpFyW2zzCnS63ZXLuHpr9wid', 'email': '[email protected]', 'password': 'xxxxx', 'remember': None}
2024-03-25 02:37:42.095 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Determined the url to submit the form to: https://supervisione.idrosistemi.it/login
2024-03-25 02:37:42.095 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Submitting the form
2024-03-25 02:37:42.095 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # Executing form_submit-request with a POST to url: https://supervisione.idrosistemi.it/login with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36', 'referer': 'https://supervisione.idrosistemi.it/', 'origin': 'https://supervisione.idrosistemi.it', 'host': 'supervisione.idrosistemi.it'}
2024-03-25 02:37:44.076 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # Response status code received: 200
2024-03-25 02:37:44.078 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # response_headers written to file: form_submit_response_headers.txt
2024-03-25 02:37:44.080 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # response_body written to file: form_submit_response_body.txt
2024-03-25 02:37:44.081 DEBUG (MainThread) [custom_components.multiscrape.form] IDROSISTEMI # Form seems to be submitted succesfully (to be sure, use log_response and check file). Now continuing to retrieve target page.
2024-03-25 02:37:44.081 DEBUG (MainThread) [custom_components.multiscrape.coordinator] IDROSISTEMI # Request data from https://supervisione.idrosistemi.it/station/TGREG_1200#tab-zone
2024-03-25 02:37:44.081 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # Executing page-request with a get to url: https://supervisione.idrosistemi.it/station/TGREG_1200#tab-zone with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36', 'referer': 'https://supervisione.idrosistemi.it/', 'origin': 'https://supervisione.idrosistemi.it', 'host': 'supervisione.idrosistemi.it'}
2024-03-25 02:37:44.083 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # request_headers written to file: page_request_headers.txt
2024-03-25 02:37:44.085 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # request_body written to file: page_request_body.txt
2024-03-25 02:37:44.734 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # Response status code received: 200
2024-03-25 02:37:44.737 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # response_headers written to file: page_response_headers.txt
2024-03-25 02:37:44.741 DEBUG (MainThread) [custom_components.multiscrape.http] IDROSISTEMI # response_body written to file: page_response_body.txt
2024-03-25 02:37:44.741 DEBUG (MainThread) [custom_components.multiscrape.scraper] IDROSISTEMI # Loading the content in BeautifulSoup.
2024-03-25 02:37:45.376 DEBUG (MainThread) [custom_components.multiscrape.scraper] IDROSISTEMI # page_soup written to file: page_soup.txt
2024-03-25 02:37:45.376 DEBUG (MainThread) [custom_components.multiscrape.coordinator] IDROSISTEMI # Data succesfully refreshed. Sensors will now start scraping to update.
2024-03-25 02:37:45.376 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 6.853 seconds (success: True)

Can somebody please suggest me where I’m wrong?

danieldotnl · March 28, 2024, 3:12pm

There should be more logging. It says that it retrieved the data, but the scraping still needs to take place.
Please also post page_soup.txt.

FuL75 · March 28, 2024, 5:48pm

sure, here you have form_page_soup.txt:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<!-- CSRF Token -->
<meta content="BfLK1kImcqmcWfpg7OmZ3kS5fFE7dn27NHWxvRng" name="csrf-token"/>
<!--   <title>s5a.eu</title>	 -->
<title>supervisione.idrosistemi.it</title>
<!-- Styles -->
<link href="/css/app.css" rel="stylesheet"/>
<!-- Scripts -->
<script>
        window.Laravel = {"csrfToken":"BfLK1kImcqmcWfpg7OmZ3kS5fFE7dn27NHWxvRng"}    </script>
</head>
<body>
<div id="app">
<nav class="navbar navbar-default navbar-static-top">
<div class="container">
<div class="navbar-header">
<!-- Collapsed Hamburger -->
<button class="navbar-toggle collapsed" data-target="#app-navbar-collapse" data-toggle="collapse" type="button">
<span class="sr-only">Toggle Navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<!-- Branding Image -->
<a class="navbar-brand" href="https://supervisione.idrosistemi.it">
<!--     <a class="navbar-brand" href="supervisione.idrosistemi.it">  -->
<!--      s5a.eu  -->
                              supervisione.idrosistemi.it
                    </a>
</div>
<div class="collapse navbar-collapse" id="app-navbar-collapse">
<!-- Left Side Of Navbar -->
<ul class="nav navbar-nav">
                         
                    </ul>
<!-- Right Side Of Navbar -->
<ul class="nav navbar-nav navbar-right">
<!-- Authentication Links -->
<li class="dropdown">
<a aria-expanded="false" class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">
                                    Fulvio <span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="https://supervisione.idrosistemi.it/associate">
                                            Associa nuovo impianto
                                        </a>
<a href="https://supervisione.idrosistemi.it/logout" onclick="event.preventDefault();
                                                     document.getElementById('logout-form').submit();">
                                            Esci
                                        </a>
<form action="https://supervisione.idrosistemi.it/logout" id="logout-form" method="POST" style="display: none;">
<input name="_token" type="hidden" value="BfLK1kImcqmcWfpg7OmZ3kS5fFE7dn27NHWxvRng"/>
</form>
</li>
</ul>
</li>
</ul>
</div>
</div>
</nav>
<div class="container">
<div class="row">
<div class="col-md-12 col-md-offset-1">
<div class="panel panel-default">
<div class="panel-heading">Elenco Stazioni</div>
<div class="panel-body">
<table class="table">
<tr>
<th><a href="https://supervisione.idrosistemi.it/home/statid">Codice</a></th>
<th><a href="https://supervisione.idrosistemi.it/home/descr">Descrizione</a></th>
<!--          	<th>Proprietario</th>          
                    	<th>Macrogruppo</th>   -->
<th><a href="https://supervisione.idrosistemi.it/home/groupid">Gruppo</a></th>
<th><a href="https://supervisione.idrosistemi.it/home/lastconn">Ultima Connessione</a></th>
<th><a href="https://supervisione.idrosistemi.it/home/subscrlevel">Sottoscrizione</a></th>
<th width="20%"><a href="https://supervisione.idrosistemi.it/home/subscrexpiry">Scadenza</a></th>
<th>Azioni</th>
</tr>
<tr>
<td><a href="https://supervisione.idrosistemi.it/station/TGREG_1200">TGREG_1200</a></td>
<td>FULVIO</td>
<td>CECERE                          </td>
<td>
<span class="badge" style="background-color: green">5 minuti</span>
</td>
<td>
                    Pro		</td>
<td><a href="https://supervisione.idrosistemi.it/stlist/renew/TGREG_1200%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20">2025-12-11</a></td>
<td>
<a href="https://supervisione.idrosistemi.it/stlist/edit/TGREG_1200%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20">
<span aria-hidden="true" class="glyphicon glyphicon-edit" title="Modifica proprietà impianto"></span>
</a>
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Scripts -->
<script src="/js/app.js"></script>
</body>
</html>

while page_soup.txt is too big, it doesn’t allow me to post it.

danieldotnl · March 28, 2024, 8:02pm

You didn’t post the missing part of the logging. What error does it contain?
Difficult to help without page_soup…