Scrape sensor improved - scraping multiple values

jeti · November 2, 2022, 4:39pm

thanks! I do not have a log entry for not finding the form so I guess this works? but searching the website I found a different approach, without the login, which I prefer:
therefore I need to scrape the id of the first article in the left-area (there will be ne articles on top with new ids and that is the new information I am looking for):

<div id="et-main-area">
	
<div id="main-content">
	<div class="container">
		<div id="content-area" class="clearfix">
			<div id="left-area">
		
					<article id="post-4334" class="et_pb_post post-4334 lmv_event type-lmv_event status-publish hentry">

				
															<h2 class="entry-title"><a href="https://muc.lebensmittelretter.org/lmv_event/lmv_event_4334/">lmv_event_4334</a></h2>
					
					<p class="post-meta"> von <span class="author vcard">Christina S.</span> | <span class="published">Nov 2, 2022</span></p><p>Gemischte Kisten von Fiona</p>
				
					</article>
			
					<article id="post-4332" class="et_pb_post post-4332 lmv_event type-lmv_event status-publish hentry">

				
															<h2 class="entry-title"><a href="https://muc.lebensmittelretter.org/lmv_event/lmv_event_4332/">lmv_event_4332</a></h2>
					
					<p class="post-meta"> von <span class="author vcard">Irene U.</span> | <span class="published">Nov 2, 2022</span></p><p>Gemüse und Obst</p>
<blockquote><p>Bitte seid pünktlich</p></blockquote>
<p>&nbsp;</p>
<p>&nbsp;</p>
				
					</article>

could this be done?

Rharo · November 9, 2022, 12:16am

Hi,
Thank s for the code! I’m getting the information in the sensor but I can’t manage to add the information as Grid consumption in the Energy section of Home Assistant. The sensor is not displayed. Have you tried to add it there?

joem · November 9, 2022, 12:33am

Never tried. Adding stuff to the energy monitoring is not the easiest thing, if it is not supported out of the box.

parautenbach · November 9, 2022, 5:16am

That’s off topic. There are many posts about adding a device to energy management. You’ll have to dive in there. Also read this: Frequently Asked Questions about home energy management - Home Assistant.

DJI · November 21, 2022, 2:05pm

Can anyone help me set an icon that depends on the new value of a sensor compared to its previous value? I’ve tried many versions of the following, none working:

      - unique_id: Loc_tide_now
        name: Current height
        select: "#left-col > p:nth-child(3)"
        value_template: '{{ (value.split("approximately ") [1]).split("m")|first|float }}'
        unit_of_measurement: "m"
        icon: >-
          {% if (value > (numeric_state.sensor.Loc_tide_now | float)) %}
            mdi:arrow-up
          {% elif (value < (state.sensor.Loc_tide_now | float)) %}
            mdi:arrow-down
          {% else %}
            mdi:arrow-left-right
          {% endif %}

parautenbach · November 23, 2022, 6:14am

The way you access state values are completely wrong. Read the templating docs and use the states function.

The next problem is: While your sensor is updating what would the current value of the sensor be at the time the template for the icon is evaluated? It could be old or new. Regardless, the devs discourage self-references in templates (because it can lead to processing loops).

Perhaps look into creating a second sensor by making a trigger-based template sensor. You will then have explicit access to the from and to states. Here’s an example.

Another option is to make another kind of second sensor that tracks the change of the main sensor. This can be done using the derivative integration or just your own calculation in a template sensor. You can then use that sensor in the icon template.

Note: None of this has much to do with the scraping sensor config (not directly) but everything with templating. It’s often best to start a new topic or find a related post on the topic.

DJI · November 23, 2022, 9:54am

Thanks. I have a great amount to learn.

bsp9493 · November 29, 2022, 7:23pm

Ok, I’ve read and I’ve read and can’t seem to get past this, so I must be missing something.

scrape sensor appears to login fine, but will NOT find my “select”

2022-11-29 10:58:55.355 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # page_soup written to file: page_soup.txt
2022-11-29 10:58:55.356 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Data succesfully refreshed. Sensors will now start scraping to update.
2022-11-29 10:58:55.356 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 18.814 seconds (success: True)
2022-11-29 10:58:55.603 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # Daily Water Consumption # Setting up sensor
2022-11-29 10:58:55.735 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # Daily Water Consumption # Start scraping to update sensor
2022-11-29 10:58:55.738 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Daily Water Consumption # Tag selected: None
2022-11-29 10:58:55.738 DEBUG (MainThread) [custom_components.multiscrape.form] Scraper_noname_0 # Exception occurred while scraping, will try to resubmit the form next interval.
2022-11-29 10:58:55.738 WARNING (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # Daily Water Consumption # Unable to scrape data: Could not find a tag for given selector
Consider using debug logging and log_response for further investigation.
2022-11-29 10:58:55.746 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # Daily Water Consumption # On-error, keep old value: None
2022-11-29 10:58:55.746 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_0 # Daily Water Consumption # Updated sensor and attributes, now adding to HA

I have tried 2 different options in my select, both give the same result.

#your-barnacle > div > dl > dd:nth-child(4) > div.compound-read-low > ul > li:nth-child(3)

.compound-read-low > ul:nth-child(1) > li:nth-child(3)

I am getting these selects from either Chrome, or Firefox respectively.

The inspection looks as follows… the 335.16 is what I would expect to be returned.

<div id="your-barnacle" class="pod">
          <h4 class="pod-header">Your Meter</h4>
          <div class="pod-body">
            <figure>
              <div class="media">
                <img id="barnacle-big" src="/html5/images/barnacle-residential-big.png">
              </div>
              <figcaption>
                Your meter measures the quantity of water used in your household. Flow is measured electronically at hourly intervals, and updated every 24 hours for billing and leak detection.
              </figcaption>
            </figure>
            <dl class="meter-stat">
              <dt class="update"><i class="icon-drop-clock"></i>Read Frequency</dt>
              <dd class="update">Daily</dd>

              <dt class="update"><i class="icon-drop-reads"></i>Meter Read</dt>
              <dd class="update">
                <div class="compound-read-low">
                  <ul>
                    <li></li>
                    <li>6:44 AM on Nov 29</li>
                    **<li>335.16 CM</li>**
                  </ul>
                </div>

Help would be appreciated:)

vitto79 · December 3, 2022, 11:06am

Hi everybody
I am trying to get the “Dieselpreis aktuell” value from this page:

https://www.spritvergleich.at/details?tid=184924

The problem is, that the value “1.684” of the span ‘preisd-a’ is a composition of 4 images, where the value of the 4 digits could be obtained by concatenation of a part of the .PNG file names.

images/numbers/g1.png
images/numbers/g6.png
images/numbers/g8.png
images/numbers/g4.png

Is that possible to achieve?

Crashbash2020 · December 5, 2022, 10:22am

no worries mate, only just got around to looking at this again today, between what you and @xbmcnut have done im sure ill be able to get it going

Cheers!

pidev · December 22, 2022, 4:18pm

Hello Matteo,

I see you are using the same “Aton” system as me.
I ask you if you were able to access with multiscrape and then obtain the energy production data etc.

Thank you

sloma · January 8, 2023, 11:40am

Hi everybody,

I am trying to scape unit price from my energy provider:
PowerNI

- resource: https://powerni.co.uk/compare-electricity-ni/unit-rates/
  scan_interval: 3600
  log_response: true
  sensor:
    unique_id: powerni_unit_price
    name: PowerNI Unit Price - Energy Online
    select: "div.table-scroll:nth-child(5) > table:nth-child(1) > tbody:nth-child(3) > tr:nth-child(1) > td:nth-child(4)"
    unit_of_measurement: "p"
    value_template: '{{ (value.split("p")[0]) }}'

but I get the error in the log:


This error originated from a custom integration.

Logger: custom_components.multiscrape.sensor
Source: custom_components/multiscrape/sensor.py:175
Integration: Multiscrape scraping component (documentation, issues)
First occurred: 11:00:09 (1 occurrences)
Last logged: 11:00:09

Scraper_noname_0 # PowerNI Unit Price - Energy Online # Unable to scrape data: Could not find a tag for given selector Consider using debug logging and log_response for further investigation.

I used Firefox to select and inspect the element and then copied CSS Selector value into sensor config.

Koal4 · January 8, 2023, 2:01pm

Anyone have a tip how I could scrape a forum where the subject can drop to other pages?

vellanix · January 10, 2023, 11:23am

I need some help with the “select” option in multiscrape.
I tried different tag but withour any success.

I want to extract the “219.08” and the “202.43” and outlined by the red arrow.

Who can help me with the select?

Thank you

danieldotnl · January 10, 2023, 12:47pm

Not sure for which element you copied the CSS selector. Try this: .green-outline > td:nth-child(4)

danieldotnl · January 10, 2023, 12:50pm

Interesting
You can scrape the src attribute of the img element.
You will probably need to add 4 different selectors and then concatenate them with a template sensor.

sloma · January 10, 2023, 5:15pm

I am not on Eco-Energy tariff, but changing to: .pink-outline > td:nth-child(4) finally made it work. Thanks a lot.

malko · January 10, 2023, 7:11pm

Hello everyone,

Thanks to Daniel I use the great multiscrape integration for Home Assisant to pull the movies playing out of my local theater, the local gas price around me…

I am trying to get the href att of an anchor tag or other attributes like the url att of a picture without any success.

I need to extract the parameter: “card=189203” from the href below

<div class="node">
    <div class="field_date_contenu">
        <span class="day">10</span>
        <span class="month">Janvier</span>
        <span class="year">2023</span>
    </div>
    <div class="field_contenu">
        <div class="title">
               <a href="./details-seance.aspx?card=189210">The movie title</a>
        </div>
        <div class="field-name-field-adresse-unique">
                                21h
              <br>Biopic, drame
        </div>
    </div>
</div>
<div class="node">
   ...
</div>

I use the select_list: filter to get all the data in one column that I store in an attribute of the sensor.

Here is how I use multiscraper :

multiscrape:
  - name: cinema
    resource: "https://url.aspx"
    scan_interval: 7200
    sensor:
      - unique_id: cinema_prog
        name: "Movies playing"
        value_template: sensor.time
        attributes:
          - name: title
            select_list: ".node .field_contenu .title a"                 ## works great
            value_template: |
             {%-set value = value.split(",")-%}
             {%for x in value%}
             - {{x}}
             {%-endfor-%}
        
          - name: link
            select_list: ".node .field_contenu .title a::attr(href)"    ## Cannot get the href att
            value_template: |
              {%-set value = value.split(",")-%}
              {%for x in value%}
              - {{x}}
              {%-endfor-%}

Does any one has an idea on how to achieve this and get any attribut of an HTML tag with multiscrape ?

Thanks for your help

danieldotnl · January 10, 2023, 8:34pm

Try this:

- name: link
  select_list: ".node .field_contenu .title a"
  attribute: href

malko · January 10, 2023, 8:45pm

Thank you so much Daniel it works great !

Is there an option to replace the “,” separator by ‘|’

Catching strings when using select_list: is very tricky and also a problem with numbers, because stupid French people (like me) use ‘,’ instead of ‘.’ in numbers