How to add cookies when scraping websites

Configuration
1

Hi everyone, i wanted to ask if it is possible to include a cookie when configuring a scrape sensor in HA.

I want to scrape the italian energy prices and display on the dashboard which are the cheaper hours to use electricity. The website provides a XML file including all the prices for every hour of the day but first it asks you to accept the terms agreement. How can i solve this problem?

For reference, this is the link reporting todays prices: GME - Gestore dei Mercati Energetici SpA

Thanks!

2

you could do something like this

  - resource: 'https://www.mercatoelettrico.org/It/Esiti/MGP/TabellaEsitiMGPPrezzi.aspx'
    name: PUN MPG 
    scan_interval: 30
    log_response: true
    form_submit:
      submit_once: True
      resource: 'https://www.mercatoelettrico.org/It/Tools/Accessodati.aspx?ReturnUrl=%2fIt%2fEsiti%2fMGP%2fTabellaEsitiMGPPrezzi.aspx'
      select: "#form1"
      input:
        'ctl00$ContentPlaceHolder1$CBAccetto1': 'true'
        'ctl00$ContentPlaceHolder1$CBAccetto2': 'true'
        'ctl00$ContentPlaceHolder1$Button1': 'Accetto'
    sensor:
      - select: '#gvFabbisogno > tbody:nth-child(1) > tr:nth-child(2) > td:nth-child(2)'
        name: 'MGP PUN ore 1'
        unit_of_measurement: '€/kWh'
        value_template: '{{ value | replace (",", ".") |float | int / 1000}}'
1 Like
3

Hi, thanks for the reply. I tried setting it up with ha-multiscrape but the state of the sensor is unavailable. I checked the contents of the page_soup.txt file and what i see is the contents of the form page.

The ha-multiscrape wiki says that if the data is not in that file, it is retrieved in the background by javascript.
How can i check if this is the case or if it is a config error?

4

Ok, after some trial and error i got it working.

This is the config i used:

multiscrape:
  - resource: 'https://www.mercatoelettrico.org/It/Esiti/MGP/TabellaEsitiMGPPrezzi.aspx'
    name: PUN MPG 
    scan_interval: 30
    log_response: true
    form_submit:
      submit_once: True
      resource: 'https://www.mercatoelettrico.org/It/Tools/Accessodati.aspx?ReturnUrl=%2fIt%2fEsiti%2fMGP%2fTabellaEsitiMGPPrezzi.aspx'
      select: "#form1"
      input:
        'ctl00$ContentPlaceHolder1$CBAccetto1': 'on'
        'ctl00$ContentPlaceHolder1$CBAccetto2': 'on'
        'ctl00$ContentPlaceHolder1$Button1': 'Accetto'
      input_filter:
        - 'ctl00$ContentPlaceHolder1$Button2'
        - 'ctl00$vai'
        - 'ctl00$LinkButton2'
        - 'ctl00$LoginButton'
    sensor:
      - select: '#gvFabbisogno > tr:nth-child(2) > td:nth-child(2)'
        name: 'MGP PUN ore 1'
        unit_of_measurement: '€/kWh'
        value_template: '{{ value | replace (",", ".") |float | int / 1000}}'
1 Like
5

Congratulations, for having found the right solution.
I have other ideas in mind in a few days I will contact you.

6

I have a problem with the YouTube page. When I try to read it, I get to the cookie consent page. I tried to consent to the cookie using the following code. Unfortunately, it doesn’t work.

multiscrape: