UK oil price multiscrape assistance required

Share your Projects!
1

Hi,

Trying multiple sites to get the daily price for UK heating oil but never shows in HA.

image
image1853×1156 184 KB

I copied the selector using the dev tools.

  • name: Oil UK price scraper
    resource: Oxfordshire
    scan_interval: 3600
    sensor:
    • unique_id: homefuelsdirect_oxfordshire_lowest_oil_price
      name: Homefuelsdirect Oxfordshire Lowest Oil Price
      select: “#county-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(2)”
      unit_of_measurement: GBP
      value_template: ‘{{ value.split(“trPrice”)[1] }}’
      icon: mdi:gas-station

Thanks in advance

1 Like
2

image
image610×901 29.7 KB 

User-Agent: >-
  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like
  Gecko) Chrome/110.0.0.0 Safari/537.36

image
image469×908 30.8 KB

gives me:

image
image1056×355 17.3 KB

1 Like
3

Many many thanks for this buddy!

Can you give me some pointers so I can do this for myself in future?

Happy to buy you a coffee / beer to show my thanks :slight_smile:

4

I used the exact same path as you did, only added a header, this gave me the price with a " p"
With the split, I rmoeved the latter, could also do this with a substring alike func of course.
I did not use any class or measurement, this may (?) hamper accepting the collected data
You can now try to change the unit of measurement to see what happens
In short: start with minimal tuning until you have data, then dig into the details

5

Sorry to sound stupid but what tool are you using in the screenshots please?

6

Regular windows ‘snipping tool’
I have snagit as well for the more fancy but it takes longer to start :slight_smile:

7

My bad, I meant what tool have you screenshot? where you have the header box etc

8

euh… the regular ‘scrape’ integration that HA provides in devices… :thinking: what did you do?

9

I’m using Multiscrape buddy

10

As luck would have it, I live in Oxfordshire! and have used this thread to learn about scraping. It’s still a little confusing to me (getting the correct select and why the header is important), but here is the yaml if anyone is interested:

scrape:
  - resource: https://homefuelsdirect.co.uk/home/heating-oil-prices/oxfordshire
    headers:
      User-Agent: >-
        Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like
        Gecko) Chrome/110.0.0.0 Safari/537.36
    sensor:
      - name: HomeFuelsDirect 500l
        select: "#county-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(2)"
        value_template: "{{ value.split(' p')[0] }}"
        device_class: monetary
        icon: mdi:gas-station
        unit_of_measurement: p
      - name: HomeFuelsDirect 900l
        select: "#county-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(3)"
        value_template: "{{ value.split(' p')[0] }}"
        device_class: monetary
        icon: mdi:gas-station
        unit_of_measurement: p

Thanks to both @adigraham and @vingerha for their examples that helped me! :+1:

2 Likes
11

Looks great mate!

1 Like
12

Looks great and i have it working for lincolnshire, my question is how often does this update? do i need to set up and automation to carry out the scrape?

13

I believe it updates hourly by default, but you can change that with scan_interval. Here’s the start of my yaml for the met office…

scrape: #- restart required for changes ----------------------------------------

- resource:  !secret met_office_scrape
  #scan_interval: 600 # default
  sensor:
1 Like
14

You can also update the sensor (force it to scrape) manually with a service.

I sometimes use automations for this incase the scrape fails and returns unavailable (if I am scraping from an unreliable website))

    - service: homeassistant.update_entity
      target:
        entity_id:
          - sensor.homefuelsdirect_500l
15

Thank you!
Updated the script for my location and I’m logging prices (you do need to restart your HA server the first time you introduce the script).

It will be interesting to see how the pence per litre varies over the next month or so, given the cold spell and the situation in the middle east.

16

Hi, I know its a bit old, but I wanted to hack the ‘Gas’ part of the HA energy dashboard to record the oil usage I’m getting from my tank sensor. This is how I did it, I put it against the Github entry for the kingsman sensit without which I would have been stumped. But this thread was crucial too.
basically you work out your own numbers (in Wh for me) but then pretend they are m³. Its a shame that unit isnt allowed, and you cant relabel a few elements, cos if you could it would be 100%. but it works for me.

https://github.com/masaccio/ha-kingspan-watchman-sensit/issues/31