Dutch Gas Priceses

metbril · July 11, 2018, 12:01pm

I did some testing and found this script working more or less.

import requests
from bs4 import BeautifulSoup

# url = "https://www.unitedconsumers.com/tanken/informatie/brandstof-prijzen/euro95"
url = "https://www.unitedconsumers.com/brandstofprijzen"

page = requests.get(url)

soup = BeautifulSoup(page.text, 'html.parser')

fuels_list = soup.find(class_='table')

fuels = fuels_list.find_all(class_='row table-row')

for fuel in fuels:
	fuel_name = fuel.find('a').contents[0].strip()
	fuel_price = fuel.find(class_='col-xs-4 col-sm-4 text-right').contents[0]
	print(fuel_name, fuel_price)

This could be implemented with the scrape sensor or with a custom component. The latter looks like fun to me, and I will give that a try.

jay448 · July 11, 2018, 12:33pm

Nice that you actually might have something working. Let me know if this works. Too bad it’s the “advies” prijs and not the current price at a gas station. They usually differ quite a lot.

I tried debugging the Android App yesterday (Direct Lease Tank Service) to figure out what API they call but could not figure it out yet. I’m a complete noob at this stuff. I will check the website in a bit. Since you mentioned the JAVA script that is called.

metbril · July 11, 2018, 12:48pm

For future reference, from the other thread:

Original thread:

Scrape Shell Express

Get all stations from Directlease Tankservice

https://tankservice.app-it-up.com/Tankservice/v1/places?fmt=web

Station by image

https://tankservice.app-it-up.com/Tankservice/v1/places/2384.png?_t=1531026250202

United Consumers

metbril · July 11, 2018, 6:59pm

With some trial and error, I managed to retrieve the Euro 95 price with the scrape component:

homeassistant:
  customize:
    sensor.diesel:
      friendly_name: "Diesel"
      unit_of_measurement: "€/l"
      icon: mdi:gas-station
    sensor.euro95:
      friendly_name: "Euro 95"
      unit_of_measurement: "€/l"
      icon: mdi:gas-station
    sensor.lpg:
      friendly_name: "LPG"
      unit_of_measurement: "€/l"
      icon: mdi:gas-station

sensor:
  - platform: scrape
    name: Euro95
    resource: "https://www.unitedconsumers.com/brandstofprijzen"
    select: ".table div:nth-of-type(8)"
    value_template: "{{ value.split(' ')[1].replace(',', '.') | float }}"
    scan_interval: 3600 # be nice; once per hour only

  - platform: scrape
    name: Diesel
    resource: "https://www.unitedconsumers.com/brandstofprijzen"
    select: ".table div:nth-of-type(14)"
    value_template: "{{ value.split(' ')[1].replace(',', '.') | float }}"
    scan_interval: 3600 # be nice; once per hour only

  - platform: scrape
    name: LPG
    resource: "https://www.unitedconsumers.com/brandstofprijzen"
    select: ".table div:nth-of-type(20)"
    value_template: "{{ value.split(' ')[1].replace(',', '.') | float }}"
    scan_interval: 3600 # be nice; once per hour only

That was fun, and useful.

Update: while I’m still at it, I added the diesel and LPG.

metbril · July 11, 2018, 7:44pm

@Bob_NL This is fun… I managed to scrape the Euro 95 price of one of the cheapest gas stations near my home (and perhaps near you). It’s from TINQ.

homeassistant:
  customize:
    sensor.euro95_bermweg:
      friendly_name: "Euro 95 (Bermweg)"
      unit_of_measurement: "€/l"
      icon: mdi:gas-station

sensor:
  - platform: scrape
    name: Euro95 Bermweg
    resource: "http://www.tinq.nl/tankstation?id=251"
    select: "#station-prijzen tr:nth-of-type(3) td:nth-of-type(2)"
    value_template: "{{ value.split('\xa0')[1].replace('&nbsp;','').replace(',', '.') | float }}"
    scan_interval: 3600 # be nice; once per hour only

metbril · July 11, 2018, 8:16pm

@Bob_NL Your scrape sensor can be improved by making an actual value of it:

value_template: '{{ value.split(" ")[0] }}'

should be replaced with

value_template: '{{ value.split(" ")[0].replace(",", ".") | float }}'

jay448 · July 12, 2018, 9:23am

This is awesome man!

Combined it into a nice card for my Home Dashboard. Tinq is usually the cheapest near me aswell. Have to add a few more.

Thanks for the great developing

metbril · July 12, 2018, 10:53am

I have Lukoil scraped, too. Look at my repo in the ‘fuel’

https://github.com/metbril/home-assistant-config/blob/master/packages/finance/fuel.yaml

Joey_Germeraad · July 14, 2018, 2:23pm

@metbril Thnx for sharing you’re setup!

For people who want to add pricing from Tango I’ve added the needed select value for it.

platform: scrape
name: Euro95 tango
resource: “https://www.tango.nl/stations/XXX” #you’re desired location
select: “#euro95 .pump_price span.price”
scan_interval: 3600 # be nice; once per hour only

metbril · July 14, 2018, 4:08pm

Has anyone found a site with Avia prices?

The regular pages don’t show any:

https://avia.nl/tankstations/detail/avia-capelle-aan-den-ijssel/

Bob_NL · July 15, 2018, 3:26pm

Wow, what a coincidence. That is like a 5 minute drive from my home indeed. Small world!

Bob_NL · July 15, 2018, 7:40pm

I can’t get it to work (with or without the value_template).

  - platform: scrape
    resource: "https://www.tango.nl/stations/tango-nieuwerkerk-ad-ijssel"
    select: “#euro95 .pump_price span.price”
#    value_template: "{{ value.split(' ')[0].replace(',', '.') | float }}"
    scan_interval: 3600 # be nice; once per hour only

2018-07-15 20:49:26 ERROR (MainThread) [homeassistant.components.sensor] scrape: Error on device update!
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity_platform.py", line 248, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity.py", line 319, in async_device_update
    yield from self.hass.async_add_job(self.update)
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/sensor/scrape.py", line 120, in update
    value = raw_data.select(self._select)[0].text
IndexError: list index out of range

metbril · July 15, 2018, 7:49pm

Same with me. I did not have the tome to look into this, but the error was the same.

Bob_NL · July 17, 2018, 8:11am

@Joey_Germeraad @metbril
My knowledge is too limited to decipher this unfortunately, but I see the following lines in the source:
.
Does it work without the “#euro95” or with “pricing open” perhaps? At the moment I’m unable to test this myself.

Joey_Germeraad · July 17, 2018, 2:12pm

@Bob_NL

I’ll check you’re specific tango station tonight. It’s strange that the exact same setup with a different station is giving you errors. You mentioned it allready, but just checking, are you using the value_ template? In my setup the Tango.nl scraper would only work without the Value_template.

The main reason for adding the #euro95 is because the classes open and pricing are to generic, there are more divs on the page that have these classes which will result in Errors.

Bob_NL · July 17, 2018, 3:29pm

I tried both with and without the value_template.

Joey_Germeraad · July 18, 2018, 7:59pm

I just tried you’re specific location with the select value as mentioned and it’s resulting in a value without any issues or errors. For the comparison, I’m running Homeassistant 73.1 with Docker.

Bob_NL · July 19, 2018, 8:47pm

Finally, it’s working.
Somehow I ended up with faulty quotes in my config…

  - platform: scrape
    resource: https://www.shellexpress.nl/nl_nl/station/se-nieuwerkerk-aan-den-ijssel
    name: Shell Express
    select: ".price-actual"
    value_template: '{{ value.split(" ")[0] }}'
    scan_interval: 3600

  - platform: scrape
    name: Tinq
    resource: "http://www.tinq.nl/tankstation?id=251"
    select: "#station-prijzen tr:nth-of-type(3) td:nth-of-type(2)"
    value_template: "{{ value.split('\xa0')[1].replace('&nbsp;','').replace(',', '.') | float }}"
    scan_interval: 3600 

  - platform: scrape
    resource: "http://lukoil.nl/go/lukoil_station.cfm?id=465"
    name: Lukoil
    select: "#tabbedInfoContent div:nth-of-type(14)"
    value_template: "{{ value.split('\x0D')[2].replace('EUR/L', '').replace(',', '.') | float }}"
    scan_interval: 3600 

  - platform: scrape
    resource: "https://www.tango.nl/stations/tango-nieuwerkerk-ad-ijssel"
    name: Tango
    select: “#euro95 .pump_price span.price” #<<<<<<<<<< FAULTY QUOTES
    scan_interval: 3600

metbril · July 20, 2018, 6:56am

If you look closely to the original example, that’s were they came from.

Tuurie · January 26, 2019, 3:09pm

Using this for the last 3 months gave me some nice stats in Grafana, cleary showing how long they wait to drop the prices at Shell Express and how fast they raise the price.

# Benzine Prijzen
  - platform: scrape
    name: Euro95
    resource: "https://www.unitedconsumers.com/brandstofprijzen"
    select: ".table div:nth-of-type(8)"
    value_template: "{{ value.split(' ')[1].replace(',', '.') | float }}"
    scan_interval: 3600 # be nice; once per hour only

  - platform: scrape
    resource: "https://shellexpress.nl/nl_nl/station/se-santpoort"
    name: Shell_Express_Santpoort
    select: ".price-actual"
    value_template: "{{ value.split(' ')[0].replace(',', '.') | float }}"
    scan_interval: 3600

But United Consumers seem to have done something in their website code resulting in a 0.0 price.
I have tried to make cheese how this scrape works with the inspect option from Google Chrome but no luck.