Scrape Gas Prices

Configuration
1

Hi, im trying out the scrape sensor but i get no result, and i find no way to debug whats wrong.

Im trying this code for trying to get the latest gas price from a twitter feed.
The value “E85” repeats 10 times (i guess the resource contain the latest 10 tweets from the gas company)
And i want the first one (latest tweet) be a sensor like this entity=sensor.e85 state=10,99 (or what ever the latest price is)

sensor:
  - platform: scrape
    resource: https://twitter.com/hashtag/st1fagerstaflorav?f=tweets&vertical=default
    name: E85
    select: "E85"
    value_template: '{{ value.split(":")[1] }}'

I dont know whats happen, i get no error and no sensor show up.
I guess maybe the “select:” and “value_template:” is all messed up?

If i look at the source of the latest tweet from the resource it looks like this:

<div class="js-tweet-text-container">
<p class="TweetTextSize  js-tweet-text tweet-text" lang="no" data-aria-label-part="0">Prisuppdatering:
E85: 10,99
B95: 13,82
Diesel: 13,83
<a href="/hashtag/St1FagerstaFlorav?src=hash" data-query-source="hashtag_click" class="twitter-hashtag pretty-link js-nav" dir="ltr" ><s>#</s><b><strong>St1FagerstaFlorav</strong></b></a>

2017-01-05 09:33:29</p>
</div>

// McFrojd

1 Like
2

I gave up and made the scraping with tasker instead…

3

Gave it another try.
Now scraping from a webpage instead of twitter and managed to figure ot how to do it.

#-------------------------------------------------------------
# Fagersta
#-------------------------------------------------------------
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/95/vastmanlands-lan/fagersta
   name: ST1 Floravägen Bensin Pris
   select: "b:nth-of-type(5)"
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/95/vastmanlands-lan/fagersta
   name: ST1 Floravägen Bensin Datum
   select: "small:nth-of-type(4)"
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/etanol/vastmanlands-lan/fagersta
   name: ST1 Floravägen Ethanol Pris
   select: "b:nth-of-type(5)"
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/etanol/vastmanlands-lan/fagersta
   name: ST1 Floravägen Ethanol Datum
   select: "small:nth-of-type(4)"
#-------------------------------------------------------------
# Avesta
#-------------------------------------------------------------
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/95/dalarnas-lan/avesta
   name: OKQ8 Bergsnäsvägen Bensin Pris
   select: "b:nth-of-type(3)"
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/95/dalarnas-lan/avesta
   name: OKQ8 Bergsnäsvägen Bensin Datum
   select: "small:nth-of-type(2)"
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/etanol/dalarnas-lan/avesta
   name: OKQ8 Bergsnäsvägen Ethanol Pris
   select: "b:nth-of-type(3)"
-  platform: scrape
   resource: https://bensinpriser.nu/stationer/etanol/dalarnas-lan/avesta
   name: OKQ8 Bergsnäsvägen Ethanol Datum
   select: "small:nth-of-type(2)"

Explanation to the swedish words:

Fagersta = Nerby town where i work.
ST1 Floravägen Bensin = ST1 Gasstation Gasoline
ST1 Floravägen Ethanol = ST1 Gasstation Ethanol
Avesta = My Hometown

1 Like
4

Now to a new problem:

The scraping of gasprice returns a value that is formatted like “13,45kr”
I guess that this is classified as a string.
When i try to format it to a number (i want to do some calculations with it) and tried to do like this:

{{ states.sensor.st1_floravagen_ethanol_pris.state|replace("kr", "")|replace(",", ".") }}

Witch transforms the value from “13,45kr” to “13.45”

But then when i try to use that value in a calculation it seems that the value might still be a string because its not possible to make a calculation with it.

If i add “|int” to it the value becomes “13” and i want those 2 decimals.

  • EDIT -

Found out that i had to add |float to it.
Now it works to calculate with it.

5

Hello,

I think you could help me because I’ve got a problem…
I don’t understand the signification of b:nth-of-type(5)
Where do you find this ?

thank you !

6

I must admit that it was pure luck and a lot of googling and other threads in this forum.
Pure trile and error…

7

Oh, ok !! :slight_smile:

Actually it’s just the (5) that I can’t find !

8

The Scrape Sensor uses BeautifulSoup to do the scraping; this may help:

https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors

9

The 5 means it’s the 5th occurrence of what is searched for, if I have understood right.

10

BUMP!

I solved this with a 2 part solution.
First:
Scrape the last tweets from “ST1Sverige”:

  • platform: scrape
    resource: https://twitter.com/St1Sverige
    name: ST1 Diesel Tweet
    select: “p:nth-of-type(3)”
    value_template: >-
    {% if value.find("#St1YOURSTATIONHASH") > 0 %}
    {{ value[value.find(“Diesel”) + 8:]|truncate(5, True, “”)|replace(",",".")}}
    {% else %}
    {{ 0 }}
    {% endif %}
    unit_of_measurement: ‘Kr’

Then in a automation update any sensor, for example an MQTT sensor, with the new Diesel price when ST1_Diesel_Tweet triggers and is above 0.

1 Like
11

I tried the Gas Prices Examples, and it’s working perfect.
Now i’ll want to try something else… free parking spaces… but i’m doing something wrong here… need some help.

The website i want to scrape is:
https://www.enschede.nl/parkeren-vervoer-bereikbaarheid

the code: