mcfrojd
(Mcfrojd)
January 8, 2017, 4:59pm
1
Hi, im trying out the scrape sensor but i get no result, and i find no way to debug whats wrong.
Im trying this code for trying to get the latest gas price from a twitter feed.
The value “E85” repeats 10 times (i guess the resource contain the latest 10 tweets from the gas company)
And i want the first one (latest tweet) be a sensor like this entity=sensor.e85 state=10,99 (or what ever the latest price is)
sensor:
- platform: scrape
resource: https://twitter.com/hashtag/st1fagerstaflorav?f=tweets&vertical=default
name: E85
select: "E85"
value_template: '{{ value.split(":")[1] }}'
I dont know whats happen, i get no error and no sensor show up.
I guess maybe the “select:” and “value_template:” is all messed up?
If i look at the source of the latest tweet from the resource it looks like this:
<div class="js-tweet-text-container">
<p class="TweetTextSize js-tweet-text tweet-text" lang="no" data-aria-label-part="0">Prisuppdatering:
E85: 10,99
B95: 13,82
Diesel: 13,83
<a href="/hashtag/St1FagerstaFlorav?src=hash" data-query-source="hashtag_click" class="twitter-hashtag pretty-link js-nav" dir="ltr" ><s>#</s><b><strong>St1FagerstaFlorav</strong></b></a>
2017-01-05 09:33:29</p>
</div>
// McFrojd
1 Like
mcfrojd
(Mcfrojd)
January 10, 2017, 1:01pm
2
I gave up and made the scraping with tasker instead…
mcfrojd
(Mcfrojd)
March 26, 2017, 10:54am
3
Gave it another try.
Now scraping from a webpage instead of twitter and managed to figure ot how to do it.
#-------------------------------------------------------------
# Fagersta
#-------------------------------------------------------------
- platform: scrape
resource: https://bensinpriser.nu/stationer/95/vastmanlands-lan/fagersta
name: ST1 Floravägen Bensin Pris
select: "b:nth-of-type(5)"
- platform: scrape
resource: https://bensinpriser.nu/stationer/95/vastmanlands-lan/fagersta
name: ST1 Floravägen Bensin Datum
select: "small:nth-of-type(4)"
- platform: scrape
resource: https://bensinpriser.nu/stationer/etanol/vastmanlands-lan/fagersta
name: ST1 Floravägen Ethanol Pris
select: "b:nth-of-type(5)"
- platform: scrape
resource: https://bensinpriser.nu/stationer/etanol/vastmanlands-lan/fagersta
name: ST1 Floravägen Ethanol Datum
select: "small:nth-of-type(4)"
#-------------------------------------------------------------
# Avesta
#-------------------------------------------------------------
- platform: scrape
resource: https://bensinpriser.nu/stationer/95/dalarnas-lan/avesta
name: OKQ8 Bergsnäsvägen Bensin Pris
select: "b:nth-of-type(3)"
- platform: scrape
resource: https://bensinpriser.nu/stationer/95/dalarnas-lan/avesta
name: OKQ8 Bergsnäsvägen Bensin Datum
select: "small:nth-of-type(2)"
- platform: scrape
resource: https://bensinpriser.nu/stationer/etanol/dalarnas-lan/avesta
name: OKQ8 Bergsnäsvägen Ethanol Pris
select: "b:nth-of-type(3)"
- platform: scrape
resource: https://bensinpriser.nu/stationer/etanol/dalarnas-lan/avesta
name: OKQ8 Bergsnäsvägen Ethanol Datum
select: "small:nth-of-type(2)"
Explanation to the swedish words:
Fagersta = Nerby town where i work.
ST1 Floravägen Bensin = ST1 Gasstation Gasoline
ST1 Floravägen Ethanol = ST1 Gasstation Ethanol
Avesta = My Hometown
1 Like
mcfrojd
(Mcfrojd)
March 26, 2017, 10:54am
4
Now to a new problem:
The scraping of gasprice returns a value that is formatted like “13,45kr”
I guess that this is classified as a string.
When i try to format it to a number (i want to do some calculations with it) and tried to do like this:
{{ states.sensor.st1_floravagen_ethanol_pris.state|replace("kr", "")|replace(",", ".") }}
Witch transforms the value from “13,45kr” to “13.45”
But then when i try to use that value in a calculation it seems that the value might still be a string because its not possible to make a calculation with it.
If i add “|int” to it the value becomes “13” and i want those 2 decimals.
Found out that i had to add |float
to it.
Now it works to calculate with it.
Hello,
I think you could help me because I’ve got a problem…
I don’t understand the signification of b:nth-of-type(5)
Where do you find this ?
thank you !
mcfrojd
(Mcfrojd)
May 11, 2017, 10:56am
6
I must admit that it was pure luck and a lot of googling and other threads in this forum.
Pure trile and error…
Oh, ok !!
Actually it’s just the (5)
that I can’t find !
rpitera
(Robert Pitera)
May 11, 2017, 12:21pm
8
The Scrape Sensor uses BeautifulSoup to do the scraping; this may help:
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors
mcfrojd
(Mcfrojd)
May 11, 2017, 12:56pm
9
The 5 means it’s the 5th occurrence of what is searched for, if I have understood right.
PetrolHead
(Petrol Head)
October 17, 2017, 4:36pm
10
BUMP!
I solved this with a 2 part solution.
First:
Scrape the last tweets from “ST1Sverige”:
platform: scrape
resource: https://twitter.com/St1Sverige
name: ST1 Diesel Tweet
select: “p:nth-of-type(3)”
value_template: >-
{% if value.find("#St1YOURSTATIONHASH ") > 0 %}
{{ value[value.find(“Diesel”) + 8:]|truncate(5, True, “”)|replace(",",".")}}
{% else %}
{{ 0 }}
{% endif %}
unit_of_measurement: ‘Kr’
Then in a automation update any sensor, for example an MQTT sensor, with the new Diesel price when ST1_Diesel_Tweet triggers and is above 0.
1 Like
merola
(Bernardo)
April 6, 2019, 9:56pm
11
I tried the Gas Prices Examples, and it’s working perfect.
Now i’ll want to try something else… free parking spaces… but i’m doing something wrong here… need some help.
The website i want to scrape is:
https://www.enschede.nl/parkeren-vervoer-bereikbaarheid
the code: