Scrape sensor: error on device

Installation Home Assistant OS
1

Hi guys,

I am trying to scrape the gold and silver spot price from this URL (http://www.perthmint.com/metalprices.aspx). the data that I need is within this section:

<span id="lblGoldAskAU">$2,712.42</span>
              </td>
              <td>
                <span id="lblGoldBidAU">$2,698.34</span>
              </td>
              <td>
                <span id="lblGoldAskUS">$1,717.23</span>
              </td>
              <td>
                <span id="lblGoldBidUS">$1,708.86</span>
              </td>
            </tr>
            <tr>
              <td>Silver</td>
              <td>
                <span id="lblSilverAskAU">$24.62</span>
              </td>
              <td>
                <span id="lblSilverBidAU">$24.08</span>
              </td>
              <td>
                <span id="lblSilverAskUS">$15.59</span>
              </td>
              <td>
                <span id="lblSilverBidUS">$15.25</span>
              </td>

Just to test, I have targeted to obtain the gold spot price so I have the following code in home assistant.

  - platform: scrape
    resource: http://www.perthmint.com/metalprices.aspx
    name: Gold spot price
    select: 'lblGoldAskAU".*'
    value_template: '{{ value }}'

I gather that if I am able to return the rest of the string after first scrapping the 'lblGoldAskAU", then I am able to obtain the actual dollar value in that string.

It is not working and HA said there is an error on device update. I am happy to print the log error here but could someone please tell me if my setting should be changed in the first place?

Thanks!

2

Hi
I would suggest to have a look here.

I am not ready to guide you through this but my recommendation is to install python and some addons which are necessary (beatifulsoup, resources)

and run the below code in order to be sure that it is working before you add the sensor in HA.

#!/usr/bin/python3

from bs4 import BeautifulSoup
import requests

# Change these 2 things
URL="http://stravon.gr/pinakas/meteostation/maroussi_m"
# This is the select line you will use in the config
SELECT=".pricetab > .price > h2"
# You may need to use a template after the fact...
INDEX=2

r = requests.get(URL)
data=r.text


soup = BeautifulSoup(data)

#print(soup)

val = soup.select(SELECT)

print("Output of SELECT: **************")
print(val)
print("**************")

value = val[INDEX].text
print(value)
3

Thanks Makis. I will look into that resource that you point me to.

4

I tried ir in python and this works for

URL=“http://www.perthmint.com/metalprices.aspx”

SELECT="#lblGoldAskAU"

INDEX=0

the output is = $2,712.42

  • please bear in mind that sometimes although it is working in python HA returns value unknown.
    I don’t know why.
5

Thanks Makis. Good to know I am on the right track. I have yet to install python but will certainly do so today.

6

Hi Makis,

I tried to install python3 + pip + bs4 + requests in an ubuntu in docker and while phyton runs fine, all those scripts when run generate errors such as etc etc

I am not too familiar with this so this looks very foreign to me unfortunately.

But given you have previously already scrape the correct value using the SELECT scrape parameter that I originally suggested, how do i get that output from home assistant’s value_template?

7

I just tried to test the sensor in my HA but id doesn’t work.
I think you should fine another site (more friendly to scrappers) to get the data you need.
I don’t know why this is happening. As you saw I am new to this also.
It happened to me the python scrapper to work ok but HA couldn’t.
But only in 1 sensor from 5-6 I tried

sensor test
sensor test949×199 23.2 KB

In addition you should try to get the python scrapper to work otherwise you will end up restarting HA every minute :slight_smile:

8

Many thanks my friend for even doing that to help out. I will try other scrapping tool.

Thanks again! :slight_smile: