Scrape - Unable to fetch data from website - Used to work though

Configuration
1

I am running HASS 0.66 in a docker on an i7 dell laptop. I am pretty sure this was working just a couple weeks ago and then it stopped. I am trying to pull/scrape prices from HomeDepot.com website. Below is the sensor I have in my sensors.yaml file:

That is generating the following error though:
2018-04-03 07:19:41 WARNING (MainThread) [homeassistant.components.sensor] Setup of platform scrape is taking over 10 seconds.
2018-04-03 07:19:41 ERROR (SyncWorker_16) [homeassistant.components.sensor.rest] Error fetching data: <PreparedRequest [GET]>
2018-04-03 07:19:41 ERROR (SyncWorker_16) [homeassistant.components.sensor.scrape] Unable to fetch data from https://www.homedepot.com/p/Andersen-70-1-2-in-x79-1-2-in-200-Series-White-Left-Hand-Perma-Shield-Gliding-Patio-Door-with-Built-In-Blinds-and-White-Hardware-PSBBGLWH/207111349

Any chance someone else can try that sensor and see if it works for them? Any suggestions/thoughts on why its not working for me anymore?

Thanks in advance for any and all help!

2

In case anyone else runs into this, for me it was related to the lack of a User Agent string in the scrape.py component. A helpful guy on Reddit offered me a work around that seems to work. I copied the scrape.py inside HASS to custom_components/sensor and made the following edit:

Change line 52 from:
payload = headers = None

to two lines:
payload = None
headers = {‘User-Agent’: ‘Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)’}

Once I did that and restarted HASS, my HomeDepot scraping sensors started working again.

2 Likes