Adding a user agent as an option would be really handy.
I’ve had some websites block my scraping and I’m guessing because of being too frequent (my bad, I’ll add scan_interval: 1800 next time) but being able to add a browser agent would be really useful.
I’ve seen that cmille34 has converted scrape into a custom component, but for the lazy (like me) that don’t want to do that, it’d be great if it was built into the standard component 
I agree - this would be a great feature to add (and and easy one to merge in I think). Ironically, my scraping of www.homedepot.com has stopped working again even with the User Agent string. I tried switching to a different User Agent and it still isn’t working. Im seeing " failed with HTTPSConnectionPool(host=‘www.homedepot.com’, port=443): Read timed out. (read timeout=10)" from rest.py and then that of course causes the scrape.py to fail. I can pull up the website fine from the same docker host though.
It was a nice to have on my HA dashboard but not the end of the world.