Scrape sensor - need help

Bob_NL · December 1, 2017, 7:11pm

Hi there,

I’m having a hard time scraping crypto currency values. I must admit not being familiair with Beautiful Soup at all.
I hope someone can help me out extracting certain values since the coinmarketcap component is broken since 0.55 or so.

For example; I would like to extract the Litecoin value from this webpage.

The following configuration is not working:

sensor:
  - platform: scrape
    resource: https://www.cryptocompare.com/coins/ltc/overview/USD
    name: LTC-USD
    select: 'price-value'

and creates the following errors in my log:

2017-12-01 20:03:40 ERROR (MainThread) [homeassistant.components.sensor] Error on device update!
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity_component.py", line 217, in async_add_entity
    yield from entity.async_device_update(warning=False)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity.py", line 306, in async_device_update
    yield from self.hass.async_add_job(self.update)
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/sensor/scrape.py", line 120, in update
    value = raw_data.select(self._select)[0].text
IndexError: list index out of range

I also tried the following without any luck:

select: ‘.price’
select: ‘price’
select: “price-value”
select: “.price-value”

Can someone help me out or point me in the right direction?

P.S. I was using the following for several weeks which was working fine but suddenly it stopped working (probably because now there is a ddos check before entering the website…(?))

sensor:
  - platform: scrape
    resource: https://www.coingecko.com/en/coins/litecoin
    name: LTC-USD
    select: ".coin-value"

fabaff · December 1, 2017, 11:44pm

The coinmarketcap sensor is working fine.

Anyway, would be simple to use the REST API than scraping the value from the website. E.g,

https://min-api.cryptocompare.com/data/pricemulti?fsyms=LTC&tsyms=USD

Bob_NL · December 2, 2017, 12:56am

I’m getting the following error:

2017-12-02 01:53:49 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up platform coinmarketcap
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity_component.py", line 171, in _async_setup_platform
    SLOW_SETUP_MAX_WAIT, loop=self.hass.loop)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 400, in wait_for
    return fut.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/sensor/coinmarketcap.py", line 55, in setup_platform
    CoinMarketCapData(currency, display_currency).update()
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/sensor/coinmarketcap.py", line 129, in update
    from coinmarketcap import Market
  File "/srv/homeassistant/lib/python3.5/site-packages/coinmarketcap/__init__.py", line 10, in <module>
    from .core import Market
  File "/srv/homeassistant/lib/python3.5/site-packages/coinmarketcap/core.py", line 6, in <module>
    import requests_cache
  File "/srv/homeassistant/lib/python3.5/site-packages/requests_cache/__init__.py", line 28, in <module>
    from .core import(
  File "/srv/homeassistant/lib/python3.5/site-packages/requests_cache/core.py", line 21, in <module>
    ver = tuple(map(int, requests.__version__.split(".")))
AttributeError: module 'pip._vendor.requests.__version__' has no attribute 'split'

Config:

sensor:
  - platform: coinmarketcap
    currency: bitcoin
    display_currency: EUR

fabaff · December 2, 2017, 8:54am

You need to clean-up your deps folder. Looks like that requests is out-dated.

Bob_NL · December 2, 2017, 10:23am

My deps folder seems empty:

Bob_NL · December 2, 2017, 11:11am

Thanks, I managed to get the data in a sensor:

sensor:
  - platform: rest
    resource: https://min-api.cryptocompare.com/data/pricemulti?fsyms=LTC&tsyms=USD
    name: Litecoin usd price
    value_template: "{{value_json['LTC']['USD']}}"
  - platform: rest
    resource: https://min-api.cryptocompare.com/data/pricemulti?fsyms=ETH&tsyms=USD
    name: Ethereum usd price  
    value_template: "{{value_json['ETH']['USD']}}"

gillrishi · February 3, 2018, 3:02am

Can you help me out .
I want to get value from this page
https://www.ptv.vic.gov.au/next5/diva/10001202/1
Departing in: _ mins

fabaff · February 3, 2018, 8:31am

Show us what you have so far.

gillrishi · February 3, 2018, 10:59am

platform: scrape
resource: https://www.ptv.vic.gov.au/next5/diva/10001202/1
name: train
select: '.departingIn

but its not working.

kayhanbelek · February 3, 2018, 6:17pm

try:

platform: scrape
resource: https://www.ptv.vic.gov.au/next5/diva/10001202/11
name: train
select: "departingIn"

gillrishi · February 3, 2018, 7:19pm

Sorry. Its not working.

kayhanbelek · February 3, 2018, 8:32pm

I think you need to use site APIs. The website may not allow direct access.

You can find API usage details here:
http://timetableapi.ptv.vic.gov.au/swagger/ui/index#/Departures

gillrishi · February 3, 2018, 10:15pm

Thanks for your reply. I email ptv to get dev id.

jivesinger · February 7, 2018, 8:13pm

I see 4 in there, which one were you after ?

If you use the restful sensor instead

value_template: {{ value_json.departures[0].estimated_departure_utc }}

should get you started

gillrishi · February 8, 2018, 4:26am

thanks for your reply.
its working with rest sensor.

platform: rest
resource: http://timetableapi.ptv.vic.gov.au/v3
name: API
value_template: “{{value_json[‘departures’][0][‘estimated_departure_utc’]}}”
thn i add this code to split the time value.
platform: template
sensors:
depart:
friendly_name: ‘depart’
value_template: ‘{{ states.sensor.api.state.split(“T”)[1] | replace (“Z”,"")}}’
now i have to solve next issue. Api page ouput time value is in UTC…
1 hour difference from my local time.
is it possible to subtract time value ?
or convert utc time output to local time.

jivesinger · February 10, 2018, 5:56am

Is your current output able to be parsed with the as_timestamp() function ?

A known workaround to this issue is to convert your time with as_timestamp(), then extract your time back out using timestamp_custom()

An example I use to calculate a last triggered time for a motion sensor.
I had the same issue in that the time returned was always UTC.
After using timestamp_custom, the time is now local time.

last_motion:
  friendly_name: "Last Motion Detected"
  value_template: '{{as_timestamp(states.automation.front_door_motion.attributes.last_triggered) | timestamp_custom("%H:%M    %d/%m/%y")}}'

gillrishi · February 11, 2018, 8:54am

Thanks a lot