Scrape sensor improved - scraping multiple values

Yes, just two different ways of doing the same thing. There are probably use cases where either way might be better.

I think I won’t be able to use the “Accuweather” card anyway. Although it has separate sensors for different things, I think I wasn’t seeing the info shown in the card in any of the sensors because on top of the sensors they create a different entity, not a sensor but a weather entity, which is not supported by multiscrape:

So I guess if there’s no workaround that, I’ll have to forget using that card.
Let’s setup first all sensors with the appropriate data :+1:, and then decide how to visualize it (or not, and just use automations)

1 Like

You can create a template weather entity referring to the sensors you scraped:

1 Like

Hi @danieldotnl,

Yes, that is an option I considered.
Also, I tried to use the Platinum Weather Card to directly map the values.

The issue is that these two options are both prepared for weather, and usually expect values like temperature (ºC), pressure, humidity, etc.

Its not set to map wave values (height, period, etc). At least I tried with the aforementioned card and it didn’t work.

For the moment I have all values in separate sensors, including some cool icon automations to indicate wind and wave direction, and I can write automations on top of. But I’ll have to re-think how to represent them in a Weather-like card.

I’ve been searching and I’m afraid I’d have to write my own custom card.

For anyone interested, here’s the code I finally used to scrape all forecast values of Wisuki:

- name: Wisuki scraper
  resource: https://wisuki.com/widget-summary?spot=2621
  scan_interval: 43200
  sensor:
    - unique_id: wisuki_waves
      name: Alibey
      icon: >-
          {% if is_state('binary_sensor.wisuki_waves_now', 'on') %}
            mdi:waves
          {% else %}
            mdi:wave
          {% endif %}
      value_template: "{{ iif(is_state('binary_sensor.wisuki_waves_now', 'on'), 'Go Surf!', 'Flat') }}"
    - unique_id: wisuki_wind_avg_0
      name: Viento
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(2)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_0
      name: Racha
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(2)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_0
      name: Cardinalidad
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(2) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_0
      name: Altura
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(2)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_0
      name: Periodo
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(2)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_0
      name: Dirección
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(2) > img"
      attribute: "alt"
    - unique_id: wisuki_wind_avg_1
      name: Viento 1d
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(3)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_1
      name: Racha 1d
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(3)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_1
      name: Cardinalidad 1d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(3) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_1
      name: Altura 1d
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(3)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_1
      name: Periodo 1d
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(3)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_1
      name: Dirección 1d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(3) > img"
      attribute: "alt"
    - unique_id: wisuki_wind_avg_2
      name: Viento 2d
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(4)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_2
      name: Racha 2d
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(4)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_2
      name: Cardinalidad 2d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(4) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_2
      name: Altura 2d
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(4)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_2
      name: Periodo 2d
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(4)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_2
      name: Dirección 2d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(4) > img"
      attribute: "alt"
    - unique_id: wisuki_wind_avg_3
      name: Viento 3d
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(5)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_3
      name: Racha 3d
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(5)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_3
      name: Cardinalidad 3d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(5) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_3
      name: Altura 3d
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(5)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_3
      name: Periodo 3d
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(5)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_3
      name: Dirección 3d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(5) > img"
      attribute: "alt"
    - unique_id: wisuki_wind_avg_4
      name: Viento 4d
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(6)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_4
      name: Racha 4d
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(6)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_4
      name: Cardinalidad 4d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(6) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_4
      name: Altura 4d
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(6)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_4
      name: Periodo 4d
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(6)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_4
      name: Dirección 4d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(6) > img"
      attribute: "alt"
    - unique_id: wisuki_wind_avg_5
      name: Viento 5d
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(7)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_5
      name: Racha 5d
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(7)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_5
      name: Cardinalidad 5d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(7) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_5
      name: Altura 5d
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(7)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_5
      name: Periodo 5d
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(7)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_5
      name: Dirección 5d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(7) > img"
      attribute: "alt"
    - unique_id: wisuki_wind_avg_6
      name: Viento 6d
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(8)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_6
      name: Racha 6d
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(8)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_6
      name: Cardinalidad 6d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(8) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_6
      name: Altura 6d
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(8)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_6
      name: Periodo 6d
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(8)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_6
      name: Dirección 6d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(8) > img"
      attribute: "alt"
    - unique_id: wisuki_wind_avg_7
      name: Viento 7d
      icon: mdi:weather-windy
      select: "tbody > tr:nth-child(3) > td:nth-child(9)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_gust_7
      name: Racha 7d
      icon: si:windicss
      select: "tbody > tr:nth-child(4) > td:nth-child(9)"
      state_class: measurement
      unit_of_measurement: kn
      device_class: wind_speed
    - unique_id: wisuki_wind_direction_7
      name: Cardinalidad 7d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down-thin
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left-thin
        {% elif value == 'E' %}
          mdi:arrow-left-thin
        {% elif value == 'ESE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'SSE' %}
          mdi:arrow-top-left-thin
        {% elif value == 'S' %}
          mdi:arrow-up-thin
        {% elif value == 'SSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'OSO' %}
          mdi:arrow-top-right-thin
        {% elif value == 'O' %}
          mdi:arrow-right-thin
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right-thin
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right-thin
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(2) > td:nth-child(9) > img"
      attribute: "alt"
    - unique_id: wisuki_wave_height_7
      name: Altura 7d
      icon: mdi:waves
      select: "tbody > tr:nth-child(7) > td:nth-child(9)"
      state_class: measurement
      unit_of_measurement: m
      device_class: distance
    - unique_id: wisuki_wave_period_7
      name: Periodo 7d
      icon: mdi:waves-arrow-up
      select: "tbody > tr:nth-child(8) > td:nth-child(9)"
      state_class: measurement
      unit_of_measurement: s
      device_class: duration
    - unique_id: wisuki_wave_direction_7
      name: Dirección 7d
      icon: >-
        {% if value == 'N' %}
          mdi:arrow-down
        {% elif value == 'NNE' %}
          mdi:arrow-bottom-left
        {% elif value == 'ENE' %}
          mdi:arrow-bottom-left
        {% elif value == 'E' %}
          mdi:arrow-left
        {% elif value == 'ESE' %}
          mdi:arrow-top-left
        {% elif value == 'SSE' %}
          mdi:arrow-top-left
        {% elif value == 'S' %}
          mdi:arrow-up
        {% elif value == 'SSO' %}
          mdi:arrow-top-right
        {% elif value == 'OSO' %}
          mdi:arrow-top-right
        {% elif value == 'O' %}
          mdi:arrow-right
        {% elif value == 'ONO' %}
          mdi:arrow-bottom-right
        {% elif value == 'NNO' %}
          mdi:arrow-bottom-right
        {% else %}
          mdi:windsock
        {% endif %}
      select: "body > table > tbody > tr:nth-child(6) > td:nth-child(9) > img"
      attribute: "alt"
  binary_sensor:
    - unique_id: wisuki_waves_now
      name: Surf
      value_template: >
        {% if states('sensor.wisuki_wave_heigth_0')|float(0) > 0.7 %}
          {{ "Go Surf!" }}
        {% else %}
          {{ "Flat" }}
        {% endif %}

Hey there!

Me again. I’ve been running for a week with the sensors, and although I still haven’t sorted a nice UI (just embedded some of the sensors to a Glance card temporary) I have been able to configure some cool automations with Telegram to alert when there is Surf forecast.

I have however some issues with multiscrape failing too much. Sensors are barely available, and if they’re not available when I want the automation to check, I miss it, so I am currently missing a lot of alerts.
To have an overview of availability, here’s an overview of last week (trying to refresh once an hour):

I’ve tried enabling the logs, but nothing useful in neither of them:

2023-05-25 21:24:48.414 ERROR (MainThread) [custom_components.multiscrape.coordinator] Wisuki scraper # Updating failed with exception: [Errno -3] Try again
2023-05-25 21:24:48.418 ERROR (MainThread) [custom_components.multiscrape.sensor] Wisuki scraper # Alibey # Unable to scrape data: Skipped scraping because data couldn't be updated
Consider using debug logging and log_response for further investigation.
2023-05-25 21:24:48.422 ERROR (MainThread) [custom_components.multiscrape.sensor] Wisuki scraper # Viento # Unable to scrape data: Skipped scraping because data couldn't be updated
Consider using debug logging and log_response for further investigation.
2023-05-25 21:24:48.426 ERROR (MainThread) [custom_components.multiscrape.sensor] Wisuki scraper # Racha # Unable to scrape data: Skipped scraping because data couldn't be updated
Consider using debug logging and log_response for further investigation.
2023-05-25 21:24:48.429 ERROR (MainThread) [custom_components.multiscrape.sensor] Wisuki scraper # Cardinalidad # Unable to scrape data: Skipped scraping because data couldn't be updated
Consider using debug logging and log_response for further investigation.
.... {"like this for every sensor"}

Where could this instability come from (pretty sure the website is stable, at least when I query it manually and I also have an iframe for that same website and its always available)? What could it be done?

I’ve thought it’d be interesting a config option within multiscrape to not change the state to unknown if value can’t be updated, but just do nothing (keep the old value).
WDYT @danieldotnl ?

This thread shows that errno -3 is related to a temporary failure in name resolution. I have no idea what is causing that.

I’ve thought it’d be interesting a config option within multiscrape to not change the state to unknown if value can’t be updated, but just do nothing (keep the old value).

That’s already possible, check the on-error functionality.

1 Like

Thanks @danieldotnl,

That might lead me to the issue.
I have a custom dns within pihole and that might be giving some issues.

I imagine something like it might be quering my custom dns and since not finding it, giving the error instead of falling back to the secondary dns. And therefore only giving proper values when the name resolution is cached (i.e: cause I opened HA and the iframe loaded the value).

Ill dig more into it :+1:

Also going to check out the onerror, since id also like such behaviour :muscle:

I was wondering if someone could help me out scraping some data?

I’m trying to extract the rainfall data from a weather station web page:

https://tempestwx.com/map/43908/41.3807/-85.0447/17

the data I’m trying to extract is this data which is in a side window on the right side of the main map window:

image

I traced the selector but it doesn’t return any values.

Here is the sensor config (ignore the misspelling of tempest for now please :laughing:):

- resource: https://tempestwx.com/map/45085/41.3807/-85.0447/17
    scan_interval: 600
    headers:
      User-Agent: Mozilla/5.0
    sensor:
      - unique_id: ms_tempestwx_auburn_rain_today_1
        name: MS Tespest WX Rain Today 1
        select: "#list-view > ul > li:nth-child(15) > span.lv-value-display.raincheck"
        unit_of_measurement: "in"
      - unique_id: ms_tempestwx_auburn_rain_yesterday_1
        name: MS Tespest WX Rain Yesterday 1
        select: "#list-view > ul > li:nth-child(16) > span.lv-value-display.raincheck"
        unit_of_measurement: "in"
    log_response: true

Here is the relevant portion of the main HA log with debugging enabled:

2023-06-16 17:04:48.852 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_1 # New run: start (re)loading data from resource
2023-06-16 17:04:48.852 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_1 # Deleting logging files from previous run
2023-06-16 17:04:50.600 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_1 # Rendered resource template into: https://tempestwx.com/map/45085/41.3807/-85.0447/17
2023-06-16 17:04:50.600 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_1 # Request data from https://tempestwx.com/map/45085/41.3807/-85.0447/17
2023-06-16 17:04:50.600 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_1 # Executing page-request with a get to url: https://tempestwx.com/map/45085/41.3807/-85.0447/17.
2023-06-16 17:04:50.857 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_1 # request_headers written to file: page_request_headers.txt
2023-06-16 17:04:51.186 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_1 # request_body written to file: page_request_body.txt
2023-06-16 17:04:53.829 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_1 # Response status code received: 200
2023-06-16 17:04:53.905 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_1 # response_headers written to file: page_response_headers.txt
2023-06-16 17:04:53.923 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_1 # response_body written to file: page_response_body.txt
2023-06-16 17:04:53.923 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_1 # Loading the content in BeautifulSoup.
2023-06-16 17:04:53.958 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_1 # page_soup written to file: page_soup.txt
2023-06-16 17:04:53.958 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_1 # Data succesfully refreshed. Sensors will now start scraping to update.
2023-06-16 17:05:10.014 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Today 1 # Setting up sensor
2023-06-16 17:05:10.029 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Yesterday 1 # Setting up sensor
2023-06-16 17:05:10.056 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Today 1 # Start scraping to update sensor
2023-06-16 17:05:10.058 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_1 # MS Tespest WX Rain Today 1 # Tag selected: None
2023-06-16 17:05:10.058 ERROR (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Today 1 # Unable to scrape data: Could not find a tag for given selector 
Consider using debug logging and log_response for further investigation.
2023-06-16 17:05:10.062 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Today 1 # On-error, set value to None
2023-06-16 17:05:10.062 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_1 # MS Tespest WX Rain Today 1 # Updated sensor and attributes, now adding to HA
2023-06-16 17:05:10.107 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Yesterday 1 # Start scraping to update sensor
2023-06-16 17:05:10.108 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_1 # MS Tespest WX Rain Yesterday 1 # Tag selected: None
2023-06-16 17:05:10.108 ERROR (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Yesterday 1 # Unable to scrape data: Could not find a tag for given selector 
Consider using debug logging and log_response for further investigation.
2023-06-16 17:05:10.111 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # MS Tespest WX Rain Yesterday 1 # On-error, set value to None
2023-06-16 17:05:10.111 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_1 # MS Tespest WX Rain Yesterday 1 # Updated sensor and attributes, now adding to HA

I obviously also have the log_response data but there’s a lot here already and I don’t want to data dump it all at once so tell me what else I need to post and I will.

Thanks in advance.

Why dont you use the HACS tempest plugin … unless you need an API key or something I guess.

Is there a way to make value_template return an array? I’ve been playing with different ways to do it - but it seems like its always returning a string:

Here is my current config:


multiscrape:
  # Debug with: https://try.jsoup.org/
  - name: IQAir Scraper
    resource: https://www.iqair.com/usa/colorado/colorado-springs
    scan_interval: 600
    log_response: true
    sensor:
      - unique_id: ms_iq_air_0
        name: IQAir Now
        select_list: ".pollutant-level-wrapper b"
        value_template: "{{value.split(',')[4] | int }}"
        select: "forecast"
        attributes:
          - name: Main Pollutant
            select: ".aqi-overview-detail__main-pollution-table td:last-child"
          - name: Array_test
            select: ".pollutant-level-wrapper b"
            value_template: "{{[1,2,3,4]}}"
          - name: Raw Data
            select_list: ".pollutant-level-wrapper b"
            value_template: |
              {%- set values = value.split(',') -%}
              {%- for x in values %}
              - {{ x | int}}
              {%- endfor -%}
          - name: History
            select_list: ".pollutant-level-wrapper b"
            value_template: "{{value.split(',')[0:4] | reverse | list | map('int') | list }}"
          - name: Forecast
            select_list: ".pollutant-level-wrapper b"
            value_template: "{{value.split(',')[5:] | map('int') | list}}"

image

But no matter how I swing it if I try to get a value:

-- DATA
{{ states.sensor.ms_iq_air_0.attributes.array_test[2]}}
{{ states.sensor.ms_iq_air_0.attributes.history}}
{{ states.sensor.ms_iq_air_0.attributes.history[0]}}
{{ states.sensor.ms_iq_air_0.attributes.history | list }}
{{ states.sensor.ms_iq_air_0.attributes.history.split(',')  }}
--

image

If the answer is no I can happily live with it - but I figure there should be a way to get an array of ints…

running the is string on all the stuff is always returning true

Sensor states in HA are always strings. If you want to preserve the data type, put it in an attribute.

Use the Beautiful Soup capture (a .txt file in a multiscrape directory in your condig directory). I recently had a case where it helped me. This is the HTML that is getting parsed. You might be surprised, but what your browser will show isn’t always identical to what will be parsed here. See the integration’s docs for more info.

Thanks for the replies…I’ve been out of town for a few days…

Right, I’m trying to just “borrow” some data from someone else’s weather station so I don’t have a key/token.

I tried that but I can’t seem to find the data field I’m looking for in that capture. I guess I’m not sure what I’m looking for to find something in a sidebar but not in the main page.

Then I think your luck is out, because that capture is literally what the parser sees. I had a recent case like that too. The issue will be a dynamic call that executes after page load, so the content isn’t actually there until later.

You’ll need to check in the browser dev tools which calls are being made during page load and see if you can use one of those to call directly.

ugg…I think this isn’t worth the aggravation.

I’ve got a few options that I’m waiting for it to rain here to test out so hopefully those will make this attempt moot.

thanks for the help tho.

1 Like

I was specifically looking at attributes which also appear to be strings… I can work around it … but just a tad confused as how to make an actual attribute which is an array of “stuff”

What I’ve done is take that text file and drop it into: “https://try.jsoup.org” and from there parse out what I needed…

There are a few ways. You can store an actual array in an attribute (but not the main state), or you could serialise the data as a string yourself and put it as the main state (making it a CSV list, for example, or even as JSON), but you then need to parse it every time you use it.

Thanks that link is useful to know.

but in my case it didn’t help and I think that’s because the info I want is in a sidebar and I’m not sure how to reference that data in the scraper.

I’m just using the | from_json filter in my templates and that seems to do what I need.

1 Like