Scrape - text longer than 255 characters

Configuration
1

Hi there…

I scrape a text from a website to show it on my dashboard. However, the state of a sensor is apparently limited to a max of 255 characters. Is there another solution how it can be done?

This is my current setup with scrape, which works as long as the text is max 255 characters:

#configuration.yaml
scrape:
  - resource: "https://www.wetter.com/"
    scan_interval: 3600
    sensor:
      - name: "Wetter heute"
        select: "div#globalText > p"

In case the text is >255 characters, the following error is shown in the log:

#home-assistant.log
Invalid state with length 331. State max length is 255 characters.

Thanks for your help!

2

Use a value template as described in the docs
Scrape - Home Assistant

3

Thanks for your reply. However, I fail to get it working. …and to be honest, I also do not understand how the value_template setting shall be of help if the text is more than 255 characters? Would be great if you could provide an example.

4

As you have already discovered, the maximum length that can be stored in an entity’s state is 255 characters. An entity’s attributes can store up to 16KB but the Scrape integration doesn’t support the use of attributes.

There’s a custom integration called Multiscrape that does support the use of attributes.

danieldotnl/ha-multiscrape: Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

Full disclosure: I am only aware of its existence but have never used it.

5

If you need the complete extract then look at what Taras wrote, else the value template can cut back the returned value to what you want…or to less than 255 {{ value[0:254] }}

6

Thanks for pointing me to multiscrape. I will try that out and will update later.

7

Yes thanks, then I at least would get some shorter text i/o an unknown state.

8

It took me some time to figure out exactly how Multiscrape works, but I finally got it right. Here is my working setup for a text with more than 255 characters:

# configuration.yaml
multiscrape:
  - resource: [any website url]
    scan_interval: 3600
    sensor: 
       - name: "[any sensor name]"
         select: "[any selector path for the sensor name]"
         value_template: '{{ value }}'
         attributes: 
           - name: [any attribute name]
             select: "[any selector path for the attribute]"
             value_template: '{{ value }}'

But note that the value for the sensor name must always be less than 255 characters, otherwise the sensor will not be created. The attribute can hold text with more than 255 characters.