Support 255+ characters for scrape sensor state

Feature Requests
1

The scrape sensor platform (or potentially command_line platform) is used to scrape web data. This can be normal numbers, but also paragraphs. Assume you want to to get a news summary or a weather report summary in text form - e.g. here https://opendata.dwd.de/weather/text_forecasts/html/VHDL50_DWPG_LATEST_html
or
https://opendata.dwd.de/weather/text_forecasts/html/VHDL51_DWPG_LATEST_html

to display the text or read it through TTS (this is my case), you would need more than 255 character. The current limitation breaks the use case.

So request:
for scrape sensor at least (as its purpose is web scraping), please change the limit from 255characters to double size (510 characters). This is sufficient at least for weather texts without extra warnings. If possible, it would be great, if the sensor can be simply extended so that the user can set the max limit where a potentially scraped text gets cut.

Thanks

P.s: In my case, HA ready weather news after I am getting up and enter the kitchen to prepare breakfast…

2

Can’t you use the attribute instead?
That is the normal work around.

3

@Hellis81 - this sounds suitable to me. But it seems, I need to adapt the scrape sensor component… Unfortunately I dont have too much knowledge about this. Or is there another way towards achieving this?
Also I dont know if changes in the sensor can be detected without state changes… I can work around the latter by simply scraping another smaller tag…

4

If the target is to scrape json then you can use this method:
Sorry for the image but I’m at work and can only use the phone.

Screenshot_2020-11-04-12-36-31
Screenshot_2020-11-04-12-36-311920×1080 171 KB

In my case the return from this scrape is a json like {"dagar":{something large}}

5

Sorry for not providing the example prior. I would like to scrape HTML, like the following:
https://opendata.dwd.de/weather/text_forecasts/html/VHDL50_DWPG_LATEST_html
I need the text between the “strong” tags (works with scrape sensor as text is not large) and the one between the “pre”-tags.
The same goes for this page:
https://opendata.dwd.de/weather/text_forecasts/html/VHDL51_DWPG_LATEST_html

6

I did a quick search and it seems dwd has an API.
Don’t know if it works for you but there seems to be something.
I suggest you look at that first

7

Thanks a lot for your efforts, if this is the only way, will try… afaik the API provides all data, but not the text form and I dont want to create grammatically correct forecasts with own code.

Anyway again thanks a lot for your effort!!! I will share my insights.

Best regards

8

I saw there was something on GitHub also perhaps that is what you need?
I agree with the request but I assume there is a reason why it’s limited to 255 characters.

If you have website of your own then you can use that to scrape and parse the data, the pass it as json to HA