Scrape sensor improved - scraping multiple values

Hi all, just wanted to follow-up if anybody has an idea on how to solve this?

My library uses the same kb platform :slight_smile:
So I gave it a try but it seems to be some kind of OAuth implementation, and thatā€™s quite complex and not supported by Multiscrape, unfortunately.

1 Like

Not possible yet, but a nice feature! Please create a feature request on github.

Iā€™m excited to share some new features and improvements in v7.1.2 which I just released. Hereā€™s whatā€™s new:

:sparkles: New Feature: Form Variables

A big shoutout to @jeremicmilan for his incredible dedication to this feature! :clap: Iā€™ve added Form Variables to Multiscrape, allowing you to scrape the (token of a) page returned after logging in on some sites (specifically PHP). This token can then be sent in a header for authentication or other purposes. For all the details, make sure to check out the README! :books:

:cookie: New Feature: Cookies support

You asked, I delivered! The long-awaited support for cookies is finally here! :tada: Now, all cookies returned in HTTP sessions are automatically transferred to the next request. Plus, Iā€™ve added logging so you can easily see which cookies are set. Sweet, right? :cookie:

:robot: Automated Tests!!

Iā€™m taking stability to the next level with the newly set up automated testing infrastructure! The first 2 automated tests have been added to Multiscrape, ensuring even more reliability in the future. Continuous improvements are on the way! :hammer_and_wrench:

As always, a huge thank you to the amazing community for your continued support and feedback. Happy scraping! :spider::computer:

PS: If you enjoy Multiscrape, please consider supporting me and buy me a coffee.

3 Likes

done, thank you!

I took note of the form variables with this release. Is it limited to capturing form response header values only? I am seeking to capture the csrftoken value from the form response page for use in the subsequent resource url.

Thatā€™s exactly what itā€™s meant for!

Now available in v7.2.0!

New Feature: Raw HTML Scraping with Multiscrape :globe_with_meridians:

Iā€™m excited to announce that it is now possible to scrape raw HTML in Multiscrape! This feature has been a recurring request over the years, and Iā€™m happy I could finally implement it. :tada:
It could for example be used to for displaying rich content on a markdown card.

A new configuration option for selectors has been added called extract. It is optional and can have these values:
- Text (default): Extracts plain text, as you are used to. :memo:
- Content: Returns the content of the selected tag. :scroll:
- Tag: Returns both the content and the tag itself. :label:

With this feature, your sensors (or attributes) can now have a state/value like:

<p>This is an <b>example</b> of what can be scraped with the <i>extract</i> feature.</p>

Thank you for your continued support, and happy scraping! :partying_face:

2 Likes

Would love to see OAuth support, as this will probably solve the issue of scraping Water-link for water meter data.