Hi all, just wanted to follow-up if anybody has an idea on how to solve this?
My library uses the same kb platform
So I gave it a try but it seems to be some kind of OAuth implementation, and thatās quite complex and not supported by Multiscrape, unfortunately.
Not possible yet, but a nice feature! Please create a feature request on github.
Iām excited to share some new features and improvements in v7.1.2 which I just released. Hereās whatās new:
New Feature: Form Variables
A big shoutout to @jeremicmilan for his incredible dedication to this feature! Iāve added Form Variables to Multiscrape, allowing you to scrape the (token of a) page returned after logging in on some sites (specifically PHP). This token can then be sent in a header for authentication or other purposes. For all the details, make sure to check out the README!
New Feature: Cookies support
You asked, I delivered! The long-awaited support for cookies is finally here! Now, all cookies returned in HTTP sessions are automatically transferred to the next request. Plus, Iāve added logging so you can easily see which cookies are set. Sweet, right?
Automated Tests!!
Iām taking stability to the next level with the newly set up automated testing infrastructure! The first 2 automated tests have been added to Multiscrape, ensuring even more reliability in the future. Continuous improvements are on the way!
As always, a huge thank you to the amazing community for your continued support and feedback. Happy scraping!
PS: If you enjoy Multiscrape, please consider supporting me and buy me a coffee.
done, thank you!
I took note of the form variables with this release. Is it limited to capturing form response header values only? I am seeking to capture the csrftoken value from the form response page for use in the subsequent resource url.
Thatās exactly what itās meant for!
New Feature: Raw HTML Scraping with Multiscrape
Iām excited to announce that it is now possible to scrape raw HTML in Multiscrape! This feature has been a recurring request over the years, and Iām happy I could finally implement it.
It could for example be used to for displaying rich content on a markdown card.
A new configuration option for selectors has been added called extract
. It is optional and can have these values:
- Text (default): Extracts plain text, as you are used to.
- Content: Returns the content of the selected tag.
- Tag: Returns both the content and the tag itself.
With this feature, your sensors (or attributes) can now have a state/value like:
<p>This is an <b>example</b> of what can be scraped with the <i>extract</i> feature.</p>
Thank you for your continued support, and happy scraping!
Would love to see OAuth support, as this will probably solve the issue of scraping Water-link for water meter data.