Can anyone help with an error I’m getting using Multiscrape? I’m trying to scrape some data from marinetraffic.com which I have access rights to as a contributor. I can see the data I want after accepting cookies and logging in. I’ve tried to replicate the steps to get there in multiscrape, but I continue to get errors: in form_page_response_body.txt I see <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
This is my multiscrape yaml:
- name: marinetraffictest
resource: https://www.marinetraffic.com/en/ais/details/ports/107?name=PORTSMOUTH&country=United-Kingdom#expectedArrivals
scan_interval: 360000
log_response: true
button:
- unique_id: marinetraffictest
name: marinetraffictest
form_submit:
submit_once: True
resource: https://www.marinetraffic.com/en/users/login
select: "#qc-cmp2-ui > div.qc-cmp2-footer.qc-cmp2-footer-overlay.qc-cmp2-footer-scrolled > div > button.css-1yp8yiu"
input:
email: "*@***.***"
password: "********"
sensor:
- unique_id: marinetraffictest_arr
name: marinetraffictest_arr
select: "h1.MuiTypography-root"
attributes:
- name: arrivals
select: ".MuiDataGrid-virtualScrollerContent"
The select in the form_submit section is the css selector for the agree to cookies button that appears on first use.
I thought this was just a problem with accepting cookies, but form_page_response_cookies.txt does appear to have a cookie in it so I don’t really know how to proceed.
I can’t get to the point of debugging the selector to scrape the table that I need, so I suspect that will be wrong too.
Any help gratefully received.