Multiscrape - odd change of behaviour

Configuration
1

So I’ve been using multiscrape for a while to obtain waste collection dates and has been all working fine, but without any conscious change the behaviour of the scrape has changed. For ref I am on multiscrape 6.7.2 HA Core 2023.8.3.

The scrape has worked for months, but the issue now is that the scrape works for a while then goes unavailable. The weird bit is the duration the scrape works is equal to the scan_interval and the period unavailable is then 3 times the scan_interval, after which the cycle repeats. This patten holds for different scan intervals.

Example code is:

multiscrape:
  - resource: https://<url>
    headers:
      User-Agent: Mozilla/5.0  
    scan_interval: 7200
    timeout: 20
    sensor:          
      - unique_id: food_waste
        name: Food Waste
        select:  "body > div.wrapper > div > div.container > div > div.waste__loading_wrapper > div > div > div.govuk-grid-column-two-thirds > div:nth-child(2) > div.govuk-grid-column-three-quarters > dl > div:nth-child(2) > dd"
        value_template: '{{ value.split(", ")[1].split(" (")[0][:8] }}'

Have I missed a change anywhere? Any thoughts?

Ta
S

2

The provider have detected your scrape and did not like the interval settings, so they put a throttle on it.

3

I was thinking that, but is that how a throttle would work, i.e the would just block 75% of my requests? If I set the scan interval for 1 sec it will correctly scrape every 4 seconds, whereas if I set it for 8 hours it would scrape once a day, etc …

In the case in point the provider is a local council, so I am assuming (perhaps incorrectly) they would not be overly sophisticated. The original scrape was running once every 2 hours, which I assumed would not be a huge issue.

If that is the case I will rework to scrape less frequently and where unavailable default to the last known response.

4

Normally a throttle would prevent pulls for a time interval after the first successful pull, bit who knows how an inexperienced administrator set his throttle up.