Scrape Sensor - 'failed with multiple Transfer-Encoding headers'

Configuration
1

Hello,
This one has got me stumped and I’ve found nothing online, so hoping the experts here can help.

When I use a particular URL with the scrape sensor, homeassistant reports:
*ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: <myURL> failed with multiple Transfer-Encoding headers*

Other URLs I’ve tried do not cause this error.

This troublesome URL is for a local authority (which is why I’ve omitted it from the post).
I’m just trying to parse info to tell me which recycling bins to put out.

I can download the page with cURL absolutely fine, and the HTML content is perfect for parsing/extracting data. Also, there’s no login, etc to deal with here.

I used cURL to download the response headers, in case this explains why homeassistant is failing to connect to the URL?..

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Tue, 29 Mar 2022 16:30:14 GMT
Content-Length: 36836
X-Via-NSCOPI: 1.0
Strict-Transport-Security: max-age=147680000
Set-Cookie: NSC_ESNS=02a26a78-3417-1243-9678-66d8cc5d1fdb_4183157558_4262058736_00000000000043858592; Path=/; Expires=Tue, 29-Mar-2022 16:30:29 GMT
Cache-Control: no-cache

Any suggestions?

Cheers

2

There is a chance that curl and Home Assistant send different HTTP request headers and then get different response headers.
Unfortunately I couldn’t find a way to log the actual request and response headers in Home Assistant.

Otherwise, you could play around with adding a Transfer-Encoding header (like Transfer-Encoding: gzip) to your scrape sensor configuration, maybe that influences the response?