Finding the source of an aiohttp.server 400 message

jl_678 · January 26, 2022, 2:15am

Hi,

I recently changed from https to http in preparation to move to a proxy. Everything is working well except I see a ton of these in the log:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 334, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Invalid method encountered'"
2022-01-25 20:57:58 ERROR (MainThread) [aiohttp.server] Error handling request

I know that this relates to something hitting the old https based URL which is no longer available. However, I am struggling to figure out how to find the rogue process/client that is doing this. This brings a few questions:

Is there any way to up debugging to get any information about the offending client or process?
Could this be an existing integration or add-on causing this?
Is there anywhere else I should look for offending configuration parameters?

Near term I have solved this problem by reducing aiohttp log levels, but that seems like a band aid.

TIA!

Pet3k · February 22, 2022, 10:19pm

Hello,
I have same issue. Is there any progress please??
My log:

2022-02-22 23:12:26 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 334, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Invalid method encountered'"

jl_678 · February 23, 2022, 4:57pm

I never heard a response on this post. However, I ended up solving the problem by using logs. The problem related to a script that was running on one host that kept hitting the old https address. Once I found it, it was simple to make the change, and the problems went away.

Unfortunately, the analysis was manual on my part.

Madelinot · March 6, 2022, 9:09pm

Similar problem here, but I know it’s related to NGINX. I was using DuckDNS and it was all working. I tried both the NGINX HA Assistant Proxy and the NGINX Proxy Manager add-ons and they don’t work, giving me this error in the logs. I tried re-installing the add-ons too. The add-ons themselves seems to work, but they don’t serve HTTPS as expected. Error paste:

2022-03-03 07:39:38 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 334, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Invalid method encountered'"
2022-03-03 07:39:38 ERROR (MainThread) [aiohttp.server] Error handling request

If I re-enable this in my configuration.yaml file, the errors disappear from the log and I get the SSL cert without issues:

http:
  ssl_certificate: /ssl/fullchain.pem
  ssl_key: /ssl/privkey.pem

I did add this as well to the http section:

  use_x_forwarded_for: true
  trusted_proxies:
    - 172.16.0.0/12
    - 127.0.0.0/8

Changed the trusted proxies combination, tried 192.168.1.0/24 and I even tried 0.0.0.0/0; same results. The logs gets exactly 14 errors as above every time I try to access the site using HTTPS. The certificate does not get transferred either.

I’m really at lost here. Nothing I tried worked. Could this be related: aio-libs/aiohttp#6566 ?

I think I will just bite the bullet and try a complete fresh install… But I may end up with the same situation.

jl_678 · March 7, 2022, 3:03am

This is a different issue, but I recently setup a reverse proxy too. I configured HA to run http locally and Apache to serve https via the reverse proxy.

The other setting that I noticed in your post that could be different is “trusted_proxy”. For me, Apache is running a different system and so I configured trusted_proxies just to contain the Apache IP address. I did not even use a CIDR. e.g. this:

trusted_proxies:
  - 10.0.0.56

Madelinot · March 8, 2022, 2:44pm

I did:

trusted_proxies:
  - 0.0.0.0/0

… and that should allow everything in (like it was before that change a few months ago). It still did not work.

RASBR · April 12, 2022, 2:19pm

Hello,
I got a similar one I guess:

Logger: aiohttp.server
Source: /usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py:405
First occurred: 17:14:51 (4 occurrences)
Last logged: 17:14:54

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 334, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Invalid method encountered'"

Any advice is appreciated.
RS

jjmuriel · May 17, 2022, 4:31pm

Hi,

I had the same issue (NGINX and DuckDNS over HA Supervised) and I resolved the problem with the next steps .
I tested this, but it doesn’t work:

trusted_proxies:
  - 0.0.0.0/0

My info:

In Debian 11, I ran:

and finally I updated my configuration file:

http:
  # Commented because I'm using a reverse proxy NGINX SSL
  # ssl_certificate: /ssl/fullchain.pem
  # ssl_key: /ssl/privkey.pem
  use_x_forwarded_for: true
  trusted_proxies:
    - 172.30.32.0/23
  ip_ban_enabled: true
  login_attempts_threshold: 3

Capusjon · July 7, 2022, 9:07pm

It may not help anyone, but after a lot of trial and error for me the source of the aiohttpserver 400 message was NodeRed. After defining the new HASS adress AND rebooting the nodered container all was fine again.

k014 · July 13, 2022, 12:41pm

that was it for me thanks

gonzacarv · December 30, 2022, 8:48pm

Hi, i’m experiencing this exact behavior.

Did you managed to solve it?

I am migrating HA OS to docker and NGINX Proxy Manager… Cant solve this.

Happy new year!!

jsbrich · August 3, 2023, 6:24pm

Thank you! I’ve been chasing this for several days. The difference was rebooting the nodered instance. Just changing the HA node in nodered didn’t do it

pixeldoc2000 · September 6, 2023, 9:15pm

@Capusjon THX!

Had the same issue:

2023-09-06 23:06:16.658 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 332, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aiohttp/_http_parser.pyx", line 557, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message:
  Bad status line "Invalid method encountered:\n\n  b'\\x16\\x03\\x01\\x01T\\x01'\n    ^"

Setup caddy Reverse Proxy for Home Assistant and change HA from HTTPS back to HTTP. Forget to change HA Base URL in Node Red from HTTPS to HTTP. After changing deploying changes in Node Red and Conatainer restart, the issue is gone.

So the issue was Node Red trying to Connect to HA via HTTPS, while HA was (re-) configured for HTTP.

If HA is configured for HTTP, make sure everything is connecting to HA via HTTP (like Revers Proxy, Node Red, Scripts, Services, etc.)

PuckStar · January 19, 2024, 8:23pm

In case anyone still can’t find the source I can say that for me it was the Hass agent that was internally still using the old https url.

bartkummel · January 23, 2024, 8:57pm

I have a similar issue:

aiohttp.http_exceptions.BadStatusLine: 400, message:
  Invalid method encountered:
    b'\x16\x03\x01\x02'

This appears in my log roughly twice a minute. Reading this thread, am I right that this is about an incoming HTTP request? I have no idea what that could be… Is there any way to make that aiohttp thing be more verbose in the logging, and e.g. log some headers or maybe the originating IP address? That would be very helpful for identifying the source of the requests.

KenCote · February 4, 2024, 10:26pm

I just started getting this today. Really weird.

zweckj · February 15, 2024, 8:17am

For me it were some Shelly devices that were configured to wss:// instead of ws://

Stoffl · March 31, 2024, 12:43pm

Dear guys,

I face similar issues since a while and I would like to get a hint how to debug. I hope you can help:

The main message always is this:

Traceback (most recent call last):
File “/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py”, line 350, in data_received
messages, upgraded, tail = self._request_parser.feed_data(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “aiohttp/_http_parser.pyx”, line 557, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message:
Invalid method encountered:

Then I have different details, which sometimess look like an IP-Address, like here

Invalid method encountered:
b’MGLNDD_91.3.85.199_8123\n’

or with a cryptic info, like here:

Invalid method encountered:
b’\x16\x03\x01’

I hope I can get support from all of you experts how to debug that.

THANKS!

Edit: What I forgot: I’m using HA on a Raspi4-64 and all the versions for core, OS and Supervisor are up to date.

Stoffl · April 15, 2024, 3:44pm

Meanwhile I could find out that when there is a kind of valid IP-address in the error-message the post comes from IOS-mobile.

Maybe this helps to get a common solution for us.