Home assistance and caddy as a reverse proxy

You are using the proxy option with the websocket preset which is a shorthand for

header_upstream Connection {>Connection}
header_upstream Upgrade {>Upgrade}

The 2nd line in combination with HTTP/2 most certainly leads to this error since HTTP/2 does not support protocol upgrade.

Please pass the flag -http2=false to Caddy and that problem should disappear.

If you’re using korylprince’s Caddy add-on you can configure the add-on option like so:

{
  "flags": [
    "-http2=false"
  ]
}