[Working] HomeAssistant Remote Access via NGINX Proxy / mTLS

Configuration
1

Hello,

I had issues in getting secured access on Apple devices via mTLS to my locally running Home Assistant. With the help of Anthropic Claude, I could finally solve this.

The setup is:
VPS running NPMplus
Having self signed certificates for mTLS
HomeAssistant dials into the VPS via Wireguard

The following post is created with the help of Anthropic Claude:

———

Fix: Home Assistant Companion App fails with OSStatus -9800 behind mTLS — RSA vs. ECDSA client certificates

TL;DR

Companion App WebSocket connection fails with OSStatus -9800 (errSSLProtocol) when the mTLS client certificate uses an RSA key, even though the exact same certificate works fine in Safari. Re-issuing the certificate with an ECDSA P-256 key instead fixes it immediately — no server/reverse-proxy changes needed. On macOS specifically, installing the cert as a .mobileconfig also fails separately with “incorrect password” (unrelated bug, use Keychain Access with the raw .p12 instead).

Setup

  • Home Assistant behind a reverse proxy (nginx-based) that terminates TLS and enforces mutual TLS (ssl_verify_client on) for the public hostname
  • Client certificates issued by a private CA, distributed to devices as .p12 bundles
  • Public domain, e.g. home.example.com, proxied to the internal Home Assistant instance
  • Tested on the official Home Assistant Companion App, both iOS and macOS (recent versions, mTLS client-certificate support still listed as an experimental/“Labs” feature at the time of testing)

Symptom

After selecting the client certificate (and, on iOS, after a successful login/token exchange), the Companion App fails to establish the WebSocket connection with:

Description: The operation couldn't be completed. (OSStatus error -9800.)
Domain: NSOSStatusErrorDomain
Code: -9800

-9800 corresponds to errSSLProtocol — a generic TLS handshake failure. The same URL, same certificate, same password work perfectly fine when opened in Safari. Only the Companion App’s WebSocket connection fails, reproducibly, every time.

What we ruled out first

Before landing on the actual cause, we eliminated several plausible server-side explanations:

  1. PKCS12 metadata (friendlyName/caName attributes baked into the .p12 at export time) — no difference with or without them.
  2. HTTP/3 (QUIC) — disabled entirely on the reverse proxy, no change.
  3. Missing proxy headers (X-Ssl-Client-Verify etc.) — not applicable, since ssl_verify_client on rejects unauthenticated connections before the request ever reaches the backend (confirmed both by testing and by inspecting the generated nginx config directly).
  4. Missing WebSocket upgrade headers — didn’t fit the failure mode, since errSSLProtocol occurs at the TLS handshake layer, before any HTTP-level upgrade headers would even be relevant.
  5. Setting ssl_verify_client optional (i.e. removing the client-certificate requirement entirely) — the exact same -9800 error still occurred, even though the app now completed login/token exchange successfully and had clearly selected the certificate. This proved the failure was unrelated to mTLS enforcement itself and sat entirely on the client side, inside the app’s WebSocket connection handling.

Root cause

The client certificate’s key algorithm was the actual variable. With an RSA 2048 client key (a very common default when generating certificates with OpenSSL), the WebSocket TLS handshake in the Companion App failed reliably with -9800. Re-issuing the exact same certificate — same CA, same subject, same server, same reverse-proxy config — with an ECDSA P-256 key instead resolved the issue immediately. The connection has since been stable with no further -9800 errors.

This lines up with previously known rough edges in Apple’s TLS stack (Secure Transport / Network.framework) around RSA client certificates in certain TLS 1.3 code paths, while ECDSA client certificates are handled reliably. Whether this is specific to how the Companion App wires up its WebSocket session (as opposed to its REST/API session, which appeared unaffected) or a broader platform quirk wasn’t fully isolated — but the fix is unambiguous and 100% reproducible in both directions (RSA → fails, ECDSA → works).

Fix

Re-generate the client certificate using an ECDSA P-256 key instead of RSA:

# Generate an ECDSA P-256 private key instead of RSA
openssl ecparam -name prime256v1 -genkey -noout -out client.key

# Create a certificate signing request
openssl req -new -key client.key -out client.csr -subj "/CN=my-device"

# Sign it with your private CA (same as you would for an RSA cert)
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 1825 -sha256

# Bundle into a .p12 for installation on the device
openssl pkcs12 -export -out client.p12 \
  -inkey client.key -in client.crt -certfile ca.crt \
  -name "My Device" -passout pass:YOUR_EXPORT_PASSWORD

No changes are needed on the reverse-proxy/mTLS side — the CA and its trust configuration stay exactly the same, since the CA itself was never RSA vs. ECDSA specific in this setup. Only the leaf (device) certificate’s key algorithm needed to change.

One separate, unrelated issue worth knowing about

While testing this on macOS specifically, installing the certificate as a .mobileconfig profile (used to get a friendly display name in System Settings, instead of just the certificate’s CN) consistently failed with “incorrect password” — regardless of key algorithm, regardless of password complexity, even with the exact same .p12 that imported successfully via Keychain Access directly. This is a distinct, macOS-specific bug in profile installation (also reported by other users for com.apple.security.pkcs12 payloads, and separately for com.apple.dnsSettings.managed payloads, on recent macOS versions), unrelated to the WebSocket/RSA issue above. Workaround: on macOS, import the raw .p12 via Keychain Access instead of wrapping it in a .mobileconfig. iOS/iPadOS are not affected by this second issue.

Disclosure: Anthropic’s Claude was used to help diagnose this issue (systematically ruling out server-side causes) and to draft this write-up.