Webhook failed with status code 503 updating geo-location over Cloudflared

0anton · January 6, 2023, 3:05pm

I’m using the Tunnel / Zero Trust App Connector to expose my Home Assistant installation to the Internet with the help of the Cloudflared plugin [1].
The Home Assistant installation is accessable over cellular plan from my phone (iOS device), I can see HA without problems, but the iOS app cannot push location updates (zone off/on events) to the HA.

The local logs have messages like:

2023-01-06 14:39:37.478 [Info] [main] [ClientEventStore.swift:8] ClientEventStore > networkRequest: Webhook failed with status code 503 [:]
2023-01-06 14:39:37.482 [Error] [main] [WebhookManager.swift:633] urlSession(_:task:didCompleteWithError:) > failed request to 6BEA5895-63A8-42B1-9FFC-5801A86BB1DB for WebhookResponseLocation: unacceptableStatusCode(503)

Messages are only written at the time of the manual location update or on device wake up, it correlated with the accepted push of location update.
No iPhone sensor updates over cellular connection are possible since I switched from Nabu Casa to Cloudflared method of exposing the HA installation.
Before geofences, location-based updated worked perfectly over Nabu Casa subscription from the same device.

I checked that Bot Fight Mode is disabled in my Cloudflare domain settings.
Unfortunately Cloudflare does not provide any access logs. I don’t see any security events in the corresponding log of Cloudflare.

I tried to reset (debug reset) my configuration to HA on in HA app on iPhone, but it didn’t change anything.

I was trying to check the access logs on Cloudflared Addon side and on the HA side, but I didn’t found any.

I’m kinda loss in troubleshooting this:

nor ingress logs to understand why and which API path WebHook requrest fails (to exclude the Cloudflare blocks the request),
nor HA / Addon side have logs to confirm that HA backend get the requests, but respond with 503 for some reason…

How can I troubleshoot “Webhook failed with status code 503” error?

Refs

GitHub - brenner-tobias/addon-cloudflared: Connect remotely to your Home Assistant instance without opening any ports using Cloudflared.

0anton · January 6, 2023, 3:44pm

I’ve noticed that Nabu Case creates webhooks. Is it possible that HA Companion App is hardcoded to use those webhooks for the location sensor update instead of the common API over external URL?

Mobile App: xxx (mobile_app)

0anton · January 6, 2023, 4:43pm

Interestingly, when I log in my HA instance to Nabu Casa Cloud, the webhook 503 error disappears, even I still use external URL to my Cloudflared app. Are there are hard coded URLs to Nabu Case to push location? Or Cloudflare for some reason rejects the “webhook” traffic? What exactly Compation app sends when it tries to push updates other those webhooks?

0anton · January 8, 2023, 8:56pm

Resolved by adding explicit accept firewall rules in Cloudflare for traffic directed to HA sub-domain.
Suggested PR Extend troubleshooting section by 0anton · Pull Request #293 · brenner-tobias/addon-cloudflared · GitHub

See Webhook failed with the status code 503 while updating location of the Troubleshooting section

github.com

brenner-tobias/addon-cloudflared/blob/c2078d9e82bf11150db1617a04143f48acdbf418/cloudflared/DOCS.md#webhook-failed-with-the-status-code-503-while-updating-location

# Home Assistant Add-on: Cloudflared

Cloudflared connects your Home Assistant Instance via a secure tunnel to a domain
or subdomain at Cloudflare. This allows you to expose your Home Assistant
instance and other services to the Internet without opening ports on your router.
Additionally, you can utilize Cloudflare Zero Trust to further secure your
connection.

## Disclaimer

Please make sure you comply with the
[Cloudflare Self-Serve Subscription Agreement][cloudflare-sssa] when using this
add-on. For example [section 2.8][cloudflare-sssa-28] could be breached when
streaming videos (e.g. Plex) or other non-HTML content.

## Initial setup

### Prerequisites

1. A domain name (e.g. example.com) using Cloudflare for DNS. If you don't have

This file has been truncated. show original

jcbshw · March 1, 2023, 7:38pm

I followed all the instructions on the wiki, but I’m still having the issue. Any ideas?

EDIT: I had to reset the companion app and remove all access tokens from my home assistant profile. All is working now.

herrzettpunkt · March 20, 2023, 12:00am

@jcbshw if only I came across your post earlier, this would have saved me hours of troubleshooting… thank you!!!

@0anton could it be possible to amend the following wiki page to delete the access token from the profile in HA if the foregoing steps were unsuccessful?

0anton · April 4, 2023, 10:13am

@herrzettpunkt Thank you very much for your suggestion! I’ve added a section about removing access token into troubleshooting document. Hope I’ve correctly reflected your situation. If not, please, let me know - I’ll improve.

junado · November 30, 2023, 1:22am

If anyone stumbles upon this thread trying to figure out this issue, I’ve had something similar happen when migrating from Home Assistant Cloud to my own domain (dynamic DNS through a domain I own to an NGINX reverse proxy and then to my local HA instance). My Android phone would update its location through the remote URL properly after the switch but not my iPhone. Location tracking was fine on local Wifi. Clearing the access token from the iPhone, then login out and back in appears to have fixed the issue.

gregshome · April 15, 2024, 3:45am

The thing that worked for me here was resetting the mobile apps completely after setting up this plugin. Left the rules in cloudflare, but they didn’t seem to fix it until I removed auth tokens for the mobile devices. For my partner I even had to delete their account (non admin user) because I couldn’t see how to reset their auth tokens. Also had to delete the devices. Then it all worked.