Today, after working for weeks on end, I’m now seeing this error when attempting to load Grafana through my Cloudflared tunnel address:
If you’re seeing this Grafana has failed to load its application files
This could be caused by your reverse proxy settings.
If you host grafana under subpath make sure your grafana.ini root_url setting includes subpath. If not using a reverse proxy make sure to set serve_from_sub_path to true.
If you have a local dev build make sure you build frontend using: yarn start, yarn start:hot, or yarn build
Sometimes restarting grafana-server can help
Check if you are using a non-supported browser. For more information, refer to the list of supported browsers.
I had not changed anything about my setup since this last worked, and nothing is on auto-update.
When I access this from local (http://homeassistant.local:8123/a0d7b954_grafana/dashboard) it works just fine, but when accessed through my Cloudflared tunnel (https://ha.mydomain.com/a0d7b954_grafana/dashboard) it fails.
Cloudflared logs:
-----------------------------------------------------------
Add-on: Cloudflared
Use a Cloudflare Tunnel to remotely connect to Home Assistant without opening any ports
-----------------------------------------------------------
Add-on version: 4.1.5
You are running the latest version of this add-on.
System: Home Assistant OS 9.5 (amd64 / generic-x86-64)
Home Assistant Core: 2023.4.2
Home Assistant Supervisor: 2023.04.1
-----------------------------------------------------------
Please, share the above information when looking for help
or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
Log level is set to DEBUG
[11:16:00] DEBUG: Cloudflared log level set to "info"
[11:16:01] INFO: Checking add-on config...
[11:16:01] DEBUG: Checking host {"hostname":"syn.mydomain.com","service":"https://192.168.1.10:5001"}...
[11:16:02] INFO: Checking for existing certificate...
[11:16:02] INFO: Existing certificate found
[11:16:02] INFO: Checking for existing tunnel...
[11:16:02] INFO: Existing tunnel with ID 499c5709-d6bd-480b-b2df-12a94ac9c237 found
[11:16:02] INFO: Checking if existing tunnel matches name given in config
[11:16:03] DEBUG: Existing Cloudflare Tunnel name: homeassistant
[11:16:03] INFO: Existing Cloudflare Tunnel name matches config, proceeding with existing tunnel file
[11:16:03] INFO: Creating config file...
[11:16:03] DEBUG: Checking if SSL is used...
[11:16:03] DEBUG: Requested API resource: http://supervisor/core/info
[11:16:03] DEBUG: Request method: GET
[11:16:03] DEBUG: Request data: {}
[11:16:03] DEBUG: API HTTP Response code: 200
[11:16:03] DEBUG: API Response: {"result": "ok", "data": {"version": "2023.4.2", "version_latest": "2023.5.2", "update_available": true, "machine": "generic-x86-64", "ip_address": "172.30.32.1", "arch": "amd64", "image": "ghcr.io/home-assistant/generic-x86-64-homeassistant", "boot": true, "port": 8123, "ssl": false, "watchdog": true, "audio_input": null, "audio_output": null}}
[11:16:03] DEBUG: ha_service_protocol: http
[11:16:05] INFO: Validating config file...
[11:16:05] DEBUG: Validating created config file: {"tunnel":"499c5709-d6bd-480b-b2df-12a94ac9c237","credentials-file":"/data/tunnel.json","ingress":[{"hostname":"ha.mydomain.com","service":"http://homeassistant:8123","originRequest":{"noTLSVerify":true}},{"hostname":"syn.mydomain.com","service":"https://192.168.1.10:5001","originRequest":{"noTLSVerify":true}},{"service":"http_status:404","originRequest":{"noTLSVerify":true}}]}
Validating rules from /tmp/config.json
OK
[11:16:05] DEBUG: Sucessfully created config file: {"tunnel":"499c5709-d6bd-480b-b2df-12a94ac9c237","credentials-file":"/data/tunnel.json","ingress":[{"hostname":"ha.mydomain.com","service":"http://homeassistant:8123","originRequest":{"noTLSVerify":true}},{"hostname":"syn.mydomain.com","service":"https://192.168.1.10:5001","originRequest":{"noTLSVerify":true}},{"service":"http_status:404","originRequest":{"noTLSVerify":true}}]}
[11:16:05] INFO: Creating DNS entry ha.mydomain.com...
2023-05-12T18:16:06Z INF ha.mydomain.com is already configured to route to your tunnel tunnelID=499c5709-d6bd-480b-b2df-12a94ac9c237
[11:16:06] INFO: Creating DNS entry syn.mydomain.com...
2023-05-12T18:16:08Z INF syn.mydomain.com is already configured to route to your tunnel tunnelID=499c5709-d6bd-480b-b2df-12a94ac9c237
[11:16:08] INFO: Finished setting up the Cloudflare Tunnel
[11:16:08] DEBUG: using /tmp/config.json config file
[11:16:08] INFO: Connecting Cloudflare Tunnel...
[11:16:08] DEBUG: cloudflared tunnel --no-autoupdate --metrics=0.0.0.0:36500 --loglevel=info --origincert=/data/cert.pem --config=/tmp/config.json run null
2023-05-12T18:16:08Z INF Starting tunnel tunnelID=499c5709-d6bd-480b-b2df-12a94ac9c237
2023-05-12T18:16:08Z INF Version 2023.5.0
2023-05-12T18:16:08Z INF GOOS: linux, GOVersion: go1.19.6, GoArch: amd64
2023-05-12T18:16:08Z INF Settings: map[config:/tmp/config.json cred-file:/data/tunnel.json credentials-file:/data/tunnel.json loglevel:info metrics:0.0.0.0:36500 no-autoupdate:true origincert:/data/cert.pem]
2023-05-12T18:16:08Z INF Generated Connector ID: cf3f1171-8f6b-41fb-82f7-d614f70b24ee
2023-05-12T18:16:09Z INF Initial protocol quic
2023-05-12T18:16:09Z INF ICMP proxy will use 172.30.33.3 as source for IPv4
2023-05-12T18:16:09Z INF ICMP proxy will use :: as source for IPv6
2023-05-12T18:16:09Z INF Starting metrics server on [::]:36500/metrics
2023-05-12T18:16:09Z INF Registered tunnel connection connIndex=0 connection=102bfb49-3798-4b54-ba9b-f60f86aa65aa event=0 ip=198.41.200.43 location=SJC protocol=quic
2023-05-12T18:16:09Z INF Registered tunnel connection connIndex=1 connection=dd541d97-26a9-48ad-a2d9-d6a601542cff event=0 ip=198.41.192.227 location=LAX protocol=quic
2023-05-12T18:16:11Z INF Registered tunnel connection connIndex=2 connection=641ff57f-9e96-4da0-ac57-ad4341a8c47a event=0 ip=198.41.200.63 location=SJC protocol=quic
2023-05-12T18:16:11Z INF Registered tunnel connection connIndex=3 connection=8be9df99-8a17-46af-a02a-a7a78ee0ef9c event=0 ip=198.41.192.67 location=LAX protocol=quic
Grafana logs:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service base-addon-banner: starting
-----------------------------------------------------------
Add-on: Grafana
The open platform for beautiful analytics and monitoring
-----------------------------------------------------------
Add-on version: 8.2.2
You are running the latest version of this add-on.
System: Home Assistant OS 9.5 (amd64 / generic-x86-64)
Home Assistant Core: 2023.4.2
Home Assistant Supervisor: 2023.04.1
-----------------------------------------------------------
Please, share the above information when looking for help
or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
s6-rc: info: service base-addon-banner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service base-addon-timezone: starting
s6-rc: info: service base-addon-log-level: starting
s6-rc: info: service fix-attrs successfully started
[10:58:51] INFO: Configuring timezone (America/Los_Angeles)...
s6-rc: info: service base-addon-log-level successfully started
s6-rc: info: service base-addon-timezone successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-nginx: starting
s6-rc: info: service init-grafana: starting
s6-rc: info: service memcached: starting
s6-rc: info: service memcached successfully started
[10:58:52] INFO: Starting Memcached...
s6-rc: info: service init-nginx successfully started
s6-rc: info: service init-grafana successfully started
s6-rc: info: service grafana: starting
s6-rc: info: service grafana successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
[10:58:52] INFO: Starting Grafana...
s6-rc: info: service legacy-services successfully started
Grafana server is running with elevated privileges. This is not recommended
logger=settings t=2023-05-12T10:58:53.464358829-07:00 level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T16:56:52-07:00
logger=settings t=2023-05-12T10:58:53.464965663-07:00 level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead."
logger=settings t=2023-05-12T10:58:53.465005301-07:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-05-12T10:58:53.465024859-07:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2023-05-12T10:58:53.465054776-07:00 level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2023-05-12T10:58:53.465069976-07:00 level=info msg="Path Data" path=/data
logger=settings t=2023-05-12T10:58:53.465083115-07:00 level=info msg="Path Logs" path=/var/logs/grafana
logger=settings t=2023-05-12T10:58:53.465096076-07:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2023-05-12T10:58:53.465109459-07:00 level=info msg="Path Provisioning" path=/usr/share/grafana/conf/provisioning
logger=settings t=2023-05-12T10:58:53.465123183-07:00 level=info msg="App mode production"
logger=sqlstore t=2023-05-12T10:58:53.465260192-07:00 level=info msg="Connecting to DB" dbtype=sqlite3
logger=migrator t=2023-05-12T10:58:53.536315194-07:00 level=info msg="Starting DB migrations"
logger=migrator t=2023-05-12T10:58:53.560345649-07:00 level=info msg="migrations completed" performed=0 skipped=481 duration=1.690822ms
logger=plugin.loader t=2023-05-12T10:58:53.81605456-07:00 level=info msg="Plugin registered" pluginID=input
logger=plugin.loader t=2023-05-12T10:59:01.570737942-07:00 level=info msg="Plugin registered" pluginID=grafana-image-renderer
logger=secrets t=2023-05-12T10:59:01.570884066-07:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2023-05-12T10:59:01.575940131-07:00 level=info msg="Query Service initialization"
logger=live.push_http t=2023-05-12T10:59:01.584302226-07:00 level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2023-05-12T10:59:05.182731271-07:00 level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning.alerting t=2023-05-12T10:59:05.184841142-07:00 level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2023-05-12T10:59:05.184888653-07:00 level=info msg="finished to provision alerting"
logger=grafanaStorageLogger t=2023-05-12T10:59:05.185246228-07:00 level=info msg="storage starting"
logger=ngalert.state.manager t=2023-05-12T10:59:05.187771547-07:00 level=info msg="Warming state cache for startup"
logger=http.server t=2023-05-12T10:59:05.193090342-07:00 level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl=/api/hassio_ingress/rGONBoOlu0HJTyZ2tv980if8k6FGbL5efwkzED7a6iU socket=
[10:59:05] INFO: Starting NGINX...
logger=ngalert.state.manager t=2023-05-12T10:59:05.230981201-07:00 level=info msg="State cache has been initialized" states=0 duration=43.205691ms
logger=ticker t=2023-05-12T10:59:05.231372447-07:00 level=info msg=starting first_tick=2023-05-12T10:59:10-07:00
logger=ngalert.multiorg.alertmanager t=2023-05-12T10:59:05.231490066-07:00 level=info msg="starting MultiOrg Alertmanager"
logger=plugin.grafana-image-renderer t=2023-05-12T10:59:07.966262204-07:00 level=warn msg="Plugin process is running with elevated privileges. This is not recommended"
logger=cleanup t=2023-05-12T11:09:05.224071581-07:00 level=info msg="Completed cleanup jobs" duration=37.114854ms
logger=context userId=1 orgId=1 uname=admin t=2023-05-12T11:15:30.726734022-07:00 level=error msg="Request error" error="net/http: abort Handler"
logger=context userId=1 orgId=1 uname=admin t=2023-05-12T11:15:30.777849404-07:00 level=error msg="Request error" error="net/http: abort Handler"
logger=context userId=1 orgId=1 uname=admin t=2023-05-12T11:15:32.056718993-07:00 level=error msg="Request error" error="net/http: abort Handler"
logger=cleanup t=2023-05-12T11:19:05.237741422-07:00 level=info msg="Completed cleanup jobs" duration=51.504792ms
logger=cleanup t=2023-05-12T11:29:05.239192588-07:00 level=info msg="Completed cleanup jobs" duration=52.886513ms
I tried updating cloudflared to no avail and also restarted the grafana add-on, but neither seem to have helped. I don’t see anything obvious (to me) in the logs nor in the developer tools on the page that’s failing to load.
I’m on HAOS 9.5, HA Core 2023.4.2, Grafana 8.2.2, and Cloudflared 4.1.5