403 Forbidden for several addons

kcslb92 · August 20, 2021, 8:42am

Hi Guys,

Apologies if I’ve posted this in the wrong category, I wasn’t sure of where it would fit best.

I am having issues with a few addons that I’ve installed on my raspi 4 through supervisor.

Of the 4 add-ons I have installed thus far (ESPHome, InfluxDB, Grafana, Node-Red), ESPHome and NodeRed are the only two that present a web page when you open their web ui from within HA.

The other 2 (InfluxDB, Grafana) both present a " 403 Forbidden" when attempting to browse to them.

Upon inspecting logs, I see these kinds of messages:

2021/08/20 18:38:22 [error] 479#479: *1 access forbidden by rule, client: 172.30.32.1, server: a0d7b954-grafana, request: "GET / HTTP/1.1", host: "192.168.8.10:8123", referrer: "http://192.168.8.10:8123/a0d7b954_grafana"
2021/08/20 18:38:32 [error] 479#479: *2 access forbidden by rule, client: 172.30.32.1, server: a0d7b954-grafana, request: "GET /api/health HTTP/1.1", host: "172.30.33.1:1337"

2021/08/20 18:16:44 [error] 797#797: *1 access forbidden by rule, client: 172.30.32.1, server: a0d7b954-influxdb, request: "GET / HTTP/1.1", host: "192.168.8.10:8123", referrer: "http://192.168.8.10:8123/hassio"

21-08-20 18:30:26 WARNING (MainThread) [supervisor.misc.tasks] Watchdog missing application response from a0d7b954_grafana
21-08-20 18:32:26 WARNING (MainThread) [supervisor.misc.tasks] Watchdog found a problem with a0d7b954_grafana application!

This leads me to believe that there is a configuration somewhere I need to modify to prevent it from restricting access via ingress to specific IP ranges?

I’ve been pulling my hair out over this for a few days.

Any help would be greatly appreciated!

Cheers,
KC.

WallyR · August 21, 2021, 8:58am

The two different IP addresses (192.168.x.x and 172.30.y.y) makes me think you are running some kind of virtual enviroment, so maybe you ned to allow the access to the network ports in the virtual enviroments.

kcslb92 · August 21, 2021, 11:41am

Hi Wally,

That’s not actually the case. The 172 addressing is used by docker for connectivity between each of the containers that make up the add ons, hassio, supervisor etc.

192.168.8.0/24 is the network that the clients will be sourcing from to access the home assistant installation which is hosted on a raspberry Pi with address 192.168.8.10.

I believe the issue is to do with the way in which ingress is restricting access to the addon containers but I’m not sure hence the post here.

Cheers,
Kane.

francisp · August 21, 2021, 1:44pm

Or you running HA OS or Supervised ? And if Supervised, on what OS ?

kcslb92 · August 22, 2021, 12:38am

Hi Francis.

I am running Raspbery Pi OS with the hassio supervisor.

Please see below output:

pi@raspberrypi:~ $ cat /etc/*release*
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@raspberrypi:~ $ docker ps
CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS              PORTS                    NAMES
71cc261f107c        ghcr.io/hassio-addons/grafana/armv7:7.1.0                    "/init"                  13 seconds ago      Up 11 seconds                                addon_a0d7b954_grafana
ad28ff20e279        15ef4d2f/armv7-addon-esphome-dev:dev                         "/init"                  18 hours ago        Up 18 hours                                  addon_15ef4d2f_esphome-dev
6491d49abed7        ghcr.io/home-assistant/raspberrypi4-homeassistant:2021.8.8   "/init"                  41 hours ago        Up 41 hours                                  homeassistant
89c8941b0935        esphome/esphome-hassio-armv7:2021.8.0                        "/init"                  3 days ago          Up 3 days                                    addon_a0d7b954_esphome
ec00db4a99cf        ghcr.io/hassio-addons/node-red/armv7:10.0.0                  "/init"                  9 days ago          Up 9 days                                    addon_a0d7b954_nodered
da77b2bfdc6a        homeassistant/armv7-addon-configurator:5.3.3                 "/init"                  9 days ago          Up 9 days                                    addon_core_configurator
a0f38dc8936e        ghcr.io/hassio-addons/influxdb/armv7:4.2.1                   "/init"                  9 days ago          Up 9 days           0.0.0.0:8086->8086/tcp   addon_a0d7b954_influxdb
d841f3b728d3        ghcr.io/home-assistant/armv7-hassio-multicast:2021.04.0      "/init"                  9 days ago          Up 9 days                                    hassio_multicast
99d7619f0c81        ghcr.io/home-assistant/armv7-hassio-cli:2021.05.1            "/init /bin/bash -c …"   9 days ago          Up 9 days                                    hassio_cli
78d714713d61        ghcr.io/home-assistant/armv7-hassio-audio:2021.04.0          "/init"                  9 days ago          Up 9 days                                    hassio_audio
4d6d240e794e        ghcr.io/home-assistant/armv7-hassio-dns:2021.06.0            "/init"                  9 days ago          Up 9 days                                    hassio_dns
d3e54016f85b        ghcr.io/home-assistant/armv7-hassio-observer:2021.06.0       "/init"                  3 weeks ago         Up 9 days           0.0.0.0:4357->80/tcp     hassio_observer
ff4ad299aed3        homeassistant/armv7-hassio-supervisor                        "/init"                  3 weeks ago         Up 9 days                                    hassio_supervisor

What’s interesting is that Node Red and ESPHome worked without issue, it’s just these other ones that are having problems.

Here are some more example logs of where I’m seeing the connection rejections:

2021/08/22 10:30:59 [error] 487#487: *1 access forbidden by rule, client: 172.30.32.1, server: a0d7b954-grafana, request: "GET /api/health HTTP/1.1", host: "172.30.33.1:1337"

Interestingly in the case of Grafana, I see that it is continually restarting after it is online for a few minutes.

Here’s a dump of the logs I see up until the crash/restart:

pi@raspberrypi:~ $ docker logs addon_a0d7b954_grafana
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-banner.sh: executing...
-----------------------------------------------------------
 Add-on: Grafana
 The open platform for beautiful analytics and monitoring
-----------------------------------------------------------
 Add-on version: 7.1.0
 There is an update available for this add-on!
 Latest add-on version: 7.2.0
 Please consider upgrading as soon as possible.
 System: Raspbian GNU/Linux 10 (buster)  (armv7 / raspberrypi4)
 Home Assistant Core: 2021.8.8
 Home Assistant Supervisor: 2021.06.8
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
[cont-init.d] 00-banner.sh: exited 0.
[cont-init.d] 01-log-level.sh: executing...
Log level is set to INFO
[cont-init.d] 01-log-level.sh: exited 0.
[cont-init.d] grafana.sh: executing...
[cont-init.d] grafana.sh: exited 0.
[cont-init.d] nginx.sh: executing...
[cont-init.d] nginx.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[10:33:03] INFO: Starting Memcached...
[10:33:03] INFO: Starting Grafana...
[10:33:03] INFO: Setting GF_DEFAULT_INSTANCE_NAME to Hassio
[10:33:03] INFO: Setting GF_AUTH_ANONYMOUS_ENABLED to true
t=2021-08-22T10:33:03+1000 lvl=info msg="Starting Grafana" logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T22:53:33+1000
t=2021-08-22T10:33:03+1000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-08-22T10:33:03+1000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2021-08-22T10:33:03+1000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_DEFAULT_INSTANCE_NAME=Hassio"
t=2021-08-22T10:33:03+1000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_AUTH_ANONYMOUS_ENABLED=true"
t=2021-08-22T10:33:03+1000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2021-08-22T10:33:03+1000 lvl=info msg="Path Data" logger=settings path=/data
t=2021-08-22T10:33:03+1000 lvl=info msg="Path Logs" logger=settings path=/var/logs/grafana
t=2021-08-22T10:33:03+1000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2021-08-22T10:33:03+1000 lvl=info msg="Path Provisioning" logger=settings path=/usr/share/grafana/conf/provisioning
t=2021-08-22T10:33:03+1000 lvl=info msg="App mode production" logger=settings
t=2021-08-22T10:33:03+1000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2021-08-22T10:33:03+1000 lvl=info msg="Starting DB migrations" logger=migrator
t=2021-08-22T10:33:03+1000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=330 duration=2.925931ms
t=2021-08-22T10:33:03+1000 lvl=info msg="Starting plugin search" logger=plugins
t=2021-08-22T10:33:04+1000 lvl=info msg="Registering plugin" logger=plugins id=input
t=2021-08-22T10:33:04+1000 lvl=info msg="External plugins directory created" logger=plugins directory=/var/lib/grafana/plugins
t=2021-08-22T10:33:04+1000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http
t=2021-08-22T10:33:04+1000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl=/api/hassio_ingress/OVYonWkzrS1n_1R7z3hKBKRgDHXRlikxDHSffn3ga1c socket=
[10:33:04] INFO: Starting NGinx...
2021/08/22 10:34:48 [error] 487#487: *1 access forbidden by rule, client: 172.30.32.1, server: a0d7b954-grafana, request: "GET / HTTP/1.1", host: "192.168.8.10:8123", referrer: "http://192.168.8.10:8123/a0d7b954_grafana"
2021/08/22 10:35:01 [error] 487#487: *2 access forbidden by rule, client: 172.30.32.1, server: a0d7b954-grafana, request: "GET /api/health HTTP/1.1", host: "172.30.33.1:1337"
2021/08/22 10:37:01 [error] 487#487: *3 access forbidden by rule, client: 172.30.32.1, server: a0d7b954-grafana, request: "GET /api/health HTTP/1.1", host: "172.30.33.1:1337"
t=2021-08-22T10:37:02+1000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated"
Signal handled: Terminated.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] 99-message.sh: executing...
[cont-finish.d] 99-message.sh: exited 0.
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.

That does not explain why the influxdb web interface is also not loading.

Any help would be greatly appreciated.

kcslb92 · August 22, 2021, 12:44am

For completeness, here is the config I have in place for the Grafana instance:

ChadGnad · September 25, 2021, 6:04am

Did you get this sorted? I had an issue, and it was Firefox, toggle the enhanced tracking on the shield icon next to the address…

pirvul · February 23, 2022, 9:51pm

Same issue here, seems the issue is related to a different ip setup. for me 172.30.32.2 is the ip of hassio_supervisor not of homeassistant.

Maybe a solution is to support configurable ingress ips here: https://github.com/hassio-addons/addon-grafana/blob/73ae34b006af22f8934611429acda5b0e3ddf601/grafana/rootfs/etc/nginx/servers/ingress.conf#L8

My workaround for now is:

docker exec -it addon_a0d7b954_grafana bash -c 'sed -i -e "s/172.30.32.2/172.30.32.1/" /etc/nginx/servers/ingress.conf;for p in `s6-ps | grep "nginx:" | grep -v grep | tr -s " " | cut -d" " -f2`; do echo $p; kill $p; done'