Z2M won't start - 502: bad gateway

A couple days ago, while I was away, Z2M stopped working. I cannot open the UI from the sidebar, but can open it if I go through the Settings menu, for some reason. In the UI, it shows as running.

Zigbee2mqtt
Mosquito Broker
TubesZB CC2652-POE-2022 coordinator (firmware: 2021.9.3 (Apr 14 2022, 12:19:59)

Here is the Z2M log:

debug 2024-01-03 13:37:22: Loaded state from file /config/zigbee2mqtt/state.json
info  2024-01-03 13:37:22: Logging to console and directory: '/config/zigbee2mqtt/log/2024-01-03.13-37-22' filename: log.txt
debug 2024-01-03 13:37:22: Removing old log directory '/config/zigbee2mqtt/log/2024-01-03.08-15-34'
info  2024-01-03 13:37:22: Starting Zigbee2MQTT version 1.35.0 (commit #unknown)
info  2024-01-03 13:37:22: Starting zigbee-herdsman (0.30.0)
debug 2024-01-03 13:37:22: Using zigbee-herdsman with settings: '{"adapter":{"concurrent":null,"delay":null,"disableLED":false},"backupPath":"/config/zigbee2mqtt/coordinator_backup.json","databaseBackupPath":"/config/zigbee2mqtt/database.db.backup","databasePath":"/config/zigbee2mqtt/database.db","network":{"channelList":[11],"extendedPanID":[221,221,221,221,221,221,221,221],"networkKey":"HIDDEN","panID":6754},"serialPort":{"path":"tcp://192.168.1.149:6638"}}'
error 2024-01-03 13:37:47: Failed to start zigbee
error 2024-01-03 13:37:47: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
error 2024-01-03 13:37:47: Exiting...
error 2024-01-03 13:37:47: Error: Read 0xa4c138f069ec759c/11 lightingColorCtrl(["colorCapabilities"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'No network route' (205))
    at ZStackAdapter.sendZclFrameToEndpointInternal (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:415:23)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Queue.executeNext (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32)

Z2M Config

data_path: /config/zigbee2mqtt
socat:
  enabled: false
  master: pty,raw,echo=0,link=/tmp/ttyZ2M,mode=777
  slave: tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5
  options: "-d -d"
  log: false
mqtt:
  server: mqtt://core-mosquitto
  user: mqtt-user
  password: [password]
serial:
  port: tcp://192.168.1.149:6638

Not even sure where to be troubleshooting.

Could be an issue with one of your lights:

Interesting. We have a few Ikea bulbs in the driveway that we were changing the color on for Christmas. Maybe I’ll pull those out and try restarting tonight.

So I pulled every Ikea bulb I have on the system and restarted my coordinator with Z2M and Mosquito Broker stopped, and the coordinator appears to restart properly and shows as Serial Connected: On as it should normally. One I start Z2M and the broker, serial turns off and I get these events in the coordinator log…

[D][binary_sensor:036]: 'Serial Connected': Sending state ON
[D][serial_server:068]: Client 192.168.1.141 disconnected
[D][binary_sensor:036]: 'Serial Connected': Sending state OFF

And they continue to cycle every minute or so. I also found some interesting lines in the Supervisor log…

24-01-03 19:44:06 INFO (SyncWorker_3) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:44:06 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:44:32 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on homeassistant/amd64-addon-mosquitto with version 6.4.0
24-01-03 19:44:36 ERROR (MainThread) [supervisor.services.modules.mqtt] There is already a MQTT service in use from core_mosquitto
24-01-03 19:44:40 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:44:40 INFO (SyncWorker_5) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:44:41 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:45:06 ERROR (MainThread) [supervisor.api.ingress] Ingress error: Cannot connect to host 172.30.33.0:8099 ssl:default [Connect call failed ('172.30.33.0', 8099)]
24-01-03 19:45:08 ERROR (MainThread) [supervisor.api.ingress] Ingress error: Cannot connect to host 172.30.33.0:8099 ssl:default [Connect call failed ('172.30.33.0', 8099)]
24-01-03 19:45:12 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:45:12 INFO (SyncWorker_6) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:45:12 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:45:43 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:45:43 INFO (SyncWorker_1) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:45:44 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:45:53 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
24-01-03 19:46:15 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:46:15 INFO (SyncWorker_7) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:46:16 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:46:47 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:46:47 INFO (SyncWorker_3) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:46:47 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:47:18 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:47:18 INFO (SyncWorker_6) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:47:19 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.check] System checks complete
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
24-01-03 19:47:36 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
24-01-03 19:47:50 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:47:50 INFO (SyncWorker_6) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:47:51 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:48:22 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:48:22 INFO (SyncWorker_2) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:48:23 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:48:43 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/zigbee2mqtt/hassio-zigbee2mqtt repository
24-01-03 19:48:43 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
24-01-03 19:48:43 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/adamoutler/mqtt-explorer.git repository
24-01-03 19:48:43 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/CoolKit-Technologies/ha-addon repository
24-01-03 19:48:43 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
24-01-03 19:48:43 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
24-01-03 19:48:45 INFO (MainThread) [supervisor.store] Loading add-ons from store: 77 all - 0 new - 0 remove
24-01-03 19:48:45 INFO (MainThread) [supervisor.store] Loading add-ons from store: 77 all - 0 new - 0 remove
24-01-03 19:48:54 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:48:54 INFO (SyncWorker_5) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:48:55 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1
24-01-03 19:49:26 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Zigbee2MQTT is failed, restarting...
24-01-03 19:49:26 INFO (SyncWorker_1) [supervisor.docker.manager] Cleaning addon_45df7312_zigbee2mqtt application
24-01-03 19:49:27 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.35.0-1

I’m just not even sure where to begin troubleshooting this issue.

SAME PROBLEM HERE!

I have a Sonoff E, Zigbee2MQTT add-on, Silicon Labs Multiprotocol add-on.
There ware a bunch of updates recently and I installed them all, except for the Silicon Labs update. That has the following revision note…

2.4.3

  • :warning: Zigbee2MQTT does not yet support this version of the Gecko SDK. Do not update if you are using Z2M! :warning:.
  • This is a re-release of 2.4.0 that is compatible only with Home Assistant Core 2024.1.0 and above.

I’m thinking that something got broken when updating to the latest HA version.
Currently I have the following…
Sonoff E Firmware: 4.3.1 (latest?)
Home Assistant Core: 2024.1.3 (latest)
Home Assistant OS: 11.4 (latest)
Mosquitto Broker: 6.4.0 (latest)
Silicon Labs Multiprotocol: 2.3.2 (old version)
Zigbee2MQTT: 1.35.1-1 (latest)

I want to try updating Silicon Labs Multiprotocol, not to the latest, but to version 2.4.1. The problem is that I do not know how to update to anything other than the current version. Is there a manual way to do this?