Node Red losing websocket connection

PS: Actually, it might be the other way around: hassio_supervisor is crashing, disconnecting the sockets, causing NR to try and reconnect. The multiple attempts happen as hassio_supervisor restarts.

There’s a point when supervisor is restarting that will cause NR to get back auth_invalid.

When HA restarts the proxy stays online and this loop occurs.

https://github.com/zachowj/node-red-contrib-home-assistant-websocket/issues/76#issuecomment-453877333

Except, it’s the other way around, the proxy is crashing while HA stays online:

pz@hermes:~$ sudo docker ps | grep ass
bbfbcaa86c43        homeassistant/amd64-hassio-dns:1                    "coredns -conf /conf…"   1 second ago        Up Less than a second                                                    hassio_dns
b36eb4b06a34        homeassistant/amd64-hassio-supervisor               "/bin/entry.sh pytho…"   3 seconds ago       Up 2 seconds                                                             hassio_supervisor
ee9d83c273c3        hassioaddons/node-red-amd64:5.0.7                   "/init"                  About an hour ago   Up About an hour                                                         addon_a0d7b954_nodered
c1460fe27f0c        sabeechen/hassio-google-drive-backup-amd64:0.98.3   "python3 -m backup"      5 hours ago         Up 5 hours              0.0.0.0:1627->1627/tcp, 8099/tcp                 addon_cebe7a76_hassio_google_drive_backup
1e7fe4e5f7b2        homeassistant/amd64-addon-samba:9.0                 "/run.sh"                5 hours ago         Up 5 hours                                                               addon_core_samba
dab2cb7832b7        hassioaddons/aircast-amd64:2.2.1                    "/init"                  5 hours ago         Up 5 hours                                                               addon_a0d7b954_aircast
e1c88b2e7b5c        homeassistant/qemux86-64-homeassistant:0.103.5      "/bin/entry.sh pytho…"   2 days ago          Up 5 hours                                                               homeassistant

My HA has been online for 5+ hours. hassio_supervisor and hassio_dns continue to “crash” and restart for some reason. I don’t know why. It happens right around the 5 minute mark, too, same time NR loses its socket.

I finally found the root cause to my issue: it was my watchtower container.

I took at look at it’s logs and boom!, found it:

time="2020-01-02T16:38:41Z" level=info msg="Creating /hassio_supervisor"
Failed to send notification email:  dial tcp 172.217.212.108:25: connect: connection timed out
time="2020-01-02T16:43:21Z" level=info msg="Found new homeassistant/amd64-hassio-supervisor:latest image (sha256:032972a1a170fba081dc67f359866683e23101010f50c73bc657d72d15b20b88)"
time="2020-01-02T16:43:27Z" level=info msg="Stopping /hassio_supervisor (9310315752ceaec2ae84db7c53c77fefd7d6b361d442e887ffffad3918936cb0) with SIGTERM"
time="2020-01-02T16:43:28Z" level=info msg="Creating /hassio_supervisor"
Failed to send notification email:  dial tcp 172.217.212.108:25: connect: connection timed out
time="2020-01-02T16:48:29Z" level=info msg="Found new homeassistant/amd64-hassio-supervisor:latest image (sha256:032972a1a170fba081dc67f359866683e23101010f50c73bc657d72d15b20b88)"
time="2020-01-02T16:48:35Z" level=info msg="Stopping /hassio_supervisor (ca7ea8346e051c6b34c2e9489db2f1f88fc00559fc221a3a4e522706adff9369) with SIGTERM"
time="2020-01-02T16:48:51Z" level=info msg="Creating /hassio_supervisor"

SMTP isn’t working, thus why I never got informed of this (time to fix that). But, looks like all this time it was an issue with Watchtower not properly pulling down the new image for the new rebuild.

Talk about a chain of events: Node Red goes offline, caused by auth_invalid token, caused by hassio_supervisor restarting every 5 minutes, caused by Watchtower failing to pull down the latest hassio_supervisor docker image, resulting an constant destroy and rebuild every 5 minutes.

I’m a bit late to this thread, but have also just run into this issue. I’m running HA, Mosquito and Node Red in Docker all on a named bridge network.
HA Version core-2021.3.4
19 Apr 15:16:31 - [info] Node-RED version: v1.3.2

I set a long life access token for nodered and then it works great for a few minutes (i.e. not just a single call, all the states are updated and it can make service api calls back into HA.

Then I get WebSocket Closed to my server address.

I’m sure it’s the auth issue that is discussed above, since to ‘fix’ it, I need to restart Node red container, when it again works for a few minutes.

Any suggestions for a fix ?

See my last reply. The root of my issue was a third-party application (Watchtower) updating the containers. This resulted in mixed up networking that supervisor could not handle.

My only solution was to stop HA completely, delete all existing containers, and then rebooting the box, letting supervisor restore everything back to the way it expected. I also had to “readd” my addons, since they’re managed individually by supervisord.

Thank you Philip for the suggestion. I read that with interest but I am not using Watchtower nor a supervisor process that I am aware of. Unless this is part of Docker itself?
I am using portainer to manage my docker containers.

a third-party application

Portainer is a third party application. :wink:

The HA stack does not like third-parties interfering with it. HA creates static assignments within its network, and when a third-party starts downing/upping the containers, the container ip addresses start changing, resulting in the errors I had.

I know this is an old topic but I’m also getting similar issues with what looks like connection. I’ve detailed them here and I think they might be related:

No solutions so far.

Hi, I’m facing the same sort of behavior but I manually manage all the docker containers.

What I’m seeing:
Node-red is able to connect to Home Assistant and receive entity updates shortly after deploying (re-deploying does indeed solve this, but isn’t really a solution) the flow. After some time (in the range of minutes to hours, haven’t yet narrowed it down further than that) the node-red-contrib-home-assistant-websocket state nodes in the flow stop receiving inputs.

I can confirm that the flow is still functioning as I’m able to receive and log MQTT events, and can confirm that the entity in Home Assistant is indeed updating, by viewing /config/devices/device/{device_id} in home assistant while interacting with the device. Therefore the only part of the chain that stops is events coming from Home Assistant making their way into Node Red.

The environment:
rPi 4
Home Assistant in Docker and network_mode: host (manually installed and run with the docker command below)
Node red installed with docker and network_mode: host

Home Assistant start command

docker run --init -d \
  --name homeassistant \
  --restart=unless-stopped \
  -v /etc/localtime:/etc/localtime:ro \
  -v /path/to/homeassistant/config:/config \
  --network=host \
  homeassistant/raspberrypi4-homeassistant: stable

Node Red Dockerfile

FROM nodered/node-red

RUN npm install node-red-contrib-actionflows \
    node-red-contrib-home-assistant-websocket \
    node-red-contrib-stoptimer \
    node-red-contrib-time-range-switch \    
    node-red-contrib-timecheck \
    node-red-node-timeswitch

I don’t have the know-how to debug what might be going on with the websocket connection but it certainly seems like the only sensible conclusion to come to based on my observations and reading this thread.

Hi all,

I’m also late to the party. I am running Node-RED as an add-on to Home Assistant, which is running in HassOS on an rpi4.

I am also experiencing websocket disconnects, although they seem to take about a day.

When it happens, I know the rest of Node-RED is working because everything else continues to function as normal, it’s only the Websocket In node that doesn’t work.

Here’s what I’m doing. I’m running Rhasspy on a dedicated rpi3B+, with some Pi Zero nodes that connect to it. I’m using a websocket connection to have Node-RED listen to the detected intents that Rhasspy picks up, and take action accordingly.

But the Websocket In node periodically dies. I know that the rpi3B+ running Rhasspy is still working because I can still talk to it and have it acknowledge my speech, I can ssh into it, and I can still log into its web interface and make it say things.

Nevertheless, the websocket loses connection. Re-deploying the relevant flow doesn’t fix the problem. The only way to get the connection back is to re-start the Node-RED add-on.

But I don’t like this solution, it’s a kludge. I could create an automation in Home Assistant to restart the Node-RED add-on on a schedule, but I feel like this doesn’t address the problem.

Does anyone know how to make the Websocket In node re-connect periodically? Or else could you recommend another websocket node that has this functionality in it?

Thank you.

I looked into it at one time and I think I traced it back to how websockets are implemented in node-red. If I remember right, there is no keepalive implemented on the connection. There seemed to be confusion across node-red, and node.js as to who is responsible for implementing a keepalive.

My fix was to create an artificial keepalive, a simple MQTT script that publishes ‘ping’ in HA and call that at regular intervals from node-red. That allows node-red to recognize an orphaned connection and re-establish.

1 Like

There’s also limits to how many connections can be made to the websocket. Using poorly configured events all nodes can do the trick. Also if you import flows, make sure that you do not have more than one HA server in the call service drop down menu.

@wuench I am a bear of little brain. If you have time, could you please write a step by step guide of how to achieve what you did? I sort of have an idea of how to do automations in Home Assistant, but it has never made as much sense to me as Node-RED. If I had your guidance, I am sure I could achieve your solution.

Thanks

You just need a script to call, can do anything you want. Mine just broadcasts a simple MQTT message.

alias: ''
data:
  topic: ping
service: mqtt.publish

Then in node-red an inject node that triggers every minute that runs your script in home assistant.

[{"id":"e2d5882e.2f28c","type":"inject","z":"77013be1.720b4c","name":"","topic":"","payload":"TEST","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":1220,"wires":[["cf673d1f.50c1a8"]]},{"id":"ace063b3.c194b","type":"debug","z":"77013be1.720b4c","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":770,"y":1220,"wires":[]},{"id":"cf673d1f.50c1a8","type":"api-call-service","z":"77013be1.720b4c","name":"test ping","server":"26939aab.c81726","version":1,"debugenabled":false,"service_domain":"script","service":"mqtt_ping_service","entityId":"","data":"","dataType":"json","mergecontext":"","output_location":"","output_location_type":"none","mustacheAltTags":false,"x":440,"y":1220,"wires":[["ace063b3.c194b"]]},{"id":"1d745d1d.f3933b","type":"inject","z":"77013be1.720b4c","name":"Keepalive","topic":"","payload":"HASS Keepalive","payloadType":"str","repeat":"60","crontab":"","once":true,"onceDelay":0.1,"x":170,"y":1280,"wires":[["cf673d1f.50c1a8"]]},{"id":"26939aab.c81726","type":"server","z":"","name":"Home Assistant","legacy":false,"addon":false,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true}]

Hi @wuench, thank you, you’ve given me a lot to think about. I’m just coming back to this problem now, after a while.

I looked at your solution and I can basically see what you’re doing, but if I understand this correctly, you are trying to keep a websocket connection to Home Assistant alive - do I read that right? If so, it’s different than my problem - I am trying to keep a websocket connection to an external rpi3B+ running Rhasspy alive.

But I think I can still use your method, somewhat. Do you think it would work, instead of using the HA Call Service node, to use a Websocket Out node to just send some data eg “ping” to the websocket url on the Rhasspy?

Actually it looks like the Websocket Out node has a Heartbeat option.

@wuench It works! Adding a Websocket Out node, unconnected to anything, but pointed at the same websocket URL, using the Heartbeat option, keeps the connection alive.

Glad it is working for you that’s good info.

Might be a good idea for the author of the HA node-red websocket node to compare code with the Websocket Out node and see if a heartbeat could be added. (To be fair I haven’t update in a while it may be in there already).

3 Likes

Seems to do the trick (had disconnect issues to HA, causing NR to restart). Added a web-socket out to the WebSocket of HA, with a heartbeat of 60 seconds, and restart that every 15 minutes.