Client exceeded max pending messages

tom_l · December 30, 2019, 5:11pm

This has been asked before and no satisfactory explanation has ever been given.

How do I work out what the heck is causing this error to occur up to 90 times a second?

2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512
2019-12-30 23:24:03 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140145700964624] Client exceeded max pending messages [2]: 512

Tinkerer · December 30, 2019, 5:20pm

What integrations, and particularly custom components, have you installed?

tom_l · December 30, 2019, 5:27pm

Untitled

I’ve just been looking at my logbook, nothing obvious happened around the time it started.

There was no appreciable increase in CPU load or network throughput (in or out of home assistant).

finity · December 30, 2019, 6:03pm

That was asked by (at least) me and I never did get an answer. I even tried to go the discord route and nothing was ever resolved there either.

The good news is that it eventually resolved itself after a month or two.

I’m not sure but I have a suspicion it might have been my main PC that I use as the main “always on” control panel for HA. I know my browser would slow way down and when I restarted the PC the errors would stop for a bit. However, the same PC is still running and has the same slow downs now too but the errors haven’t returned in a long while.

I can’t say if that was it but it might be something to check into.

tom_l · December 31, 2019, 12:34pm

Found what was causing it and can reproduce 100%

Open and edit a Lovelace card in Firefox (have not tested Chrome yet).
Close the card edit and exit UI edit mode, do not refresh the browser.
Navigate to another view or sidebar menu a few times.
Sooner rather than later Firefox starts throwing up rapidly repeating “Refresh Lovelace” toast notifications in the lower left corner. This may eventually cause the screen to show a blank page with a useless “Refresh Lovelace” button in the centre. This causes thousands of errors in a very short time.

Standard browser refresh clears the error loop.

So I guess I just have to remember to refresh my browser after editing a card.

EDIT: I opened an issue: https://github.com/home-assistant/home-assistant-polymer/issues/4404

petro · January 9, 2020, 2:36am

Just ran into my page going nuts and did a CTRL+F5 to refresh the page. Ended up looking in the logs and saw this

2020-01-08 21:31:08 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140433372066768] Client exceeded max pending messages [2]: 512

Specifically came on here to tell you that a browser refresh removes the issue because I remember you posting this. But it looks like you figured it out already.

Anyways, can confirm. Clear cache and refresh the page.

rccoleman · January 17, 2020, 5:53pm

I just started seeing a large number of these errors in the last week or so and I think I’ve tracked it down to a Home Assistant browser window left open on a machine that I don’t use frequently. I was getting the “exceeded max pending messages” constantly and after turning on debug for homeassistant.components.websocket_api, I could see an initial connection for a specific client ID and then almost Lovelace constant updates coming as fast as possible. On my laptop, which is my main machine, I tried everything I could think of - clearing the cache, trying a different browser, temporarily clearing out ui-lovelace.yaml, restarting HA, etc., and nothing helped.

I then thought about the other machine that I use periodically and noticed that I had the HA Chrome app open (installed via the “Install…” option) from a week or two ago. Closing that instance immediately stopped the constant stream of “exceeded max pending messages” errors in the HA logs.

After being quiet last night, I noticed that there were more “exceeded max pending messages” in the morning, albeit a smaller number. Simply waking up my laptop to investigate seemed to stop them.

I’m wondering if something has changed to make Lovelace or one of the custom cards less tolerant of a browser session going dormant (such as a machine going to sleep). There definitely seems to be a positive correlation between these errors and having an inactive HA browser session somewhere, and it’s a new phenomenon since 0.103 or 0.104. I do have a number of custom components and cards, so I’ll keep poking around.

These are the cards that I’m actively using:

resources:
  - url: /community_plugin/lovelace-hass-aarlo/hass-aarlo.js
    type: module
  - url: /community_plugin/weather-card/weather-card.js
    type: module
  - url: /local/github/thomasloven/card-tools.js?track=true
    type: js
  - url: /community_plugin/lovelace-fold-entity-row/fold-entity-row.js
    type: module
  - url: /community_plugin/upcoming-media-card/upcoming-media-card.js
    type: js
  - url: /community_plugin/mini-graph-card/mini-graph-card-bundle.js
    type: module
  - url: /community_plugin/lovelace-card-mod/card-mod.js
    type: module
  - url: /community_plugin/custom-header/custom-header.js
    type: module

And for custom components, I currently have aarlo, alexa_media, hacs, isy994, plex, plex_recently_added, smartweather, templatebinarysensor, and templatesensor installed. I’m not actually using the last two, though.

Edit: I just noticed that I’m using an old version of card_tools, so I fixed that and will continue to monitor.

pjv · January 26, 2020, 12:07pm

Another data point, though I’m not sure what this indicates, if anything, because from scanning everything I could google up related to this error log message, it seems like there are multiple potential causes for this symptom. For instance, the “solution” post in this thread is totally non-relevant to my situation. There are zero Firefox clients for my HA instance, and I do not use the Lovelace editor (ever) because I have Lovelace in YAML mode.

Anyway, for a couple weeks I had been having the same issue with my RPI-based hassio instance getting totally overwhelmed by tens of thousands of “client exceeded…” error messages each few minutes. I did everything I could figure out to debug, including everything anyone in this thread mentioned and other stuff as well. I updated everything I had to latest versions. I set the websocket_api to debug logging and carefully traced through thousands of log messages. I cleaned up everything in my main hassio config and my lovelace config, removed any custom code I was not using (and even some that I was using), and etc.

Some things I did, including several times doing wipe and restore from snapshots, seemed to have fixed the problem for a while, but it would eventually come back within 24 hours.

I had been planning for a while to migrate my HA instance from the RPI to an intel NUC home server running linux that I have doing other stuff but with plenty of spare capacity. I needed to first come up with a solution for handling a binary sensor input for which I was using the RPI GPIO pins. I did that using a $4 nodemcu-esp8266 card (awesome!) + ESPHome (awesome!!!). So I then migrated my HA to the NUC, restoring my latest snapshot from the RPI and since then - now almost a month later - my logs have been perfectly clean.

finity · January 26, 2020, 2:44pm

I’ve got almost the same as you do now and I was still getting the same issue.

It would be ok for a few dates and then for a day or two it would go crazy with those thousands of errors. I think I narrowed it down to the chrome browser on my pc I always use to view the UK. I found if I restarted the pc the problem would go away for a few days but would eventually return…until one day it didn’t come back. And it now hasn’t happened in a couple of months. Tho as far as I know nothing has changed except my HA version.

andriej · March 30, 2020, 11:36am

Here is the issue on github, if you could participate in bug-hunting it would be great. It seems (but not sure) that issue is caused by custom cards using websockets, but nevertheless HA is not handling them properly and destroying it’s performance.

stratotally · April 1, 2020, 5:34pm

Having to log into all my tablets and force a remote refresh any time I fiddle with the front-end is not the solution I was expecting…

Anyone know if there is an event that fires when the front-end/lovelace is updated? I could then trigger a remote API call in FullyKiosk to reload the page it’s displaying.

stratotally · April 1, 2020, 6:24pm

Well, at least I can control Fully Kiosk reloads using this - makes is super simple.

Thomas does it again!

stratotally · April 6, 2020, 6:30pm

So, just wanted to update. There is a “lovelace_updated” event that you can listen for in NodeRed, and then use browser_mod to reload the lovelace on your browsers. However, while it does reload the lovelace, it doesn’t get rid of the “refresh” notification at the bottom of the page.

So instead I used fire an HTTP request node to “reload” the start URL in fully kiosk using the Rest API, which is super easy. HTH.

Fraschi · April 25, 2020, 9:43am

Same here with 0.108.8, didn’t know how to handle this.

I didn’t use node red, only some custom_cards: lovelace-card-tools, search-card, gauge-card, plan-coordinates,

I get a lot of:

 [homeassistant.components.websocket_api.http.connection.139674928668176] Client exceeded max pending messages [2]: 512

until it crash and needs a restart

Closing the browser tab during the issue generates the following error:

2020-04-25 11:29:55 WARNING (MainThread) [asyncio] socket.send() raised exception.

At this time it get worst, I can’t restart hass, I get the first error at startup without any browser client connected… Only Appdaemon I suppose

Fraschi · April 26, 2020, 10:48am

After many tries I’ve found that the “problem” is hacs and/or card-tools… I have to find if directly related to that integrations or if a sum of other issues… Let me explain:

Removed all lovelace resource from gui
Removed all hacs integration and hacs itself
Commented out entities that uses other custom_components
Cleaned manually all the folders from custom components and resource
Removed (after 2 restart as mentioned in the hacs removal guide) the hacs files in .stored folder

During the clean i’ve found a lot of unused old stuff and some js that where initially installed manually and then via hacs… So probably the issue is from that confusion beside the custom_components itself…
Now, in these days, i will re-add progressively all the appdaemon,components and entities I’ve removed or commented out to see if the problem re-appears…
One thing is sure, until the steps described above, the issue was still there.

petro · April 26, 2020, 11:17am

Dude, all you gotta do to get rid of the error is CTRL+F5 on all the tabs that are open when the errors occur. If you have 3 tabs open with HA open, thats 3 CTRL+F5s with each in focus. Then it goes away. No cleaning, no crashing, no restarting. Just clear the cache and move on.

Fraschi · April 26, 2020, 11:20am

I know, but in my case CTRL+F5 did not solve the issue

petro · April 26, 2020, 11:23am

Are you using firefox or chrome? Both have had issues where 3rd party extensions break CTRL+F5. Try manually clearing the cache.

Fraschi · April 26, 2020, 11:26am

Chrome, now I’ve solved with the steps above… now, when reintroducing extensions probably I have to deal with ctrl+f5

Fraschi · April 29, 2020, 7:52pm

After more investigation I’ve found that the problem comes from appdaemon (from the client id). Didn’t know if appdaemon is the cause or a symptom. (All appdaemon apps are running for years without issues till now - AppDaemon V 4.0.3)

Here what I’ve found:

That’s from Home Assistant Log --> The client is appdaemon (the only client)

2020-04-29 21:14:05 ERROR (MainThread) [homeassistant.components.webhook] Error processing webhook cb2a6ba9b5719bbc9ca5cc794e7f3aeffde42ddf8c51a950206f38ed1204e58c
2020-04-29 21:15:58 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140692626535536] Client exceeded max pending messages [2]: 512
2020-04-29 21:15:58 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.140692626535536] Client exceeded max pending messages [2]: 512

That’s from AppDaemon (things starded go bad a little time before):
After Some Messages Like this:

2020-04-29 21:12:27.404924 WARNING cloro: Coroutine (<coroutine object ADAPI.set_state at 0x7fcb742552c8>) took too long (10                                                                 seconds), cancelling the task...
2020-04-29 21:12:27.408978 WARNING HASS: Timeout in set_state(default, sensor.cloro2, {'state': 'Out Of Range'})
2020-04-29 21:12:27.410966 WARNING AppDaemon: callback ppm() in cloro has now completed

I get:

2020-04-29 21:13:30.925645 WARNING HASS: Error setting Home Assistant state default.sensor.cloro1, {'state': 'Out Of Range'}
2020-04-29 21:13:30.926696 WARNING HASS: Code: 502, error: <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3</center>
</body>
</html>

After Some messages like this one, I get the error in Home Assistant log, that degenerates in unresponsive ui…

Meanwhile in nginx:

2020/04/29 21:13:00 [error] 4974#4974: *8530 connect() failed (111: Connection refused) while connecting to upstream, client: ::ffff:127.0.0.1, server: sitename.duckdns.org, request: "POST /api/states/sensor.cloro2 HTTP/1.1", upstream: "http://[::1]:8123/api/states/sensor.cloro2", host: "127.0.0.1"
2020/04/29 21:13:17 [error] 4977#4977: *8508 upstream timed out (110: Connection timed out) while reading response header from upstream, client: ::ffff:127.0.0.1, server: sitename.duckdns.org, request: "POST /api/states/sensor.cloro2 HTTP/1.1", upstream: "http://127.0.0.1:8123/api/states/sensor.cloro2", host: "127.0.0.1"
2020/04/29 21:13:20 [error] 4977#4977: *8511 upstream timed out (110: Connection timed out) while reading response header from upstream, client: ::ffff:127.0.0.1, server: sitename.duckdns.org, request: "POST /api/states/sensor.flag_ph HTTP/1.1", upstream: "http://127.0.0.1:8123/api/states/sensor.flag_ph", host: "127.0.0.1"
2020/04/29 21:13:24 [error] 4974#4974: *8541 connect() failed (111: Connection refused) while connecting to upstream, client: ::ffff:127.0.0.1, server: sitename.duckdns.org, request: "POST /api/states/sensor.flag_ph HTTP/1.1", upstream: "http://[::1]:8123/api/states/sensor.flag_ph", host: "127.0.0.1"
2020/04/29 21:13:30 [error] 4974#4974: *8517 upstream timed out (110: Connection timed out) while reading response header from upstream, client: ::ffff:127.0.0.1, server: sitename.duckdns.org, request: "POST /api/states/sensor.cloro1 HTTP/1.1", upstream: "http://127.0.0.1:8123/api/states/sensor.cloro1", host: "127.0.0.1"

and then:


`2020/04/29 21:25:01 [error] 4977#4977: *8713 connect() failed (111: Connection refused) while connecting to upstream, client: ::ffff:192.168.1.211, server: sitename.duckdns.org, request: "POST /api/webhook/<xxxxxxxxxxxxxxxxxxxxxx> HTTP/1.1", upstream: "http://[::1]:8123/api/webhook/<xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>", host: "sitename.duckdns.org"`

As it seems is the API that starts not responding to AppDaemon and then this causes all the things

Any Idea?