Excessive time spent in scheduler loop

All of a sudden I am seeing a sever slow down of my automations. My log is full of “Excessive time spent in scheduler loop” and “Scheduler clock skew detected” - see log below:

/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 07:47:23.954571 WARNING Excessive time spent in scheduler loop: 4018.0ms
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 07:47:23.956001 WARNING Scheduler clock skew detected - delta = 3.023887872695923 - resetting
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 07:57:25.391482 WARNING Excessive time spent in scheduler loop: 5432.0ms
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 07:57:25.393098 WARNING Scheduler clock skew detected - delta = 4.437093019485474 - resetting
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:07:29.227897 WARNING Excessive time spent in scheduler loop: 8831.0ms
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:07:29.229230 WARNING Scheduler clock skew detected - delta = 7.836138010025024 - resetting
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:17:25.014035 WARNING Excessive time spent in scheduler loop: 3781.0ms
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:17:25.015410 WARNING Scheduler clock skew detected - delta = 2.786176919937134 - resetting
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:27:27.277029 WARNING Excessive time spent in scheduler loop: 5258.0ms
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:27:27.278439 WARNING Scheduler clock skew detected - delta = 4.263017177581787 - resetting
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:37:29.730642 WARNING Excessive time spent in scheduler loop: 7449.0ms
/home/homeassistant/AppdaemonLogs/appdaemon.log:2017-09-24 08:37:29.731469 WARNING Scheduler clock skew detected - delta = 6.45307183265686 - resetting

Any ideas for me to debug?

Thank you for your help!

Tom

Its happening every 10 minutes, so do you have anything that runs regularly in that sort of period?

its a warning that you are overloading appdaemon.
so you have something running that takes up to much time, causing that AD runs behind schedule.

due to the fact that its every 10 minutes, you probably can easy figure out what it causing the overload.

Yes - I agree … it should be easy - but I am struggling with it! It appears that this is not at all related to Appdaemon or HASS! My entire Pi is running very (very!) slow. Even when I shut down Appdaemon and HASS it is still running so slow. Seems to be related to my network - even typing in putty the letters are lagging as I type. Multiple reboots and I can’t figure it out. “top” not showing high CPU utilization, but anything that I do that goes over the network - either WAN or LAN and it is very slow.

Thanks for your help…
Tom

if your network has problems then you cant be sure if the RPI is slow when you use putty.
the delay can be from the network delay.

do you have other things connected to your network that use it excessive?

but that shouldnt give such a regular warning, unless …
do you have an app or dashboard that checks a website every 10 minutes? weather or calender or such?
with network problems such a thing could hold up your AD, resulting in such a warning.

Did you ever get to the bottom of this? My RPI is showing the exact same symptoms you are describing, randomly after not changing anything for over a week. Removing apps that run periodically doesn’t help

Sort of…I think it has mostly gone away, but I can’t tell you exactly what I did. I do know that I was having some issues with a couple of my apps that were still a bit buggy around starting multiple listeners for the same event (although, I am not at all convinced that that was the the problem!). Also, I moved from the RPI to an Oracle VirtualBox environment running Ubuntu and that may also have helped to fix the problem.

Sorry, I don’t have a more definitive answer :confused:

Tom

Do you have the same 10 minute interval between messages?

Yes exactly the same 10 min interval, even tho no apps were running per a 10 min schedule.

However, i found that it was due to the fact that the Arlo component introduced a bug with the latest release which blocks I/O, thus making the machine very slow. A pull request for this has already been made in #9892 (not by me)

So disabling the component removed this problem for me

1 Like

Do you have an app that tries to interact with the Arlo component?

yes one app that arms/disarms the cameras

Ive got two Yeelights and having the same issue. Tips?

I found out that appdaemon does check if anything hass been changed in hass, every 10 minutes.
if people have problems with delay every 10 minutes that means there is a slow connection to hass.
several things can cause that.

  1. if you have appdaemon setup so that it connects to an outside dns, (like duckdns) you go with everything over the internet. anything slowing down in that connection also slows down appdaemon.
  2. a router problem causing a slow home net also causes appdaemon to slow down.
  3. a very busy homenet (because off tv, cameras, etc.) also will slow down appdaemons connection with HA

to make sure that you minimize the problems:
make sure that appdaemon connects to HA locally (through http://ip) and not through outside DNS
appdaemon is designed for homeuse. it can be used outside the homenet and even with encrypting, but that will for sure slow things down.

1 Like

This makes a lot of sense to me … but when I specify the internal IP address, appdaemon will not connect. Here’s my appdaemon.yaml file:

AppDaemon:
  logfile: /home/homeassistant/AppdaemonLogs/appdaemon.log
  errorfile: /home/homeassistant/AppdaemonLogs/appdaemonerr.log
  logsize: 100000
  log_generations: 3
  threads: 10
  time_zone: America/Chicago
HASS:
  #ha_url: https://xxxxxxx.duckdns.org     <--- This Works!
  ha_url: https://192.168.1.22:8123          <----This does NOT!
  ha_key: xxyyxxxxx

If I enter the address https://192.168.1.22:8123 into Chrome, I get a website not secure message - I’m assuming that’s why appdaemon can’t connect.

What am I doing wrong?

Thanks for your help!

Tom

that has to do with the way homeassistant is setup.

if encryption is set directly in homeassistant it seems that it isnt possible to connect to homeassistant without encryption and without the DNS that is linked to homeassistant.

thats why i use NGINX as frontend.

that gives me 2 options:

  1. from outside: duckdns> router> NGINX> homeassistant
  2. from inside: ipaddress> homeassistant
1 Like

That appears to be my problem since I am using encryption directly in HA. Looks like I should maybe look at NGINX.

Thank you for your help and insight…

Tom

1 Like

Thanks, I set AD to run using my local IP and yet have the same issue. Perhaps there is some traffic congestion, i’ll have to check it out. The Yeelights are rather chatty…

do you have your device where HA and/or AD are running, running on wifi or on hardwire connection?
if you have a device for HA and AD make sure you have it hardwired with the best possible connection to your router.
anything else can cause problems, more then you like for home automations.

Oh yes, im 100% wireless for the moment. The Pi3 is also my multimedia center so is next to my tv and the router is in the living. However, the latency between then is 20-60 milliseconds at most.

now you know where the small latency is coming from, and you know that if you get more stuff running in the future it could get worse.
but most off all, if it gets to be a problem, you now know how to solve it :wink:

1 Like