How to deal with long computations properly? (Excessive time spent in utility loop)

Third party integrations AppDaemon
#1

Hi, I wrote an AppDaemon app that controls a power station in order to shift electricity use to times of lower prices. I am using the A* (A-star) algorithm to calculate the charge plane a day in advance. Running this algorithm can take >10 seconds to complete. It works fine, but I keep getting warnings in the log file:

2022-10-19 21:50:44.768316 WARNING AppDaemon: callback control_battery() in battery_manager_living_room has now completed
2022-10-19 21:50:44.818072 WARNING AppDaemon: Excessive time spent in utility loop: 13043.0ms, 13042.0ms in check_app_updates(), 1.0ms in other

This leads me to believe that I am doing something wrong. I already tried making the app async, I unpinned it, and I use the run_in_executor thing mentioned in the docs. But the warnings keep coming. Could someone point me in the right direction how to deal with callbacks that take a lot of time actually computing something (and not just waiting for IO)?

This is the code and the function call that takes so long: appdaemon_battery_manager/battery_manager.py at a73c343fba34219c9cb422181679e28bb3d50ce8 · ast0815/appdaemon_battery_manager · GitHub

Thanks in advance!

#2

I have run in to the same issue and cannot seem to find the root cause. My implementation is made in a similar way as yours, so if you find a solution, please post it :slight_smile:

#3

Have you tried using numpy? Have you tried using pythons cache decorator Here is a video how?

#4

I am already caching the things that can be cached. And I am using a library that does the A-star algorithm for me, so I cannot speed that up. I am in no mood to re-write that myself.

The problem from my point of view is not that the algorithm takes >10 seconds to run. I am happy with that. The problem is the lack of feature or lack of documentation that tells me how to properly have functions that run that long without producing warning messages in the logs.

#5

You don’t really need to make the entire app async and kind of discouraged if you’re not very familiar with it. Try to narrow down the entry point of your callback code running into a single method that can dispatch other methods and use run_in_executor() or create_task() if the methods are written as async. I think there is probably a number of things in your code that is holding up the thread when the app’s callback is running.

#6

Sorry the correct answer is to set the thread_duration_warning_threshold

Blockquote
thread_duration_warning_threshold (optional) - AppDaemon monitors the time that each tread spends in an App. If a thread is taking too long to finish a callback, it may impact other apps. AppDaemon will log a warning if any thread is over the duration specified in seconds. The default is 10 seconds, setting this value to 00 will disable the check.

#7

I set that setting to 30, but I still get these messages:

Excessive time spent in utility loop: 3710.0ms, 3709.0ms in check_app_updates(), 1.0ms in other

This is below the default 10 seconds of the setting, too, So I assume the utility loop is blocked by the long-running thread of the app? Any ideas how to avoid that?

You don’t really need to make the entire app async and kind of discouraged if you’re not very familiar with it. Try to narrow down the entry point of your callback code running into a single method that can dispatch other methods and use run_in_executor() or create_task() if the methods are written as async. I think there is probably a number of things in your code that is holding up the thread when the app’s callback is running.

So not all methods need to be async when the app is async? I had to make all sorts of API calls await because they were switched to async behaviour. Or do you mean I can have a sync app and only call a single async method in it?

#8

Well, if the entire app is async then obviously any method requiring an await will need to be async. What I’m suggesting is that you do not need to make the entire app async. Write the app as sync but dispatch your long running methods with run_in_executor (if they are sync) or just write those methods as async and use create_task() to dispatch them to the event loop. Using those built-in helper methods will make AD responsible for clean-up and you don’t have to do it yourself. If you use those async helper methods, your code will be dispatched to the async event loop and the callback will complete and you wont see any errors.

Also, I don’t believe you’re making proper use of async. When you’re doing a lot of awaiting in async code, you’re code is still sitting around waiting (blocking) and holding up the callback from completing.

#9

I just realized that the warning appears even when using run_in_executor!

For example:

await self.run_in_executor(lambda: time.sleep(2))

WARNING AppDaemon: Excessive time spent in utility loop: 2054.0ms, 2053.0ms in check_app_updates(), 1.0ms in other

Isn’t the purpose of run_in_executor exactly to schedule long sync jobs?

#10

I am still very unclear on how the run_in_executor is supposed top work. According to the docs, I cannot provide a callback with that one. Also it seems like run_in_executor needs to be awaited, so I can only use it in an async app? Then what is the point?

I tried to use run_in_executor in a sync app, but then I get either an error message about it never being awaited (if I do not await it), or that I cannot have an await in a sync function.

#11

run_in_executor turns a sync fuction into async, so that you can you use it in any async context. Since you cannot do:

async def async_function(self):
    await sync_function()

you do

async def async_function(self):
    await self.run_in_executor(sync_function)

It will run sync_function in a thread (so that it doesn’t block the event loop) and return when sync_function finishes.

#12

So it is “just” a wrapper and does not really do anything if I can as well write my own function as async in the first place? At least it does not seem to do anything about the timing warnings, as you found out too.

#13

I don’t have a ton of experiencing using run_in_executor() as I always try to find an async method of doing things so hopefully what I am saying here is accurate. I believe the built-in run_in_executor() (wrapper) method has some tracking built into it so if AD is shutdown or the app is restarted, it can clean up the tasks. Also, I do not think you need to use run_in_executor() from within an async method and you definitely do not need to await run_in_executor(). It is simply a nice easy way to tell async to create a new thread for your blocking code to run and allow the callback to complete its execution as it no longer has control of that code. If you were to use the built-in create_task() method then yes, the method you are calling needs to be async but you don’t need to await the create_task() method.

Edit:
Also, remember, when you await something, you are essentially telling your code to block and wait for something to complete. Sure, this allows the async loop to process other tasks but it still holds up the app thread which means it’ll still throw errors that it is running for excessive time.

Second edit:
Ok, just realized I basically repeated what I said in an earlier post but what I said earlier in this post still stands. Do not await run_in_executor() or create_task() (if you decide to use that). These are sync methods, they are not meant to be awaited, you’re just blocking your app by doing so.

#14

Well, I tried to not await the run_in_executor method, and then I got a warning message in the log, that it was never awaited. So I assumed that is not the intended use of it either.

Maybe I will try create_task next, and see if I can get that to work.

#15

Are you still trying to run it un-awaited within an async method?