I have a fairly sizeable home automation system and I use AppDaemon for all my automation.
I’m particularly keen on presence modelling in so much where people are within my home in order to set the lights or whatever appropriately.
I do this using PIR detectors, button presses and more recently sonar sensors (used as a kind of trip wire beam across a doorway). I do most of this via a MySensors network.
I have apps to listen to those sensors which start ‘cool down’ timers in each zone/room which gives a nice visual colouring/heatmap to my house floorplan based on how recently someone was there.
Lots more to do here but it works quite well - except for the reaction speed to events.
This is most noticeable when there are several people moving around the house a lot - lights will start to come on slowly, there is slow reaction to button presses and I’ll sometimes see warnings ’ Excessive time spent in utility loop:’. Things always catch up eventually but of course that isn’t quite good enough.
I’m running HA on a Raspberry Pi 3B (not +) in a Python virtual env and AppDaemon runs on the same Pi (but not in the venv).
Load on the Pi appears to spike >1 during those busy times, often to 1.5 or more.
CPU usage on HASS seems to hang around 10-30% but AppDaemon is 20-60% almost all the time (even when there no ‘action’ in the house). Then add in my periodic nmap scan taking ~15% CPU so at busy times the Pi is definitely working hard to keep up.
I’m considering upgrading the Pi to an NUC (like the ASRock Beebox perhaps) but wanted to ask if that will really solve my problem or that maybe I’m doing something wrong in my apps? I’m careful to cancel timers/listeners so things don’t multiply. My handlers do have some work to do - mostly checking states and doing the appropriate service call/state set. Nothing that looks particularly ‘heavy’ from a programmatic point of view. My ‘cool down’ timers (just an integer sensor- seconds since confirmed detection) were initially a problem as they updated every few seconds but I switched that to every 20 seconds which improved things. But I do feel that a few counters like that shouldn’t be that hefty for a modern application.
Any thoughts? Should I just get on & upgrade my hardware, run HA in a Docker on an NUC - or - am I do something crazy and my load levels look excessive and should rework my apps?
Advice appreciated!
This does not seem right. This is a dump of my PI 1 B+ usage over the last 24 hours. I don’t have as many sensors, but that shouldn’t make much difference to AD
If you have a large number of apps, production_mode might help as AD spends a quite a lot of processing deciding if it needs to reload apps.
i got 4x AD running on 1 system. and i got hundreds of apps, still i hardly see any CPU load.
in the years i gathered a lot of experience with AD, and what i noticed most is:
utility loop warnings are mostly generated when you make excessive use from network. updating sensors is obviously also 1 that comes to that because a sensor is in HA and AD connects to HA over the network.
any app that keeps a thread busy for to long is a potential problem. make sure that there are no internet calls that take to long, dont use things like time.sleep, etc. an app should be an accespoint (for instance a state change) a short moment to handle things (for instance turn lights on) and finish (for instance some logging)
external modules or outside programs make that it seems that AD is using the cpu load. for example if you use things like subprocess to call a shl script or vlc or another non py program, or when you got libs imported that do some datacalculations.
disk access. writing data or reading data on a RPI is giving trouble. on an RPI its important to keep the access to the SD card to a minimum. over time the SD card will go bad, and make sure you have a backup SD card lying around. when i used a RPI for my AD and HA, i quickly connected an external disk where i moved all data and the config areas from AD and HA (at least the HA DB should not be on the SD card)
its very well possible that your SD card is slowly going down. another system would at least get rid of that problem.
AD 4.0 is in the making at the moment and it wont be long before the beta gets released. with that version you will get way more info about which apps get called a lot, which ones take up a long time, etc.
that kind of data will amaze a lot of us, and show how heavy we really use our stuff.
i am a heavy user, but i never expected that i have literally thousands of callbacks every hour.
but a little calculation brings us far. (a sensor that is updating every 10 secs gives 360 callbacks every hour)
so my advise:
yeah, a better system then a pi will be good thing for every heavy user (i got a beebox)
and secondly take a closer look at your app with the things i said in mind
and third have a little patience for AD 4.0 which will give you way more insight.
Great thanks for the reply!
Figuring out this slow down is certainly proving tricky. I’d already displaced the HA database to my MSSQL instance on my Windows Server box which is consumes vast amounts of disk & memory space. I still have the config on the SD card (backed up daily) so I guess AD is scanning my apps there regularly. I tried turning on production mode but didn’t notice much difference.
I’ve put in a good router to try to eliminate any network issues between HA-AD on the Pi, the HA database + InfluxDB + MQTT broker on the Windows Server and the lighting/MySensors gateways.
Uprating to a Beebox seems a good ploy to eliminate cpu issues and host other unix services on. The Pi is also running a NextCloud instance on it but low traffic though- mostly for the calendar feature. I don’t really know the Pi’s limitations but NextCloud doesn’t run very quickly so maybe I just need something more powerful.
AD4 sounds music to my ears! Would really love an overview of what’s going on in there.
I don’t feel like I’m working the threads that hard and I only ever use AD’s functions for timing.
Perhaps I make 5+ get_state() calls in a handler and reference other apps too. I do setting of a state or two. I’m not using any internet calls or long running scripts. The main states I set in apps are ones I create in AD by using set_state. Otherwise just the light service calls and occasional MySensors message sends.
I can replicate it so easily by pressing scene set buttons on a touch panel for lighting. At first it responds immediately but if you keep toggling from one scene to the next it will eventually slow down and only slowly catch up visually. I can be lagging 15-20 seconds behind if you do it long enough. The whole house responds slowly for a while after that till things calm down.
what kind of entities are in the scenes?
it sounds like that 1 or more entities take longer to respond (zwave perhaps? i heard that zwave can slow down when commands are to frequent)
Would really love an overview of what’s going on in there.
My lighting is Osram Lightify (so Zigbee) and I do wonder if it is holding things up. If the component is locking up it might explain it. Will keep trying to isolate it.
For sure that AD UI looks great, should really help with things!
I’ve had a go with the dev version- and wow what an improvement!
The UI overview is great, I can really see what’s going on in there. Would like to expand the columns a bit as some of my entities etc don’t fit but otherwise looking brilliant.
But what’s even better is that my whole system runs significantly faster! Now I can walk into a room and the light is on very quickly. Before there was often a 1 second or more delay. It’s more like 1/4 of a second now.
I can press buttons to call scenes and it keeps up a lot better although does start to slow after a while. I can see some thread starvation issues going on but think I’m in a lot better place to find out what I’m doing wrong. If nothing else less complaints from the wife over slow lights
Credit to the work being done here, really good stuff!
thanks for the compliments. ill forward them to @aimc who is doing all the heavy work on this.
the UI is still a work in progress, so please feel free to comment where you think there is room for improvement.
i think in the future it must be possible to set columnwidth, colorthemes, etc.
maybe even set order of columns and select columns.
i think you are not on discord yet?
if you want to talk more about the UI and what you miss or change, feel free to come by
Very pleased to hear that 4.0 is giving performance improvements - a lot of work has gone on under the covers, in particular I have made the core 100% async (it was a kind of hybrid before) and have added thread pinning which gives each app it’s own thread. I had hoped that some of this would help improve performance in certain situations, and I am glad to hear it has for you!
