While on the hunt for my memory leak (it seems everybody has one ) I realized that my memory consumption jumps up (and is not dropping back down) when I have very high CPU utilisation.
As I seem unable to track memory consumption for components directly I started looking at the CPU Consumption.
However what “top” gives me does not make much sense to me.
If I understand that output correctly (please confirm) i have 0% idle time, and 84% of processor time spend in User Mode and 15% in Kernel Mode (sorry! I am a Windows guy by trade ). However none of the Usermode processes show any CPU Consumption!? How can that be?
How can I figure out what is consuming all available CPU time at random intervals?
My Home Assistant is running as VM in an Hyper-V Server (in case that matters).
yes it looks weird , can’t help you alot, in this context, my cpu is around 1-2% ( including sys, where the top command seems most active
However you dont tell much in regards to whether that high CPU is a “constant” or maybe after an upgrade, startup etc.
I have my HA in a VM, in VMware, on windows, and windows system monitor can be a certain help in some cases, i.e does you have a lot network-usage coming/going to HA. ( This was actually the case for me at one time where i also had unexplained high CPU, turned out that i.e “streams” in HA is not very smart. … so even if you didn’t have a browser open, with the cam-views, the (ALL CAMS) were steaming, so if this is not fixed yet, i wouldn’t integrate a 4K cam, not to mention handful … this “area” seems like it’s very quite about, in regards to usages from HA, and in local network
but indeed i would also love that there was better options to monitor the various processes in HA
The high CPU Usage was sporadic at some times, usually for half a minute or so, then dropping back down and comming back to 100% a minute or so later. Completely random. I was not able to establish any pattern or relation to any changes.
I say “was” because accidantely I figured out that one of my scripts, using a loop waiting for a change in power consumption bigger then a certain value was running unexpectedly aggressive.
Since I put a delay in that loop to always wait for 3 seconds, it has become a lot better. So CPU usage wise I am good now, but the Memory Leak remains
Yes scripts and automations like the one you describe ( loops , monitoring/listening to entities(some) etc. and even some “realtime functions/ templates” in cards can consume memory.
And to be honest “wait 3 seconds” (most likely instead of 1 seconds) for a change in power consumption, did in your case cause MORE power-consumption … good example where one have to think twice before adding “features” … no matter how you look at it, you apparently created a script which is in “conflict” with it’s purpose, so maybe you should look at another option/ way of accomplice you “needs”
In this case i would think ( if you monitor your power consumption/cpu-usage etc. as you do, likely with various graphs/counters) , then use “Thresholds” in these cards, to use as “Triggers”
Lately i’ve seen alot of people ( fascinated by the various Template/Script-Options, which is not wrong , i love the flexibility it give us ) , but i.e creating additional entities, to extract/show i.e attr , or listening for entities(which one already is "monitored in a card or other functions) , is just like pouring water in a glass until it runs over, instead of looking in/through the transparent glass, and stop when it’s full …
However it did not produce more power consumption
My things producing more power consumption are elswhere in my config
But you have a very valid point with redundant work / functions.
I am currently thinking about redesigning / streamlining my entire load handling concept.
Maybe you have some ideas to share?
I would much appreciate your input!
Regarding the HA Script that was to aggressive:
What it essentialy does is to run a loop that kicks of a phyton script, that checks how much power is currently be send to the grid and depending on that find the right combination of certain devices to use up exactly that amount of power (yes, thats a very long story short). This Phyton Script takes between 30 to 80ms to run and then execution focus returns to the HA Script.
The next step for the HA Script is to get the current Excess Power and write it into a variable.
Then it waits for the current Excess Power to be over +30 or -30 Watts in comparison to the Value in the Variable with a timeout of 20 Seconds to execute anyway if there was no sufficient change and repeat the loop.
This generaly works well… however what I did not realize back then is that depending on the Consumer the decission to switch on or off the consumer can produce a significant change in the Excess Power that is not instant but over the next seconds. This triggered and retriggered and retriggered the Script. That Script (including the Physcript) executing constantly seems to have been what was eating up my CPU.
What I did was to put that 3 Second delay between storing the cuirrent value in the variable and checking against it.
Its still agile enough to cover my needs without putting a lot of stress (max. 25% overall) on the CPU even in times of rapid changes (cloudy but sunny days).
Yeah that a “valid” reason for “monitoring” , unfortunately,or not I haven’t come to that part, yet, i spend my money on travels, when i had “over” so i still haven’t got solar-power, thou have “prepared” for it during my renovations of the heating-system and ventilation, and i “save” bookmarks everytime i see some “good” solutions/scripts etc , in here , so still my main focus is on my current device(HA-Households) and in particular my 3 laptops and 2 PC ( im also Windows guy, by choice and profession ) always loved GUI, dropNdrag, Copy/Paste ever since first Windows ver. ( unfortunately Windows have made it more difficult to “avoid/disable/delete/uninstall” alot of progs/services, it was pretty easy until WIN2000, but still in windows 10 i manage to “slim” it, and i guess that “behavior” also is reflected in my HA-setup (what do i need ? etc) … LOL, i know im “failing” alot by the years, more and more devices, one gotta have some “fun”
PS: in regards to your Brainstorming
i suggest you take it step by step, meaning, i.e pt2 "action trigger switch on/of(by another trigger) spite it’s already is in “executing” state(which doesn’t exist :), then if you “publish” the related/involved scripts/templates/manual “triggers” , it would be easyer for others to help with a “solution”
And yes it would involve yet another, or change in scripts/templates, verifying against all parts ( if is_state ) etc, also a “delay” might be needed when 1 trigger(first) has executed on/of ( give a damn about me, im busy!)
Not sure whether it matters, i ones tried Hyper-V in windows, and it actually had higher usage than what i see in VMware in Windows, beside i find back-up way better, as i backup the whole VM-folder, while i have to restart HA anyways, all configs are also easy to /copy-paste, when ever, or by script
Thats why I put up 4 solar panels on the balcony of my rental apartment end of last year and ever since I am optimizing my HA Control. One gotta have some “fun”
Including the Inverter the entire setup (id did the work myself) cost me 1200€. That is excluding all the Shellys that moved in since
The entire thing is up and running since early november. With the current electricity price of 54 Ct/kWh I saved allready 170€ on my energy bill. So essentialy next week the first of the 4 pannels is payed for and that despite very foggy rainy winter months.
Admitedly I now am consuming power for stuff that I did not before. I had one insekt lamp instead of 3, I had no electric towel heater, no Air Conditioning unit, no electric room heater. But hey… if I feed it to the grid I get absolutely nothing for it.
Its all small loads though… 4 x 400 Watts sounds like much, but on panel is east, one is west and two are south. All are very steep so 70° and 80° out of the horizontal. (90° would be vertical). Hence the peak production I have seen so far was slightly above 1kW.
The main thing is that several components such as the Air Dryer, Air Conditioning, UPS Battery Charger, Bitcoin Miners, etc. assign themselfs a certain priority based on some factors.
For the Air Dryer its the Humidity in the Bedroom, for the Battery Charger the Charge level of the Battery and so on. The Miners have fixed Priorities and are just there to use up excess power.
The System goes from Prio 0 (always runs independant of Solar power) to Prio 8 (only runs when all previous Prios are allready enabled).
One Script just checks if there is enough excess power to run all consumers on the next Prio (there is a changable factor i.e. 80% needs to come ouf of the panels, 20% can come from grid plus a hystersis)
If the power is sufficient all consumers on the next prio are enabled and the current prio increased by one.
30 Seconds later I check again. If the Excess power is now below the threshold the consumers are disabled again and the prio lowered by one.
So walking up from Prio 0 up to Prio 5 (its the higest one while I am at home and my PC is running) takes some time. Also when big consumers like dishwasher or oven drop off, it takes time before the priority system has ramped up again. But thats ok… its mainly for consumers that you dont want to switch of every minute or so. Works pretty well after some finetuning, but generaly the concept is very sound I believe.
In order to make use of the power while my PrioSystem is ramping up I have a group of devices that I call QuickLoads. They can be switched on and off on demand even in short intervals. Those are things like an electric towel heater, insect lamps, a small electric room heater. That script that was too aggressive was controling these QuickLoads.
Maybe Concept-Wise this is something you want to bookmark for the time when you start playing with solar.
But I get the Windows licenses for free and while I am not an expert I am fairly proficient in Hyper-V and would need to start with VMWare from scratch.
Shouldn’t hold you back, VMware Workstation Player is free(still ) , and yes i do find your thoughts here and your brainstorming interesting, i’ve seen lots of ideas/solution, while searching the Forum, some way over my scripting/template “competence” , but never to late to learn new.
Some of your ideas/thoughts is actually direct applicable to my Water-heating system, which i build my self, main source is fire-heating-boiler(currently), but with additional source options, like electricity, and other sources in 2 extra multi-tanks( all 3 units have electrical heater-units installed ) , and im in the “startup” of adding an excaust-air-heat-pump(to water), to the equation(but taking the air(mainly) from the top of a 70sqm black tin-roof, facing south-west) beside using the same(hot-air) for the garage/basement, lots of Pipes here
) I realized that my memory consumption jumps up (and is not dropping back down) when I have very high CPU utilisation.
). However none of the Usermode processes show any CPU Consumption!? How can that be?
