Python3 high CPU Usage

Hi, recently…i believe with the 103 update i have an ever increasing cou load on my homeassistant container. Checking with Glances, it turns out to be the pyhton3 process. When i restart, it lies within normal 3-9% but over the course of 12h it goes up to more than 100%. This is on a Intel NUC.
It seems not to be any of the usual culprits like mqtt or mariadb.
Is there a way to easily check what is using python3 so much?

So what i did so far:

Turn off ALL Custom Components :white_check_mark:
Disable almost all integrations: :white_check_mark:
Disable ZWave: :white_check_mark:

None of that helped in isolating the problem or identifying it.

Some help in the Discord Chat pointed me to analyze the running python 3 process. I used a tool called Py-Spy for that. https://github.com/benfred/py-spy This allows you to profile a running process and even works with Python 3.6 and above.

Now I have a sample output from a fresh restart, and a sample output from several hours later when my process monitor actually shows the high CPU Usage.

However, I still don’t understand what processes are actually calling the different functions in python. Thought I’m sharing the results, maybe somebody has an idea on what is using so much of my CPU and how to fix it.

Freshly restarted:


image

Running for several hours:


image

A big difference seems to be the initialize connection (socket_client.py). Any Idea what that is?

I suffer from the same problem. I initially had HA running on RPi3+B (HassOS), but it was crashing over night. Then I moved it to ESXi (HassOS as well) and same thing was happening. As of today I have assigned to it 2 CPUs (Xeons) and 2GB of RAM with vm hosted on SSD. At least it is not crashing anymore, but the CPU runs high shortly after reboot. I have very straight forward setup with few addons (mosquito, zigbee2mqtt, ssh, samba, configurator and node red) I have disabled all of them, but no change. I’m really getting sick and tired of not being able to look in historical logs. This is my second shot at HA in two years and it seems it always fails in the middle of the project. I’m really close on giving up on it :frowning:

For me it resolved itself after updating to 105 and redoing the integrations like netatmo, spotify, etc the new way.