High CPU usage after 2022.12

3MAJ · December 12, 2022, 8:59am

Hello,

I am currently running HA 2022.12.3 on a RPi 4 2GB with SSD and I haven’t had any major issues I couldn’t resolve either on my own or by finding useful information in this community, for which I am very grateful. However, troubleshooting not so straightforward issues is what grinds my gears, especialy if it’s causing home automations to fail or have a noticable lag.

Ever since I upgrade to 2022.12.0 (currently running latest version 2022.12.3 as of writing this post), I have noticed an increase in CPU usage and overall lag in UI and automation response (light automation is most obvious).

Here is a snip of my RPi/HA usage stats over the past 7 days. It is clear when I installed 2022.12 - when average CPU usage went from 5% to 15-20% and when storage started to get filled linearly. I have set DB/log purging and storage usage was also constant until latest major version.

Can anyone help me troubleshoot what is causing this sudden increase in CPU and storage usage?

amaximus · December 12, 2022, 9:16am

In case you have some cameras you might consider reading this:

github.com/home-assistant/core

Drop aiohttp to 3.8.1

home-assistant:dev ← home-assistant:revert-aiohttp-to-3.8.1

opened 01:32AM - 12 Dec 22 UTC

balloob

+3 -3

## Breaking change  ## Proposed change  Drop aiohttp to 3.8.1 from [3.8.3](https://github.com/aio-libs/aiohttp/releases/tag/v3.8.3). 3.8.3 included "fix" https://github.com/aio-libs/aiohttp/issues/6719 to align the behavior of aiohttp with the documentation, which is that request handlers no longer get cancelled when the client disconnects. We did not anticipate such a behavioral change in a patch release and so didn't catch this in time. It seems like it contains a bug where a `StreamResponse` will not raise when written to, causing our MJPEG streams to eat up all the memory (as it writes a new image every 0.5s to the stream without any back pressure). In https://github.com/aio-libs/aiohttp/pull/7056 a new parameter was added that allows enabling the old cancellation behavior. This has not been released yet. Before upgrading, we should explore if we want cancellation and if we don't, we should fix the `StreamResponse` eating all the memory. Two tests got fixed in #82788 for the 3.8.3 upgrade, which are at least 2 spots that would need to fix. Fixes #83661 ## Type of change  - [x] Dependency upgrade - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New integration (thank you!) - [ ] New feature (which adds functionality to an existing integration) - [ ] Deprecation (breaking change to happen in the future) - [ ] Breaking change (fix/feature causing existing functionality to break) - [ ] Code quality improvements to existing code or addition of tests ## Additional information  - This PR fixes or closes issue: fixes # - This PR is related to issue: - Link to documentation pull request: ## Checklist  - [ ] The code change is tested and works locally. - [ ] Local tests pass. **Your PR cannot be merged unless tests pass** - [ ] There is no commented out code in this PR. - [ ] I have followed the [development checklist][dev-checklist] - [ ] The code has been formatted using Black (`black --fast homeassistant tests`) - [ ] Tests have been added to verify that the new code works. If user exposed functionality or configuration variables are added/changed: - [ ] Documentation added/updated for [www.home-assistant.io][docs-repository] If the code communicates with devices, web services, or third-party tools: - [ ] The [manifest file][manifest-docs] has all fields filled out correctly. Updated and included derived files by running: `python3 -m script.hassfest`. - [ ] New or updated dependencies have been added to `requirements_all.txt`. Updated by running `python3 -m script.gen_requirements_all`. - [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description. - [ ] Untested files have been added to `.coveragerc`.  To help with the load of incoming pull requests: - [ ] I have reviewed two other [open pull requests][prs] in this repository. [prs]: https://github.com/home-assistant/core/pulls?q=is%3Aopen+is%3Apr+-author%3A%40me+-draft%3Atrue+-label%3Awaiting-for-upstream+sort%3Acreated-desc+review%3Anone+-status%3Afailure  [dev-checklist]: https://developers.home-assistant.io/docs/en/development_checklist.html [manifest-docs]: https://developers.home-assistant.io/docs/en/creating_integration_manifest.html [quality-scale]: https://developers.home-assistant.io/docs/en/next/integration_quality_scale_index.html [docs-repository]: https://github.com/home-assistant/home-assistant.io

It might be valid for non camera usage as well.

I had the same on 2022.12.1 (see HA 2022.12.2 causes high CPU load · Issue #260 · JurajNyiri/HomeAssistant-Tapo-Control · GitHub), reported it and I’m now upgrading to 2022.12.3 in the hope it’s fixed.

3MAJ · December 12, 2022, 9:35am

I have 2 cameras but no image/video analysis (Frigate), only a picture card that pulls the image via RTSP, aiohttp is not used (or I don’t know about it). The issue you linked states that is eats up memory which didn’t change on my side, only CPU and storage usage, I don’t think it applies to my case sadly.

Storage space was reclaimed after a hard reset though, I had to power cycle RPi while upgrading to 2022.12.3 since it was stuck for more than 30 minutes.

The update to 2022.12.3 did not fix my issue with CPU usage but I will continue to monitor storage usage as well.

jlambert121 · December 12, 2022, 10:20am

I had extremely high I/o on my pi after 2022.12.1 (skipped.0) I couldn’t find and rebooting seemed to have significantly improve. Hopefully the reboot helps the storage at least.

tescophil · December 12, 2022, 2:28pm

It’s not…, my CPU is now double what it was before the upgrade to core-2022.12

3MAJ · December 14, 2022, 1:26pm

An update on the situation is due. I have found multiple issues with automations after upgrading to 2022.12, which caused HA log file to grow constantly (I had “logger” enabled).

First, I have disabled logger to stop log file from growing and CPU usage to drop, then I fixed all issues with automations (had to delete a few automations in yaml file, that were not visible in UI).

Everything is back to normal, it seems.

Lunkobelix · January 4, 2023, 12:24pm

I have very high CPU usage as soon as I view Home Assistatn on web or app. And as soon as I close down/web/app it is back to normal. So something is making the CPU go bananas just by browsing around in Home Assistant. Very frustrating. Started a few weeks ago.