High CPU usage -- potential steps to diagnose? -- possible solution!

An idea! Should there be more than one SyncWorker_n thread for any value of n? It would seem that there should be only 1 SyncWorker_6 thread, for example. When I get threads that are fully utilizing a cpu core they always have the same “n” following SyncWorker_.

I’m not sure where/when those are created.

It’s very odd! I can see where the SyncWorker threads get created as part of Threading. Later, one of the SyncWorker threads seems to get copied … but only sometimes.

I have no idea if it is within HA or some other part of python.

Update: I had it happen again with the thread being ImportExecutor. My best guess is it happens sometime between starting the Recorder thread and the DbWorker threads.

This is as far as I think I can easily get –

  1. All seems fine up through the Recorder thread initializing
  2. A thread shows up in ps named “Thread-2 (_run_safe_shutdown_loop)” - this might be before or after the cloning – I’m confirming relative timing.
  3. Before DbWorker threads are created, one of the previously created threads appears to be “cloned” – could be SyncWorker_n or ImportExecutor. These threads are not created by threading.Thread like other threads in HA.
  4. The “clones” of the thread consume ~100% of a cpu core. The original of the thread seems to operate as normal.
  5. The number of clones created is related to the number of cores I let HA use. It is number of cores - 1.
  6. So I think I have solved my problem by restricting HA to run on only 1 core. My setup is so simple that this is more than enough CPU.

I tested all of this on a different Android device, running a Android 11 (my main device is Android 7) and the behavior is the same.

I can try to figure out what is happening between the Recorder thread and the DbWorker thread that might be causing this “cloning” but since I have no prior knowledge of the system, that feels like a complicated fishing expedition. Maybe I’ll try that in a month or two if I have more time.

If anyone has thoughts, please share!

I installed HA on a Mac using Docker -- same problem with duplicate (or zombie) threads. I think that is a supported installation, so should be diagnosed.

Posted the issue here: Duplicate (zombie) threads in container (docker) installation · Issue #174359 · home-assistant/core · GitHub

I may have a solution. If I set one or more of the following environment variables the problem appears to go away.

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

It seems that there is a conflict between multithreading in HA and multithreading in OpenBLAS.