Your swap file is being used, to 53%, which probably indicates significant memory paging is taking place. Memory page thrashing is always going to lead to long pauses in program execution (and disc wear).
When paging is taking place, both disc and CPU activity is focused on that. With several add-ons that may be partially used, memory pages are possibly being swapped out then required again on an alternating intermittent basis.
This is not necessarily a case of having too much in memory, but rather having stuff that is not used and has a high swapiness factor, so the pager moves it to swap, then it is required again later, having to be swapped back in.
Looking closely, the used memory is 1.8G, which is about half of available, being a typically balanced approach taken in memory management. HA, as the top process, is using 2.0G, which to me suggests that HA itself is being actively paged, which maybe is the primary reason for poor performance. Constant thrashing.
Running glances on my own, Odroid n2 machine (HA Blue) I note that I have the same basic memory size as you. 4GB total, 3.7GB available, 1.8 used. The only real difference is that I have 6 core, utilised at 0.2, Home Assistant process is CPU 16% and 1.6G memory, and my swap file is almost empty. No swapping, and HA is not overloading a core [but then, I don’t do very much on my HA].
You could try tinkering with swapiness, or even turn the swap off, but 50% free memory and a slightly-used swap file is preferred to running out of memory when large buffers are called for.
Adding memory is not always physically possible, so perhaps it is time to upgrade to a Pi5 with 8GB.