Very slow compiling process

Hello,
I am in a situation where the download and the compiling time of each ESPHome device is very slow: I recently reinstalled the entire system to Debian 12 bookworm 64-bit, Rpi 3b+, using this time the Home Assistant 64-bit version (raspberrypi3-64-homeassistant:2023.8.1) following precisely the guide for supervised installation (GitHub - home-assistant/supervised-installer: Installer for a generic Linux system) and meeting all the prerequisites, while before I was on a 32-bit Debian 11.
Now my system is much more responsive and works much better but I see a prolonged compilation time on ESPHome: before on the slow OS it was pretty quick the entire compile time for each device, but now it is more than an hour that it is running, but it is still running. The download itself is slow but I have seen the progress (the initial download was like 15 minutes before it was less than 5 minute) the compilation is also slow, every element (*.o) take more than one minute each to compile.
The CPU and memory look fine all the time:

# w
14:47:58 up  4:24,  2 users,  load average: 0.96, 0.85, 0.83
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
XXXXXX
# free -h
               total        used        free      shared  buff/cache   available
Mem:           953Mi       651Mi        41Mi        84Ki       323Mi       301Mi
Swap:          1.5Gi       525Mi       974Mi

Is it like a bug in the 64-bit I am using now?
Anyone can suggest where to look at?
Thank you in advance,
Michele

For all but the simplest of projects, I was never able to compile ESPHome YAML on my 3b+, it would just hard-crash it after a few minutes. I was running HAOS, so maybe your setup recognizes the performance issues and slows it all down so it doesn’t crash? Completely spit-balling… The slow download doesn’t make any sense though.

Does it? Looks like virtually no RAM is left and instead swapping (to sd card?) happens. :grimacing:

The CPU load looks “ok-ish”, maybe the default changed that Esphome only uses 1 cpu core now? :thinking:

What usally comes with the change from 32-bit to 64bit is a higher memory usage for the same task. The Raspi 3’s only got 1GB anyways - depending on how demanding your HA installation is there is really little left for esphome. :man_shrugging:

Long story short: Install ESPHome on your client, it will compile in record time :horse_racing::dash:

Hi,
as I showed, my system is completely working normally, any slowdown on the whole system, just a very slow compilation. It is a pity there aren’t any timestamps on the compilation because I could say more :slight_smile:

The system is on an SSD and it runs fine.

How can I see if ESPHome is running on a CPU or more? Anyway the single CPU should be always at 100% utilization and that is not the case.

Still your system utilizes swap (over usb 2 :warning:) for compiling which is something you want to avoid because it is very slowwwwwwwwwwwwwwww :snail::dash:

Not sure, the settings (esphome ha addon) have a “compile process limit” but it doesn’t show what the default limit is actually.

The default compile process limit. This is the maximum number of simultaneous compile processes that ESPHome will run.

Not always, only if there is stuff to do for the CPU and it doesn’t need to wait for the memory bottleneck because no free RAM is available and using a SSD over USB 2 is magnitudes slower - so CPU sits and waits… :zzz:

Beside a load average: 0.96 pretty much says that one core is utilized 96% if all other cores are idle :bulb:

Hi orange-assistant,
I know about the few memory and it could be as you say if the system is running slow and not responsive and, maybe, the light of the disk always blinking because of the writing (swapping you guess) but it is not what is happening now, the disk became idle after a write and it is fast as never been before: I am telling this while the compilation is occurring right now in the meantime so there is no doubt in what I am writing and no guess.
Any advise?
Thank you in advance,
M.

Running ESPHome on lower powered machines

People have noticed that trying to compile their projects on machine such as the Raspberry Pi 3 would crash either the whole machine or the ESPHome add-on, or Home Assistant. This is due to having less RAM and CPU power available to run all of the things at the same time. A new config option has been added as per the example below that should help to allow compiling on machines such as Raspberry Pi 3. By default ESPHome tries to compile using all of the processor cores, and this allows you to limit it to a lower amount.

So default is all cores, try to limit it to one core only to see if it behaves any better.

But still…

1 Like

Okay,
maybe I understand what you mean, there are problems in the memory management of the 64-bit version so it runs like this and there is no other option for the Rpi 3b+?

Will try it.
Thnx,
M.

It’s not an actually problem. It’s just fact that 64bit OSes need more memory to allocate the same amount compared to a 32bit OS :bulb:

Well, you could go back to 32bit (HaOS) system to confirm the theory - and stay with it if it fits you. :tshirt:

Problem is that the RAM from a Pi is not upgradable and only 1GB for OS+HA+ESPHome and compiling is just :pinching_hand:

1 Like

Anyway, just to analyze it to the end, even the download process of ESPHome uses all this RAM that became 3 to 4 times slower than before? While downloading it is not compiling at the same time I guess.
The test I can do is to turn off all the other services and addons I am running (not much but they can free some memory) and see if it actually changes something in speed. I know it can remain the same situation.

Worth a try. Still I guess ESPHome is actually using most of the RAM when compiling. :chart_with_upwards_trend:

You could use the systemmonitor so your CPU/Memory usage will be saved to history :point_down:

I use “top” and monitor each CPU and see the current memory utilization.
Do you need the data since you are suggesting systemmonitor?

I just stopped now the last compile because it was no more going on (any error but stopped).
I’ve added the line:

esphome:
  ...
  compile_process_limit: 1

and cleaned the previous compilation just to start again from scratch.
It looks like it is back to the old speed, the download was a flash and extraction too, so I think that is enough to hope it’ll be able to finish, I’ll see.

2 Likes

Yes, it is running fine now and I must add that it is the first time I am able to compile and use the esp-idf framework instead of the usual Arduino :ok_hand: With the previous installation, I never was able to compile due to an error in the compilation process.
That is good because it should run a better bluetooth proxy, without the random reboot.