Hey guys and gals,
recently I started to do some benchmarking of various hardware platforms I have at my disposal. After all I was curious about some metrics, given that Raspberries aren’t really the recommended platform anymore.
Luckily Home Assistant (at least with the Core-Installation) includes a benchmark function specific to some tasks Home Assistant performs internally. Below you’ll find a table with my results, my opinion about the results, my testing methodology, and some further details on the machines if been doing this on.
Maybe this will inspire others to benchmark their systems, eventually allowing us to compose a list of hardware platforms and how they theoretically perform in regards to Home Assistant. I would be especially interested in the ODRIOD platform that has become popular recently. Maybe someone even comes up with a Repo for a Jekyll-driven site that puts all the data nicely together.
Anyways, here is the result-table. I have abbreviated some table headings. They match the order of the selected benchmarks described in the testing methodology.
Results
| Machine | f_e | s_c_h | s_c_e_h | f_e_i | v_e_i | j_s_s | Idle (W) | Load (W) |
|---|---|---|---|---|---|---|---|---|
| Eee PC | 8,68 | 30,76 | 19,71 | 2,24 | 5,06 | OOM | 12 | 16 |
| Pi 3 | 8,80 | 29,86 | 16,83 | 2,35 | 4,98 | OOM | 2 | 3 |
| Pi 4 | 3,97 | 12,65 | 8,08 | 1,05 | 2,09 | 37,83 | 3,5 | 4,5 |
| Mini PC | 3,54 | 15,60 | 10,26 | 1,02 | 2,13 | 84,94 | 2 | 3 |
| i7 6700K | 0,64 | 1,96 | 1,15 | 0,17 | 0,36 | 7,40 |
Notes:
- OOM stands for “out of memory”, which resulted in the process terminating and not completing the benchmark.
- For every measurement: smaller is better
Opinion
For a while now I have been using a Raspberry Pi 3 for Home Assistant. I don’t have that many automations, and stuff like the SQL for the recorder is running on a different machine. So generally the system does pretty well for my requirements and doesn’t see much load. But I had that Mini PC laying around for quite a while as well - doing nothing - and thought maybe it would be a better choice. I really hate the poor IO on the Pi, and some of you probably are familiar with dying SD cards. So the Mini PC with its internal eMMC storage and an Intel Atom x5-z8350 processor might be the better alternative. And it absolutely is. It has mostly double the performance compared to the Pi 3 at the same energy consumption. One observation I can’t explain: the json_serialize_states benchmark was really slow in the first 2 iterations, but then significantly faster during following ones. Neither the Pi’s nor the Desktop PC were showing this behaviour.
What could make this platform interesting for others might be, that the CPU has Intel QuickSync support. This means that it’s capable of Hardware-accelerated transcoding when using Plex on the same machine. A nice article about this can be found here, where someone is running Plex an Atomic PI board.
Since I got a Pi 4 for Christmas, I chose to benchmark it as well to see if that again would be a better choice than the Mini PC. Overall it’s slightly faster. Especially the json_serialize_states performs way better. But it also consumes about 1,5 watts more than the Mini PC.
I also had a really old Asus Eee PC 901 laying around and was surprised, that it actually keeps up pretty well compared to the Pi 3. The big downside though is, that it’s a 32bit processor. And for how it performs it consumes way too much energy. So it’s not really a system to run Home Assistant on for anything else but testing.
Finally, just for comparison, I did the same benchmarks on my Desktop PC (Windows 10 in a Debian WSL) to get a feeling of how far behind those small platforms are. And it for sure crushes all of them. But as you can see I didn’t even bother to check the energy consumption. Because of the harddrives, Graphics card etc. it eats too much energy to be useful as a system that’s running 24/7.
My conclusion so far is, that the Mini PC actually is a good choice. Because of the more robust eMMC storage it wouldn’t be necessary to rely on a Pi 4 + non-SD-storage. And the transcoding-capability for Plex might be very attractive as well (provided this would also work within the Plex-addon, which I haven’t tested). Being an x86_64 CPU could be beneficial in some usecases as well. Even though nowadays most software is available for ARM CPUs, pretty much anything will be available on x86.
Testing methodology
Except with my Desktop PC and the Eee PC, I always booted the machine from a freshly installed Operating system. Which ones exactly will be mentiond along with the machine details below. For the Eee PC I chose to boot from a Knoppix live-image to not mess with the installed system.
Once I had a shell, I have installed some required libraries, created a virtual environment for Python, installed the core version of Home Assistant 2020.12.0 (this one still supports Python 3.8), and then ran the benchmarks. If you want to benchmark your system in a similar fashion, this is essentially what I did on the shell:
sudo apt update
sudo apt install python3-venv python3-setuptools python3-dev build-essential
cd /tmp
python3 -m venv venv
source venv/bin/activate
pip install homeassistant==2020.12.0
The benchmarks
Home Assistant includes several benchmarks, like firing a ton of events, doing stuff with entity IDs, whatever. As I didn’t really set up Home Assistant, not all benchmarks were working out of the box. At least I assume that’s why they didn’t start. Hence I stuck with the ones that were (almost) running on all of the platforms I have benchmarked. All benchmarks ran for 3 iterations, except json_serialize_states, which ran for 6 iterations. The reason for the additional iterations is, that the Mini PC started poorly, but gained speed the longer it was running.
I then added the results of each iteration, and divded the sum by the number of iterations to get an average. So for example: 1.2 + 1.3 + 1.1 = 3.6 / 3 = 1,2
Before starting the benchmarks I let the systems settle for a moment after preparing everything. I then wrote down the current enery consumption as measured by my rather basic energy-monitoring-plug (which is measuring in 0.5 steps). That way I had the baseline consumption of the machine while being idle. Then while running the fire_events benchmark, I kept an eye on the consumption as well and wrote down a rough average of what the meter was reporting (sometimes it was jumping by 0.5 - 1.5 watts).
These are the benchmarks I was running. The order of the list corresponds to the table headings from left to right, starting at the second column:
- fire_events
- state_changed_helper
- state_changed_event_helper
- filtering_entity_id
- valid_entity_id
- json_serialize_states
These benchmarks can be started with the command hass --script benchmark $BENCHMARKNAME, where $BENCHMARKNAME refers to one of the available benchmarks above. After the required iterations have passed, I have terminated the process by pressing CTRL+c.
The hardware
Asus Eee PC 901
CPU: Intel Atom N270 (32bit)
RAM: 1GB
Storage: RAM (live Linux distribution)
Operating System: Knoppix Live-Linux
Raspberry Pi 3 Model B
CPU: BCM2837 -> ARM Cortex-A53
RAM: 1GB
Storage: 16GB Samsung class 10 MicroSD
Operating System: Raspbian OS (Buster)
Raspberry Pi 4 Model B
CPU: BCM2711 -> ARM Cortex-A72
RAM: 4GB
Storage: 16GB Samsung class 10 MicroSD
Operating System: Raspbian OS (Buster)
Wintel Pro Mini PC (no vendor resources, there are tons of variations on Amazon, Aliexpress etc.)
CPU: Intel Atom x5-z8350 (64bit)
RAM: 2GB
Storage: 32GB eMMC
Operating System: Ubuntu 20.04 LTS
Desktop PC
CPU: Intel i7 6400K
RAM: 16GB
Storage: Samsung NVMe SSD
Operating System: Debian (Buster) on Windows 10 (WSL2)