Hello,
After having some discussion with @orange-assistant in the Home Assistant OS 8 release thread I started doing some I/O write improvements research. Let’s discuss the results in this new thread.
The main goal is to find if ext4
commit intervals, tuned VM dirty writeback timeouts (vm.dirty_expire_centisecs
) and potential other tunables can have a positive effect on writes. The goal is to essentially cause fewer writes to the storage, which should lessen the write amplification effect typically seen on flash storage.
For testing I use a iomonitor.sh script, which is a envolved version of the armbian monitor command found here. My version only looks at the actual write I/O’s merges and bytes written to the main data disk reported by /proc/diskstats.
My test setup is a virtual machine with a vanilla HAOS 9.0 development build, and a vanilla Home Assistant Core 2022.8.7 configuration with SQLite. No add-ons other than the Terminal add-on is installed. I did some Idle measurements as baseline, but those are really not that useful. Most tests run with a single automation which increment a counter every second. This changes the state of an entity every second, which essentially causes the recorder to record a state every second. So this should cause continuous writes.
Note: HA Core recorder by default pools state changes, so SQLite is not causing a transaction every second.
The test is somewhat crude, and mainly represents a simple HA installation. Add-ons (e.g. different database backend) etc. could generate different results. But I think this tests should still be helpful to make a somewhat informed picture on the tunables.
Test | I/O Count | I/O merge | I/O kbytes |
---|---|---|---|
Idle, Commit 5s, VM 30s/5s (vanilla) | 90 | 41 | 453 |
Idle, Commit 30s, VM 30s/5s | 50 | 42 | 274 |
1s Increment, Commit 5s, VM 30s/5s (vanilla) | 572 | 187 | 31139 |
1s Increment, Commit 30s, VM 30s/5s | 381 | 99 | 27316 |
1s Increment, Commit 60s, VM 30s/5s | 415 | 88 | 27702 |
1s Increment, Commit 60s, VM 60s/30s | 483 | 123 | 28215 |
1s Increment, Commit 120s, VM 120s/60s | 431 | 101 | 26994 |
1s Increment, Commit 600s, VM 30s/5s | 475 | 108 | 30257 |
What puzzles me a bit is that higher Commit caused fewer I/O merges. I guess the merge happens at a lower layer, hence when fewer writes hit that lower layer there are also fewer opportunities for I/O merges… In any case, the main value we try to optimize is I/O Count, which is the effective completed I/O operations on the block device.
I’ve looked into iotop, and all writes are basically from Home Assitant Core itself. I’d assume most of them are generated by the recorder.
So my interpretation of the results are the following: The difference is measurable, but not massive. As I wrote in the HAOS 8 release thread, higher levels do some write pooling already, which should also translate to fewer writes even with a 5s commit interval, and this seems to be largely true.
However, we can definitely save some writes. It seems that we have about ~30% fewer writes when increasing the commit interval to 30s. It seems that after 30s there is really diminishing returns to increase the value further. I’d guess that is because the Recorder commits data around this rate.
Thoughts, other ideas?
Best regards,
Stefan