Steps to reduce Write Cycles and extend SD/SSD life expectancy

Community Guides
,
3

Thanks.
I’d think that in a “normal” Home Assistant implementation there are the following sources of disk writes, and depending on how many sensors you have and how frequently they refresh, the order will change:

  1. HA database:
    Capturing the sensor changes and events. This is the only disk writes with real added value to our system, and should ideally be the only ones.
    So you can achieve a lot by just rethinking what you really need, and exclude the rest.
    • If you don’t need to know what time the door closed or the tv was switched off, don’t log it.
    • If you don’t need to see in HA that your phone is charging, exclude that integration.
    • If you don’t need to see the WiFi strength of your MQTT device, disable it (in TasmoAdmin or wherever).
    • Etc.
  2. HA, the app
    • logs generated by internal processes, like hassio_supervisor checks and refreshes. Same goes for hassio_audio, even HACS.
    • automations, scripts, notifications, … are all captured if you don’t exclude them in the logger and other filters.
  3. OS
    Linux is logging stuff all the time. When you do a “sudo” there’s a log created. Samba, VNC, X, cron, … are all quite vocal when it comes to logging.
  4. Other apps (in your container stack)
    e.g. Mosquitto, that in my case logs info like database updates, device connections etc. to a EXT4 log file (means it’s not in RAM but on disk).

As you may have picked up, my biggest gripe is with how Home Assistant just generates lots of crap. And even worse, all these debug info is logged as errors. Thing is that it is completely superfluous for the average user, even when you’re troubleshooting a specific problem. My ticket to change the hassio_supervisor logging was to me wrongfully rejected, so I’m now trying it as a feature request.

1 Like
4

Ok, thanks. I can assume from here https://www.home-assistant.io/integrations/recorder that entities, automations etc. disabled in Recorder will subsequently not be logged in Logbook and History, so above points A.4 and A.5. would be redundant, is that correct? Is there a case scenario/need, in which I would disable something in History/Logbook but have to keep it in Recorder (or vice versa)?

5

Yes, you are correct. In the context of disk writes Logger determines what is stored in the database, and what therefore can be displayed in History, so History filters are not relevant. And that also applies to Logbook, unless you use the Logbook service to add custom Logbook events that “come in from the side” and may bypass the Logger filters. I will update the text accordingly.

I had one small use case, where I used a Node-Red automation to switch on lights in the house based on sunset, that varies over here from 5pm in winter to 10pm in summer. Turned out that it was too dark inside when the event fired, so I used an offset to switch on the lights earlier, based on whether it was cloudy or not. This only worked if the entity (sun.sun) was logged, but I did not care for how it was displayed in History, so I excluded it from the view. Now this is addressed much better after I added a lux sensor to measure the actual light values.

6

Ok, thanks again.
I’m now going through your other proposals/ideas. It is actually recommended here https://ikarus.sg/using-zram-to-get-more-out-of-your-raspberry-pi/ to completely uninstall default pi swap file (dphys-swapfile uninstall) if log2ram is used along with zram. I just wanted to add that for anyone who also wants to use zram, without knowing if it’s even a good idea to uninstall default pi swap.
I take it that for a supervised install the most important changes will be those suggested in point D.?

7

@Tryfos, thanks for actually going through the content and providing feedback. It’s still a work in progress, will update it over time so that people don’t need to read through all the comments.

For journald I’d say the most important part is to make sure it is in memory, by setting the Runtime.. parameters. I actually don’t know what is the default for OOB Debian.

Setting the max log levels would also be an efficient way to immediately prevent all lower-level “noise” from even entering the journal and logs. But, as mentioned, Home Assistant currently logs everything as errors so that is a missed chance. We can’t filter out the debug errors, you will then also miss the error errors. :roll_eyes:

You could also consider updating the rsyslog rules, to prevent all those duplication of messages to different logs. The risk is small if you make a backup of the /etc/rsyslog.conf file to revert back to.

My changes in /etc/rsyslog.conf
(these are only the changed lines, the file contains more stuff that I did not touch):

#- prevent all logging to syslog, by hashing out the complete entry:
#*.*;auth,authpriv.none         -/var/log/syslog

#- only write errors and warnings to other files:
cron.err                        /var/log/cron.log
daemon.warning                  -/var/log/daemon.log
kern.warning                    -/var/log/kern.log
user.warning                    -/var/log/user.log

#- Ignore debug, info, notice and warning messages. 
#- Relevant messages were already covered in previous rules:
*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     stop
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          stop

Maybe you could only write warnings (and above) to syslog, to start with:
*.warning -/var/log/syslog

And I added a new rules file: /etc/rsyslog.d/01-hassfilters.conf

# Ignore all PulseAudio messages that are not errors.
if ($msg contains "pulseaudio") and not ($msg contains " E: ") then { stop }

# Ignore all HA Supervisor info messages.
if $msg contains "supervisor" and $msg contains "INFO" then { stop }

I noticed the ZRAM option, but was hesitant to mention it. All other settings here I actually implemented myself, and been running stable for a couple of weeks now. This I did not do, and I did not want to get burned by leading the guys down the wrong path.

8

In point 3. I cannot find a “purge_interval” variable in the official docs: https://www.home-assistant.io/integrations/recorder
Instead it is recommended to create an automation for the purge intervals.
Perhaps it was available in earlier versions? Relevant, with references to setting purge_interval to 0 for disabling default daily purge, is the following thread: Recorder purge and repack
The latter can now be achieved by setting auto_purge to “false” according to the docs.
In any way the default setting of “1” should be good for our purposes.

9

Yep, the Recorder parameters changed. :astonished: I will update the doc to not confuse the peeps. Thanks…

Regarding your previous comment about disabling swap altogether, I don’t have enough knowledge to say something meaningful about it. Some say swap space is a good thing, others say no. Actually the same guy says no in another article. My take on this is to play it safe - limit the server’s affinity to use swap space (swappiness), like actually don’t use it, but to have swap space available as backdoor when its needed. But this is case specific, a guy running a 8GB RPi 4 with lots of memory available, may think different about it than a 1GB guy.
Would be great if some swap guru could chime in…

10

Actually I referred to a recommendation to disable swap only if zram is used, not in general. After a lot of research I’ve also come to the conclusion that Linux needs a kind of swap, even if it is ultimately not used.
I would also add regarding Database, that the best solution is to put it in memory all together (after the suggested optimizations in Recorder). And use InfluxDB for long-term data in case there is a system shutdown/restart etc.
Further on, in point C.4. the correct syntax for WPA log level is sudo wpa_cli log_level error, not sudo wpa_cli log _level=ERROR . The latter did not work for me.
I think, if log2ram is used (which I installed and seems great!), points D. and E. are redundant in my mind if there is enough RAM in the system. Since all logs of var/log are written to RAM with log2ram, there is little to worry about/tinker. Of course I’ll check the memory more often from now on.
I haven’t installed ZRAM too, since I see that there is a lot of headroom regarding memory. I have the Pi4 4GB version and only 1GB is used for now. I’ll see how things go on from now and react accordingly.
Again, many congratulations for the recommendations!

11

Is there a chance to store the data on a NAS in LAN ?
That is running 247 or on an external USB drive where size does not matter ?

I am asking cause relying on Pi OS and Docker Image I want to have small SD Cards of which I create IMGs that can easily be restored. But the data monster are grabbing 50 MB or more each day and I do not want to go for a 32 GB SD Card which takes ages to backup and the same amount of time to copy back to a PC and restore that image to a SD.

Thanks

1 Like
12

Thank you for this topic. Are there any updates to your recommendations to consider?

I only ran my HA setup with an SSD for 4 months and the SSD is already dead -.-
Next setup will include as many write inhibitors as possible.

13

I had a similar experience with my first SSD, a cheapie from KingSpec. It lasted me a full three months before it crapped out on me. Advice here is to avoid it as the pest. I’m currently using a Kingston A400 (with a UGREEN housing) and it’s running reliably for over a year now.

If you’re using a Raspberry Pi 4, for SSD’s and adapters that work with the RPi 4 see this guide by James Chambers.

14

I am currently looking into all your recommendations and got thinking about the recorder. I opened a feature request in case someone is interested in discussing pros and cons of a split database (remote + local).

15

The most important step which can reduce flash wear by magnitudes is missing here.

Don’t use the default commit interval from ext4 (which is from the hard drive age) but use something at least 100 times higher (or more at best).

This setting can reduce write amplification a lot end extend your sd card life by many years!

and nodiratime too btw.

16

I’m curious: has anybody checked the impact of the MQTT Broker? Even when you filter the messages and only track and store a defined amount of sensors, are not all incoming payloads processed and stored because most have a retain flag (not in the database, but maybe in a temp file or cache)?

17

Hello everyone, I would like to add some sensors that perform very frequent readings related to the use of the cpu, ram, disk occupation, etc. If I exclude them from Recorder, History and Logbook is it sufficient or is there a risk that they compromise the life of the SSD anyway?

18

I’m not sure if any of the developers will be interested but I have logged a feature request here to add some options to HA that might reduce disk writes/extend SSD life expectancy. I would welcome any inputs from you guys there. I’m visualizing some sort of array of checkboxes that might individually contribute to reduction in disk writes, each of which the user could turn on / off as they wish, accepting whatever caveats may come with each option.

Especially for those of using HAOS which makes it that bit more difficult to manually make the excellent suggestions on this page.

19

This might be of your all interest, as this post has inspired me to get a step further down the road using anything sync daemon in order to prevent almost all IO to the disk: Wiki: reducing IO in home assistant (supervised)

1 Like
20

I edited the syslog instructions in order to clean up the syslog, in order to get mainly warnings and errors:

:msg, startswith, " s6-rc: info: " stop

if ( $programname startswith "addon_" or $programname startswith "hassio_" ) and ( $msg contains "[info]" or $msg contains " info: " or $msg contains " INFO: " or $msg contains " [INFO]" or $msg contains " INFO " or $msg startswith " #033[34m" ) then { stop }

if $programname == "hassio_audio" and ( not ( $msg contains " E: " ) or ( $msg contains " E: [alsa-sink-Generic Analog] alsa-sink.c: " ) )  then { stop }

if $programname == "addon_core_mosquitto" and ( $msg contains "New connection from " or $msg contains " closed its connection." ) then { stop }

if $programname == "NetworkManager" and ( $msg contains "<info>" ) then { stop }

if $programname == "hassio_observer" and ( $msg contains " done." or $msg contains "exited 0." ) then { stop }

if $programname == "kernel" and ( $msg contains "hassio: port " or $msg contains "device veth" ) and $msg contains " entered " then { stop }
2 Likes
21

That’s fantastic. Do you think it would take much to have this as an official feature in the likes of HAOS, that could be toggled maybe in some advanced/system menu or even via the configuration.yaml?

It feels so close and could help all of us HAOS users if it was somehow added there as a supported option to reduce disk writes.

22

Essentially it’s really peace of cake to do so, but apparently reducing IO in order to lower the wear Home Assistant is imposing on hardware isn’t really a top priority.

Actually I think all journaling services even could be disabled. Since as far I can see, they aren’t accessible using HA interfaces like the GUI or HA CLI. See for example: https://community.home-assistant.io/t/wth-is-there-no-home-assistant-logs-to-remote-syslog-integration/473949/10. Besides all logging is being done at least twice, because the log files in add-on overview haven’t changed, even after applying these rules (and no logging to syslog).

However, I think keeping things as long as reasonably possible in memory is quite more interesting as it really pays more of > logging is really small potatoes compared to the IO-traffic of the database for example, besides also for logging counts that one sequential write imposes a lot less wear on the hardware than many random writes.