Log files against the SD card

Do you know the base OS? I expect it to be RaspberryOS or Armbian. The commit interval for the ext4 root filesystem will then probably ether be 5 seconds or 600 seconds.

I try to make a simple explanation:

First thing to know is that any type of flash storage contains blocks and pages. The smallest possible amount to be written is one page that can typically keep 16 thousand characters/letters (16KB) or even more.

let’s make the simple assumption that every second one character is “send” towards the filesystem (doesn’t matter if that’s from the logger, the database or whatever). The filesystem layer will “collect” this characters and writes it to the flash storage in the case there is enough for a page or if commit is forced:

  • In case the commit interval is 5 seconds it happens that every 5 seconds a whole page which could hold a total of 16 thousand characters is wasted for only 5 “usable” characters (bytes). That means that instead of writing 3600 characters (bytes) in one hour the flash storage is worn out if there would been 11.796.480 (3600/5=720 > 720x16384) characters (bytes) written. To calculate the write amplification factor (waf) we only need to take divide the bytes written by the ones we wanted to write: 11.796.480/3600 = 3276,8.
    In this simplified example the amount written to the flash storage in one hour was over 3276 times higher than the amount we intended to write :writing_hand: And the flash storage was actually worn out by that high amount and not the one we intended to!

  • If a more flash friendly commit interval of 600 seconds is used we can calculate the same:
    3600/600 = 6 > 6*16384 = 98.304. So this time we have a waf of “only” 27,3 (98.304/3600)

Best would if a OS from the 21th century just ship with default settings for the 21th century :tada:

And what do you think your SSD is using ? Hint: MLC. Or even worse, TLC or QLC.

Actually, high endurance SD cards using vertical TLC (‘3D NAND’ in marketing speak) in pseudo-SLC mode (at least SanDisk high endurance cards do, Samsung probably too) are more reliable than a consumer grade SSD (which will not run M/T/QLC in pSLC mode, because that would mean they’d lose storage capacity per price unit, and that’s all people care about).

You can easily check your commit interval(s) by having a look into your fstab (cat /etc/fstab).

If you don’t see a number like for example 600 after the fs type (ext4) but a default(s) instead it will be most likely the flash unfriendly 5 seconds :face_vomiting:

A average small setup probably doesn’t even create 16KB of data in 5 seconds so in the end “saving” here can make no different on the wear of the flash storage. It can be actually the opposite and increase the waf. For example instead of “asking” to write 2KB every 5 seconds you might save 50% and only ask 1KB to be written every 5 seconds… but in the end ether way one full page get’s written resulting in 16KB valuable flash worn :do_not_litter:

This is actually the least harmful (most efficient way) to write to flash. Large contiguous blocks of data or sequential writes. Other than the small bits and bytes this doesn’t introduce any significant write amplification and is nothing to really worry about.

This small junks together with a small commit interval is what breaks flash fast - this is the thing to worry about!

RaspianOS
PRETTY_NAME=“Debian GNU/Linux 11 (bullseye)”

fstab:

proc            /proc           proc    defaults          0       0
PARTUUID=01ff7241-01  /boot           vfat    defaults,flush    0       2
PARTUUID=01ff7241-02            /       ext4    noatime,nodiratime,defaults     0 1
# >>> [openmediavault]
/dev/disk/by-uuid/27d9fd83-3122-4b4f-b4e1-2cc7e2c1de64          /srv/dev-disk-by-uuid-27d9fd83-3122-4b4f-b4e1-2cc7e2c1de64      ext4    defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl      0 2
# <<< [openmediavault]
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

defaults = 5 seconds for ext4. That’s the classic sd card killer :boom:

But (to be fear) it includes two minor tweaks: noatime, nodiratime which avoid a writing to the filesystem when a file or folder is accessed! (logically there is no “last accessed” time stamps than available).

You now have the option to actually monitor the “page writes” to flash before you optimize this value to be more flash friendly (and then compare it with the “page writes” afterwards).

The armbianmonitor features a function to get this stats and despite it’s name it should run on raspbian/raspberryOS and debian too.

A quick look at RPi Zero 2 W # SD card endurance (the later what matters)

If you love your SD card then Raspberry Pi OS defaults are not for you: swap on SD card, default ext4 commit interval and logging to card.

It seems like I were lucky, I have a home assistant running 24/7 in my home, with many addons, on an sd card. And it is running since 6/7 years ago!