Too much disk writes

JF002 · November 18, 2018, 4:52pm

Hi,

I run home-assistant in docker on an Odroid-XU4 board (running on a SD card). I recently upgraded the system from debian 8 to debian 9, and reinstalled HASS from latest docker image (0.82.1).

I quickly noticed that the system was too slow, due to continuous disk IO from home-assistant causing high CPU load.

Here is the output of iotop:
Total DISK READ : 0.00 B/s | Total DISK WRITE : 262.93 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 376.13 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
2360 be/4 root 0.00 B/s 262.93 K/s 0.00 % 93.53 % python3 -m homeassistant --config /config

I see ~300KB/s written on the SD card, coming from home-assistant.

CPU load :
17:45:05 up 3 days, 20:44, 2 users, load average: 1.37, 0.79, 1.16

In htop, I see that home-assistant burst from 5 to 20% every ~5s.

I use a basic Sandisk SD card, which is not optimised for continuous IO, and I think that continuous writes from home-assistant is not good : it slows down the system and wears the SD card.

I don’t know why home-assistant is writting so much, on which file,…

Note that as soon as I stop the HASS docker, the load and disk writes go down to nearly 0.

The configuration is really simple : 3 MQTT temperature/humidity sensors (they publish messages every 1 minute) and 2 media player (from logitech media server).

Any idea to diagnose this issue?

Thanks!

gpbenton · November 18, 2018, 5:42pm

To start with, I would remove the recorder section from configuration.yaml to disable the HA database to see if that is what the problem is.

JF002 · November 20, 2018, 7:48pm

This section does not exist in my configuration file.
I added it, but the behavior was the same.

Here is my config file:

homeassistant:
  name: Home
  latitude: 50.58
  longitude: 5.5333
  elevation: 246
  unit_system: metric
  time_zone: Europe/Brussels

frontend:

config:

mqtt:
  broker : 192.168.1.109

http:

updater:

discovery:

history:

logbook:

sun:

sensor:
  - platform: yr
  - platform: mqtt
    name: "Température"
    state_topic: "/bd:a0:f0:7f:cf:5c/temperature"
    unit_of_measurement: "°C"
  - platform: mqtt
    state_topic: "/bd:a0:f0:7f:cf:5c/humidity"
    unit_of_measurement: "%"
    name: "Humidité"
  - platform: mqtt
    state_topic: "/4f:2f:3c:94:1:60/temperature"
    unit_of_measurement: "°C"
    name: "Température bureau"
  - platform: mqtt
    state_topic: "/4f:2f:3c:94:1:60/humidity"
    unit_of_measurement: "%"
    name: "Humidité bureau"
  - platform: mqtt
    state_topic: "/d:9e:f0:7f:cf:5c/temperature"
    unit_of_measurement: "°C"
    name: "Température salon"
  - platform: mqtt
    state_topic: "/d:9e:f0:7f:cf:5c/humidity"
    unit_of_measurement: "%"
    name: "Humidité salon"
  - platform: mqtt
    state_topic: "/bd:a0:f0:7f:cf:5c/temperature"
    unit_of_measurement: "°C"
    name: "Température cuisine"
  - platform: mqtt
    state_topic: "/bd:a0:f0:7f:cf:5c/humidity"
    unit_of_measurement: "%"
    name: "Humidité cuisine"

media_player:
  - platform: squeezebox
    host: 192.168.1.104
  - platform: squeezebox
    host: 192.168.1.108

group: !include groups.yaml

And groups.yaml:

default_view:
  view: true
  entities:
    - group.salon_
    - group.cuisine_
    - group.bureau_
salon:
  view: true
  entities:
    - group.salon_
salon_:
  name: Salon
  view: false
  entities:
  - sensor.temperature_salon
  - sensor.humidite_salon
  - media_player.wolfson

cuisine:
  view: true
  entities:
    - group.cuisine_
cuisine_:
  name: Cuisine
  view: false
  entities:
    - sensor.temperature_cuisine
    - sensor.humidite_cuisine
bureau:
  view: true
  entities:
    - group.bureau_
bureau_:
  name: Bureau
  view: false
  entities:
    - sensor.temperature_bureau
    - sensor.humidite_bureau

Note that I’m not using the MQTT broker from HA. I’m running the Mosquitto MQTT broker.

Thanks!

gpbenton · November 21, 2018, 8:17am

I think you also have to remove the history component, otherwise it loads the recorder component on its own.

If that doesn’t work, try excluding all components from the recorder, which should have the same effect. It is normally the database that does all of the writing.

The other alternative is the log file, which you can reduce by just saving warning messages

logger:
  default: warning

How are you finding the XU4 apart from this? I saw a couple of videos about the HC1 last week, which were very positive, and it looked like an interesting machine to run a lower power server on.

JF002 · November 24, 2018, 7:21pm

I think you also have to remove the history component, otherwise it loads the recorder component on its own.

Indeed, removing the history component reduced the disk writes next to 0.
I’ve also configured the logger level to warning.

EDIT : there are still frequent bursts (200-300KB/s), but I still don’t know on which file HA is writing…

Ok, now, we know what was writing on the disk like crazy : the history/recorder component.

BUT the history is a feature I use, and I would like to be able to re-enable it… Do you have any idea why it was working fine (without so much disk I/O) on the previous install/version of home-assistant, and why it’s working so bad now?

How are you finding the XU4 apart from this?

The board is really fine for my use : USB3 / Gb ethernet / powerfull CPU. The only downside is the SD card, which is not so reliable (this is true for all SBC that use SD card as system storage). I’ve never tried EMMC memory.
I use the XU4 as a NAS (using OMV4 now, OMV3 before), with 2 USB3 external hard-drive (the first one is shared on the LAN, the second one is the backup, sync’ed every night). I run openmediavault for multi-room audio, MQTT broker, home-assistant, paperless, pihole, and various other personal projects.

gpbenton · November 24, 2018, 8:57pm

After over 2 years of using the sqlite database, I recently switched to using postgresql, and I wish I had done it ages ago. I suggest you try switching to that (or any other external db) as a start.

Another thing to try is to store the database on one of your other drives.

And of course, try reducing the amount of data being stored by using the include/exclude options for the history component.

Thanks for the info on the XU4. I’m currently using an old tablet (with a broken screen) as a server for some other services. If that fails, the HC1 looks like a good replacement.

JF002 · November 25, 2018, 1:05pm

After over 2 years of using the sqlite database, I recently switched to using postgresql, and I wish I had done it ages ago. I suggest you try switching to that (or any other external db) as a start.

Yes, it could be a better solution. But until now the I/O generated by home-assistant were very light, it it fit perfectly on the sd card.

Another thing to try is to store the database on one of your other drives.

I thought about it, but it would mean that the disk would be spinning all the time (they spin down after a while, for now).
I could add a small SSD, one day, for this kind of use-cases!

And of course, try reducing the amount of data being stored by using the include/exclude options for the history component.

That’s exactly what I’m trying right now. I opened the sqlite file in SqliteStudio (that’s why I love sqlite : all in one file ), and noticed that the component ‘media_player’ (used for my 2 squeezebox clients) was writing a lot of states/events. I excluded them from the recorder, and it seems to improve the situtation (not so much disk write, the DB grows slower.

I’ll keep you up to date !

If you want more info about the XU4, send me a PM, I’ll send you my mail address