4.2GB in /var/lib/docker/overlay2 !?

Wow! 8GB SD card in my Pi3B running HA core in a Docker, on to of Raspbian (stretch), has used up half the disk. What gives folks?

root@pi:/home# du -h /var/lib/docker/overlay2 --max-depth=1
2.2M    /var/lib/docker/overlay2/f09780df2adeb46df9293862d62a0ad42c63140aac17d6aa9abe0796a200db25
65M     /var/lib/docker/overlay2/52e85dc5b91aa1eef60023052afa4d44874b7755ed2df1e994e77ec73c54b567
12M     /var/lib/docker/overlay2/b13e699b3c1f6cb69c8d8d19f763f7c67b32b5c9b1d463055768dc560d890cab
103M    /var/lib/docker/overlay2/cb0cfda5aaa6bb788790001967acd81b05e0a371dbfe87309f48f1a9766fe039
15M     /var/lib/docker/overlay2/923f325d12ff5751ee9eb40ddaa20cdc97b3ec6771adab4736ccec0825ed18e8
4.2M    /var/lib/docker/overlay2/936e63bbe8e9f8fe85ebf9dcc2870278a1f05a97d1b5a57e136a375d69c235d6
264K    /var/lib/docker/overlay2/bde17bce84fb407148806ce6efcbf87ce49464384aa8cfebfd5ac6c431cdc6fd
5.2M    /var/lib/docker/overlay2/6ea02347953daa3d20c77c8c956049f184fcaf6541acc9b9fc7485c7b98c2048
3.9M    /var/lib/docker/overlay2/79f4717477770dd45e1a176e84775b3f4dade07f25d3b97774a10503c8ca7692
952K    /var/lib/docker/overlay2/bbf778538f54add8eae356d71086d77dcf40bada16031bcf49af114453a02015
76K     /var/lib/docker/overlay2/a98aa1db4ae940d9eb06c5e38e634295f6dea3b965f8a85c765f3ec00826af9c
4.2M    /var/lib/docker/overlay2/be597d3d7ec26dbeb6b9bc5364319daf040ba7afc312a89a170703f7bd30d3e3
66M     /var/lib/docker/overlay2/972405f4e2708e380dea018844e0c4a0bdd85b96647bd6daca1ea8ab652dfc00
1.2G    /var/lib/docker/overlay2/82402775347f74479b5aa04ad047bcd2b2c16c4348934a4a79a46540ccf2b145
44K     /var/lib/docker/overlay2/aa1291a4e20dd79b5ac7b1ba88c4f69db731408307f272bca4a751f26bb1f681
204K    /var/lib/docker/overlay2/l
40K     /var/lib/docker/overlay2/c604f81e2502c30dccd3f14cc8a5bf276a0b0480ba44e6063d4185520045f13f-init
32K     /var/lib/docker/overlay2/f7b194bb9c52470680f288676454254984a5c5653b166dc9107b73e705a07233
4.0M    /var/lib/docker/overlay2/c469f8ce851a98c1327d04330906c320d7d023b99cb15494641a1596a0676128
28K     /var/lib/docker/overlay2/5e0e3bb0119b161b7be5d68983dbc63891a5009b58dad6a21e03821e2c8224ab
32K     /var/lib/docker/overlay2/7ce925279eee27798c3f08b42ff70c0b2926f52ef7c48346d4e2bcc9e12de37d
9.0M    /var/lib/docker/overlay2/c604f81e2502c30dccd3f14cc8a5bf276a0b0480ba44e6063d4185520045f13f
40K     /var/lib/docker/overlay2/eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c-init
1.5M    /var/lib/docker/overlay2/9db9d1bc43ebd40b73e204d1d848ca8405356fbeba67f2f46c0a22350d12801d
32K     /var/lib/docker/overlay2/361b3ade22b547369db0c132b14d590e2ae5444408774e307f32c8883a010cef
24K     /var/lib/docker/overlay2/f15f0a1fa53c60f1bd16a2f1be8599b6061439df1e4a4285c71d7f68f70fa4f6
96K     /var/lib/docker/overlay2/eb2c3a511edbfd49ac6557fc6ed5f3555220006eccba19c379861b2e09351ec2
100K    /var/lib/docker/overlay2/163bbf4f0adbf1ec765a9cd39f85a87026e98f39ca714d0bce0388df3bfdb777
32K     /var/lib/docker/overlay2/d826798290d818ee49b6b21053bea478723c7595de322c26c2d3da93e57ddfe2
851M    /var/lib/docker/overlay2/106c8d218ce0e759fd5b8d2c800266769c74d410ace04cad7da9c45be194568c
160K    /var/lib/docker/overlay2/3bdac59314f90dc8a1b981f09bfb1033a50c03b5b8324353f205215e43ce02e8
15M     /var/lib/docker/overlay2/38e1250f65d8a0fa5f42591094b85ceeafc7cf95cc8a6f63fcac92ee73721afa
58M     /var/lib/docker/overlay2/2dfaa7dcbf489b54b838f07e8a7dd9e60cb1bcf6305be9d5f7bf4c972f15c774
890M    /var/lib/docker/overlay2/eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c
16M     /var/lib/docker/overlay2/ea118c230eef4378c5c2de358bcdcd24f02397fc0ab9f4db5704d67127a6fd58
32K     /var/lib/docker/overlay2/3e900270978cbe0486b0b5b97286d04d0ef0b786bcb267aee85ae639825158e4
80K     /var/lib/docker/overlay2/9680541c75eb9aad2f9cdbcb084b89b3d1c9e275961149224f46a12150ed09c9
105M    /var/lib/docker/overlay2/6c13e97672a5c4cc638c55186179e936889a6017c1398c49c78a8dae72abceae
40K     /var/lib/docker/overlay2/82402775347f74479b5aa04ad047bcd2b2c16c4348934a4a79a46540ccf2b145-init
1.5M    /var/lib/docker/overlay2/362e47b38075a8bb66779360cb2791d1a1bf5aa7d3286c3fce6c80d81176be18
28M     /var/lib/docker/overlay2/d804444e59dbf9bc93537bb8af6e866f13efec56688655559576a7c7898d3cca
130M    /var/lib/docker/overlay2/39f1f50ae2696ac8f72a163256ae15f20d3260cd5f5e19860c36400b25b6f9f7
156K    /var/lib/docker/overlay2/ffb48dd8b197d12f482c9a2a967b02ebc7047c4b67b69af5d4c92b136ea0ff97
504K    /var/lib/docker/overlay2/1472f29eebc0b6e62c20509feb4ec9c16d050dbfc60e73263c3ac2743fca8d14
400M    /var/lib/docker/overlay2/812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6
1.4M    /var/lib/docker/overlay2/d6a813c4c04e049d2b6498fff6c121a1e36d30289421dbd595113127a6c3343f
24K     /var/lib/docker/overlay2/1abbd66e1e86db60c9cfb8a69c23b40fb7180a39e159e0737b9cc8812834b34a
32K     /var/lib/docker/overlay2/d29e4d82225455c31f95711960586a48e8c0c7ed6e111cd84c71091713b5cf92
40K     /var/lib/docker/overlay2/8169bb69e7145d04e291662813f240991cdb32f1dfdb69d2d2da0e12bfcc8b77
219M    /var/lib/docker/overlay2/9f119fbca72541dee2b6dca2cf3717e35dce101b3423a9a6ef4780132bfdbb93
19M     /var/lib/docker/overlay2/868debe8192188ffb137da82ccc5a7f9215453fcf9b7017168191449aaed496c
4.2G    /var/lib/docker/overlay2

I can wipe this folder - yup drastic but done - and running the HA container recreates the overlays. Defo just HA doing this. Image sizes alone…

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              3                   3                   1.899GB             0B (0%)
Containers          3                   3                   12.19kB             0B (0%)
Local Volumes       4                   2                   0B                  0B
Build Cache         0                   0                   0B                  0B

I’m considering moving to HA Supervised which shouldn’t be too difficult but will it likely use the same space? I’m not really doing that much with HA:-

Integrations:
MQTT
Z-Wave
Google Cast
IPP
Mobile App

Docker images

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
homeassistant/raspberrypi3-homeassistant   stable              56950eaa6e90        5 days ago          1.02GB
eclipse-mosquitto                          latest              0571b72448f9        2 weeks ago         8.07MB

The rest is a bare Raspbian headless base OS

mmm?

Move to HA Supervised?

Are you using portainer by any chance?

Nope, it’s not my image / container that’s the issue so portainer aint going to be able to do much. The image is built from a pull and the container is what it is.

EDIT:
Granted 4.3GB is no big deal on SSD/HD PC / servers, but this is a Pi with 8GB card. Why does HA need so much space on this platform? When I ran it in a python venv it used like around 1GB all in.

Do you have any old stopped containers maybe?

docker ps -a should show you

I know that doesn’t really answer your question, but you should avoid running such small SD cards on the Pi. Most SD cards have a very simplified form of wear leveling that mostly works with a lot of free space. So it makes sense to use a 32 or even 64GB card, even if you only use 4GB of it. It gives the controller a lot of headroom to play with for wear leveling. Using a small card will kill it a lot faster.

End of offtopic :slightly_smiling_face:

No . I’ve done all the usual prune and housekeeping.

I found the issue:

Overlay becomes large because the data written to disk by a process inside a container writes to the overlay unless you mount a volume to the location in the container where you are writing.Could be the way HA images are built/designed? This isnt a docker issue perse, The diff files in the overlays are small except for one.

If one used the command:

ls | xargs -I {} du -shx {}

within /var/lib/docker/overlay2/ the output is familiar. Identify the large disk allocations and go into that directory. Repeat the command, and so on, and eventually we get to:

/var/lib/docker/overlay2/106c8d218ce0e759fd5b8d2c800266769c74d410ace04cad7da9c45be194568c/diff/usr/local/lib#

where now the command produces:

817M    python3.8

Here’s the full list of Python space used on my box for HA container (well granted not 100% accurate given how overlays work but it’s about 80% true here):-

root@pi:/var/lib/docker/overlay2# find . -name python3.? -exec du -shx {} \;
616K    ./f09780df2adeb46df9293862d62a0ad42c63140aac17d6aa9abe0796a200db25/diff/usr/local/lib/python3.8
0       ./cb0cfda5aaa6bb788790001967acd81b05e0a371dbfe87309f48f1a9766fe039/diff/usr/local/share/man/man1/python3.1
98M     ./cb0cfda5aaa6bb788790001967acd81b05e0a371dbfe87309f48f1a9766fe039/diff/usr/local/lib/python3.8
1012K   ./cb0cfda5aaa6bb788790001967acd81b05e0a371dbfe87309f48f1a9766fe039/diff/usr/local/include/python3.8
8.0K    ./cb0cfda5aaa6bb788790001967acd81b05e0a371dbfe87309f48f1a9766fe039/diff/usr/local/bin/python3.8
15M     ./923f325d12ff5751ee9eb40ddaa20cdc97b3ec6771adab4736ccec0825ed18e8/diff/usr/local/lib/python3.8
3.9M    ./79f4717477770dd45e1a176e84775b3f4dade07f25d3b97774a10503c8ca7692/diff/usr/local/lib/python3.8
0       ./82402775347f74479b5aa04ad047bcd2b2c16c4348934a4a79a46540ccf2b145/merged/usr/local/share/man/man1/python3.1
935M    ./82402775347f74479b5aa04ad047bcd2b2c16c4348934a4a79a46540ccf2b145/merged/usr/local/lib/python3.8
8.0K    ./82402775347f74479b5aa04ad047bcd2b2c16c4348934a4a79a46540ccf2b145/merged/usr/local/bin/python3.8
1012K   ./82402775347f74479b5aa04ad047bcd2b2c16c4348934a4a79a46540ccf2b145/merged/usr/local/include/python3.8
817M    ./106c8d218ce0e759fd5b8d2c800266769c74d410ace04cad7da9c45be194568c/diff/usr/local/lib/python3.8
8.0K    ./eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c/merged/etc/python3.7
360K    ./eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c/merged/usr/share/doc/python3.7
4.0K    ./eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c/merged/usr/share/lintian/overrides/python3.7
4.0K    ./eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c/merged/usr/share/binfmts/python3.7
22M     ./eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c/merged/usr/lib/python3.7
3.4M    ./eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c/merged/usr/bin/python3.7
8.0K    ./eaae03f2e13bcaf5a0980791f3a0e8110987616274a5ecae0136cc2ea548b17c/merged/usr/local/lib/python3.7
1.8M    ./d804444e59dbf9bc93537bb8af6e866f13efec56688655559576a7c7898d3cca/diff/usr/local/lib/python3.8
128K    ./1472f29eebc0b6e62c20509feb4ec9c16d050dbfc60e73263c3ac2743fca8d14/diff/usr/local/lib/python3.8
8.0K    ./812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6/diff/etc/python3.7
360K    ./812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6/diff/usr/share/doc/python3.7
4.0K    ./812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6/diff/usr/share/lintian/overrides/python3.7
4.0K    ./812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6/diff/usr/share/binfmts/python3.7
22M     ./812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6/diff/usr/lib/python3.7
3.4M    ./812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6/diff/usr/bin/python3.7
8.0K    ./812b7ba9268949a8ce142a2845826d6dee2a16c05bea941ecd8f542c01249ac6/diff/usr/local/lib/python3.7
root@pi:/var/lib/docker/overlay2#

So the problem is the size of python install which includes a mind boggling array of modules - hundreds - to support HA I guess , modules most will not use directly. So I think for me this is a design ‘feature’ where HA has just thrown the kitchen sink into this, and not installed modules based on integrations/config being used. There is a list of around 300 python modules in the in the 935MB directory.

In my local host system, the python lib is just 89MB.

89M     /usr/lib/python3.7

Well, that’s that ! That is the reason. HA installs every module it can - apparently - whether you use it or not.

1 Like

Nothing to do with it. See my answer above. Nothing wrong with 8GB in a Pi. MORE than sufficient - should be. I could go down a rabbit hole with you on SD card IO interfaces and drivers. Dont go there :wink:

Okay. It’s your data. Shrug.

Another thing to watch out for is log files. By default the containers stdout/stderr write to disk and they never truncate or wrap around. This can fill up small disks quickly.

@HeyImAlex
I found a 64GB Class1 lying around , using that now, problem solved and much faster response time from the OS.
Well , I was being stubborn :slight_smile:

reviving the topic as I am in the same situation
I have space left but it seems to me really huge, compared to the installation size, the space allocated.
I tried to see if I can change the parameters of add ons in terms of log retention but it seems it is not possible for journals driver that is mostly used

tried to delete the whole content of /var/log/journal/ as this being logs it is safe to delete them (read everywhere in internet): I get a fs is read only error.

Anybody as some clues?

I solved… some 11 months after.
Here the solution :https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux