bighead85
(Bighead85)
January 19, 2024, 6:29pm
1214
yes I did same for Ubuntu and working again. Thanks for posting, I was about to as thought I was going crazy
1 Like
brianuab
(Brian)
January 19, 2024, 9:33pm
1215
So my logs from - cat /var/log/apt/history.log
Upgrade: docker-compose-plugin:arm64 (2.21.0-1~debian.11~bullseye, 2.24.1-1~debian.11~bullseye), docker-ce-cli:arm64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), docker-buildx-plugin:arm64 (0.11.2-1~debian.11~bullseye, 0.12.1-1~debian.11~bullseye), docker-ce:arm64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), docker-ce-rootless-extras:arm64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye)
Installation - Home Assistant Supervised on a Raspberry Pi using Debian 11
So when I downgrade (do I need to include āarm64ā or remove it; can someone suggest how to fix it to address my scenario)
sudo apt install
docker-compose-plugin=arm64~2.21.0-1~debian.11~bullseye
docker-ce-cli=arm64 5:24.0.7-1~debian.11~bullseye
docker-buildx-plugin=arm64 0.11.2-1~debian.11~bullseye
docker-ce=arm64 5:24.0.7-1~debian.11~bullseye
docker-ce-rootless-extras=arm64 5:24.0.7-1~debian.11~bullseye
GitWally
(Git Wally)
January 20, 2024, 8:08am
1216
Hello may be I was the first to report in GItHub, but I can confirm as a temporary workaround, you must downgrade Docker packages to the previuos working version.
You can find a brief guide in GitHub post, I wrote for newbies like me!
Hope there will be a long term solution!
Bye
GitWally
(Git Wally)
January 20, 2024, 8:11am
1217
See the commands I used in github post : I think we are in the very same configuration.
It worked for me and now my HA is working 100%
Bye
1 Like
kanga_who
(Jason)
January 20, 2024, 8:24am
1218
@AlexMPH , @bighead85 @brianuab Please make sure you adhere to the ADR HERE and have Docker CE 20.10.17. If you are not running that specific version you may encounter problems with Supervised.
Also note that Ubuntu is not a supported OS and hasnāt been for about 2 years now. If you have not followed the ADR and the recommendations in this guide, please do not post for support or advice on how to fix problems.
3 Likes
AlexMPH
(Alex)
January 20, 2024, 10:38am
1219
Except that it says that:
Docker CE >= 20.10.17
So an update should in theory - clearly not in practice - break it, as it does not ask for a specific version just a minimum version
kanga_who
(Jason)
January 20, 2024, 10:44am
1220
Unless something introduced in the newest Docker version does break it, which it seems to have.
Pretty simple solution, follow the ADR and additionally lock your version from updating. No need to update Docker if your HA is working and you are adhering to the ADR.
1 Like
smoltron
(Smoltron)
January 20, 2024, 11:06am
1221
I can confirm that docker-ce update breaks also a running Debian supervised Home Assistant. MariaDB will not start any more. See MariaDb container suddenly stopped starting
AlexMPH
(Alex)
January 20, 2024, 2:03pm
1222
That is not what the ADR says though, is it?
Anyway, a fix on Docker side is on the way , which is expected to be retrofitted in version 25.0.1 (itās the version 25 that breaks things) (ref:
opened 05:28PM - 19 Jan 24 UTC
bug
### Describe the issue you are experiencing
Today MY HA went crazy, probably ā¦ after a Debian OS update: it all started after an OS reboot.
From Settings > Integrations I see "Home Assistant Supervisor Failed setup, will retry"
![immagine](https://github.com/home-assistant/supervisor/assets/19230076/d356c312-ffd4-4ce0-96d4-5e1a4581b677)
Several other integrations same behaviour, some other are working (Shelly, MQTT, ZHA, HomeKit ...)
![immagine](https://github.com/home-assistant/supervisor/assets/19230076/0217dfef-1b04-4c38-9f69-c03797762aa1)
From HA Core Logs I see (among others):
```
Logger: homeassistant.helpers.system_info
Source: helpers/system_info.py:72
First occurred: 18:12:57 (1 occurrences)
Last logged: 18:12:57
```
No Home Assistant Supervisor info available
```
Logger: homeassistant.components.hassio
Source: components/hassio/__init__.py:557
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 18:02:11 (4 occurrences)
Last logged: 18:17:21
```
Can't read Supervisor data:
If I try to read HA Supervisor Logs: Failed to get supervisor logs, 502: Bad Gateway
What could be happend?
In Debian I found this updates to Docker Packages, today (with command `cat /var/log/apt/history.log`):
```
Start-Date: 2024-01-19 17:14:36
Commandline: packagekit role='update-packages'
Upgrade: docker-compose-plugin:amd64 (2.21.0-1~debian.11~bullseye, 2.24.1-1~debian.11~bullseye), docker-ce-cli:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), docker-buildx-plugin:amd64 (0.11.2-1~debian.11~bullseye, 0.12.1-1~debian.11~bullseye), docker-ce:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), docker-ce-rootless-extras:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye)
End-Date: 2024-01-19 17:14:49
```
It all began after a reboot I did one hour after Debian OS Updates.
Tried to reboot Debian and HA several times: no changes
Tried to go for a backup: not visible when I press the button
Thanks
### What type of installation are you running?
Home Assistant Supervised
### Which operating system are you running on?
Debian
### Steps to reproduce the issue
1.
2.
3.
...
### Anything in the Supervisor logs that might be useful for us?
```txt
not available
```
### System Health information
not available
### Supervisor diagnostics
cannot find download diagnostics
### Additional information
_No response_
)
1 Like
AlexMPH
(Alex)
January 24, 2024, 11:15am
1223
Docker ce 25.0.1 was just released with a fix for this, and multiple users (including me) have indicated it now works.
1 Like
Tamsy
(Tamsy)
January 24, 2024, 1:39pm
1224
Better be wise and wait before upgrading Docker. There is absolutely no reason to rush but let it settle and mature before hopping to 25.0.x.
While for some Docker CE 25.0.1 seem to work with HA Supervisor other users are still having issues with some addons like i.e InfluxDB and ebusd. More about the latter HERE .
thatās not what the ADR actually says. The ADR only specifies a MINIMUM docker version. v25.0.1 seems to have fixed all issues I was seeing
1 Like
kanga_who
(Jason)
January 24, 2024, 11:54pm
1226
I understand that, I suggested in my next postā¦
If you have a working system and have the ADR minimum version of all the dependencies there is no need to update them and risk breaking your install.
Docker updates breaking HA installs has happened a few times now, I donāt see the point in updating it unless itās to meet a change in the ADR or patch a security flaw.
petro
(Petro)
January 25, 2024, 12:09am
1227
I only update when HA forces new dependencies. Iāve only update handful of times since 2018. No reason to stay current with Debian when itās meant to be LTS.
3 Likes
nickrout
(Nick Rout)
January 25, 2024, 12:11am
1228
What are the potential security flaws in the 20 odd upgrades to docker-ce between 20.10.17 and the present? No, I donāt know either, but frankly Iād rather keep my system up to date.
Then again maybe I should just pin my desktop to windows XP?
2 Likes
nickrout
(Nick Rout)
January 25, 2024, 12:12am
1229
LTS means they provide updates Long Term, not that you shouldnāt use them!
1 Like
Tamsy
(Tamsy)
January 25, 2024, 12:23am
1230
But neither you should use them. I have not read something about a security flaw or therelike with Docker CE 24.x.x which would make it wise to upgrade to 25.0.x.
But here again it seems to be a matter of the point of view: While some prefer to stay on the ābleeding edgeā when it comes to upgrades (even these are not mandatory security updates), others prefer to stay on the more conservative side and wait a while to let the latest releases āmatureā before installing them.
To each his own.
1 Like
nickrout
(Nick Rout)
January 25, 2024, 9:19am
1231
Yeah take your point. Then again many make the legit comment that there are never enough testers.
The release of docker-ce 25.0.x appears to have introduced multiple issues for Supervised HA.
docker-ce 25.0.0 introduced a change in networking behavior the prevented Supervisor from starting. That issue was resolved with the release of docker-ce 25.0.1.
Once HA Supervised was operable on docker-ce 25.0.1 it became apparent there is an issue with the HA InfluxDB integration that may be caused by a change in docker file system mounting behavior introduced in moby v25.
In line with the cautionary approach advised by @Tamsy , it may be advisable to not use docker-ce 25.0.x until more is understood about the impact of the docker file system mounting behavior change.
See
Update, January 25th 2024 - Staying on Docker 24 is advised
The release of Docker 25.0.1 has resolved the networking issue that prevented Supervisor from starting, but there appears to be another Docker related issue that impacts the InfluxDB integration.
Docker 25.0.x has introduced a change in file system mounting behavior that appears to be impacting python multiprocessing. See these issues core/issues/108810 , moby/issues/47158 . The HA InfluxDB integration uses multiprocessing.
Itās possā¦
opened 09:16PM - 24 Jan 24 UTC
integration: influxdb
### The problem
Following an upgrade from docker-ce 24.0.7 to 25.0.1 the [Influā¦ xDB](https://www.home-assistant.io/integrations/influxdb/) integration fails with the error message,
`ERROR (influxdb) [homeassistant.components.influxdb] Cannot connect to InfluxDB due to '[Errno 30] Read-only file system'. Please check that the provided connection details (host, port, etc.) are correct and that your InfluxDB server is running and accessible.`
No data is received in InfluxDB from HA after the upgrade from docker-ce 24.0.7 to 25.0.1.
The '[Errno 30] Read-only file system' error message appears to be coming from a influx.write_api call in [this](https://github.com/home-assistant/core/blob/909e58066d202df9f599dcedeabbac59cf9336af/homeassistant/components/influxdb/__init__.py#L365) code block of the [InfluxDB](https://www.home-assistant.io/integrations/influxdb/) integration.
The error may be caused by a change in mounting behavior introduced in moby v25.0.x that prevents multiprocessing from creating a /dev/shm file, as reported in this [moby/issues/47158](https://github.com/moby/moby/issues/47158) issue. The influxdb python client [uses](https://github.com/influxdata/influxdb-client-python/blob/d7181fab9612dd0558822fc40e5d9ccf5a15d56b/influxdb_client/configuration.py#L17) multiprocessing.
If the issue is python multiprocessing and docker-ce 25.0.x related, other HA integrations that use multiprocessing may be impacted
Downgrading from docker-ce 25.0.1 to 24.0.7 restores the HA InfluxDB integration functionality, suggesting it is an issue brought about by a change introduced in docker-ce 25.0.x.
### What version of Home Assistant Core has the issue?
core-2024.1.5
### What was the last working version of Home Assistant Core?
_No response_
### What type of installation are you running?
Home Assistant Supervised
### Integration causing the issue
InfluxDB
### Link to integration documentation on our website
https://www.home-assistant.io/integrations/influxdb/
### Diagnostics information
_No response_
### Example YAML snippet
```yaml
# influxdb section from configuration.yaml
influxdb:
api_version: 2
host: 192.168.1.2
port: 8086
ssl: false
token: REDACTED
organization: REDACTED
bucket: home_assistant
override_measurement: sensor
default_measurement: state
```
### Anything in the logs that might be useful for us?
```txt
grep -i influxdb /usr/share/hassio/homeassistant/home-assistant.log.*
/usr/share/hassio/homeassistant/home-assistant.log.1:2024-01-24 17:46:44.641 ERROR (influxdb) [homeassistant.components.influxdb] Cannot connect to InfluxDB due to '[Errno 30] Read-only file system'. Please check that the provided connection details (host, port, etc.) are correct and that your InfluxDB server is running and accessible.
```
### Additional information
The commands used to downgrade and upgrade docker-ce on debian 12 bookworm.
#### Install docker-ce v24.0.7
```
sudo apt install \
docker-ce=5:24.0.7-1~debian.12~bookworm \
docker-ce-cli=5:24.0.7-1~debian.12~bookworm \
docker-buildx-plugin=0.11.2-1~debian.12~bookworm \
docker-compose-plugin=2.21.0-1~debian.12~bookworm \
docker-ce-rootless-extras=5:24.0.7-1~debian.12~bookworm
```
#### Install docker-ce v25.0.1
```
sudo apt install \
docker-ce=5:25.0.1-1~debian.12~bookworm \
docker-ce-cli=5:25.0.1-1~debian.12~bookworm \
docker-buildx-plugin=0.12.1-1~debian.12~bookworm \
docker-compose-plugin=2.24.2-1~debian.12~bookworm \
docker-ce-rootless-extras=5:25.0.1-1~debian.12~bookworm
```
#### InfluxDB host
The InfluxDB instance used during testing is on a separate server to the one that hosts HA.
No changes were made to the InfluxDB host environment.
The InfluxDB version is v2.7.5
opened 11:58AM - 22 Jan 24 UTC
kind/question
status/0-triage
version/25.0
### Description
Since Docker upgrade to v25 the sub-filesystems in /dev (in pā¦ articular /dev/shm) get mounted read-only if the Docker container has a volume specified that mounts /dev as read-only, which is a change from behaviour in v24.
Upgrading from docker-ce version `5:24.0.7-1~ubuntu.22.04~jammy` to `5:25.0.0-1~ubuntu.22.04~jammy` (and containerd.io version `1.6.15-1` to `1.6.27-1`) causes the change in permissions of /dev/shm that breaks a lot of software, in out case Python multiprocessing module fails to create a new instance of https://docs.python.org/3.10/library/multiprocessing.html#multiprocessing.Value as it is unable to create a file in /dev/shm
### Reproduce
1. docker run --rm -d -v /dev:/dev:ro --name test123 ubuntu:jammy sleep 3600
2. docker exec -it test123 mount | grep /dev/shm
3. `tmpfs on /dev/shm type tmpfs (ro,nosuid,nodev,inode64)`
Other file systems are also affected:
```
devpts on /dev/pts type devpts (ro,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (ro,nosuid,nodev,inode64)
mqueue on /dev/mqueue type mqueue (ro,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (ro,relatime,pagesize=2M)
```
### Expected behavior
docker exec -it test123 mount | grep /dev/shm
`tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)`
```
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
```
### docker version
```bash
Client: Docker Engine - Community
Version: 25.0.0
API version: 1.44
Go version: go1.21.6
Git commit: e758fe5
Built: Thu Jan 18 17:09:49 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 25.0.0
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: 615dfdf
Built: Thu Jan 18 17:09:49 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.27
GitCommit: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc:
Version: 1.1.11
GitCommit: v1.1.11-0-g4bccb38
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```
### docker info
```bash
Client: Docker Engine - Community
Version: 25.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Containers: 23
Running: 22
Paused: 0
Stopped: 1
Images: 45
Server Version: 25.0.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-79-generic
Operating System: Ubuntu 22.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 7.532GiB
Name: xxx
ID: WJDD:VNV2:5UUD:M5GY:AVR6:4GFI:WFDT:V66E:OPBY:CGB5:DQ3A:DTPM
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
```
### Additional Info
_No response_
kanga_who
(Jason)
January 25, 2024, 11:45am
1233
Itās already been discussed in this thread, read above.
1 Like