Installing HA Core on RPi - python 3.11 question

Installation
1

Hello all

I am running HA Core on a RPi 3B+ but the version is about a year old. I am of the opinion if it ain’t broke, don’t fix it. Just recently there was such a break; Life360 changed its servers and the API stopped working, and I thought maybe it’s time to look at bringing HA up to date. Rather than upgrading and risking my system/integrations breaking, I decided to build a new one from scratch and then cut over when I’m happy everything works as expected. I work in an industry which is risk-averse so this seems like the sensible way to go about it.

I can see from the release notes that Python 3.9 (my current release) is no longer supported, and 3.10 won’t be supported for long, so I’d like to go straight to 3.11 and try to future-proof it a bit. The thing is, the latest RPi OS still comes with 3.9 so I have to install 3.11 myself. I have done this a couple of times on a fresh OS install but each time I come to install the venv for HA Core it fails, seemingly looking for a dependency but I am not sure how to find out which dependency it is missing. The error is :

Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/local/lib/python3.11'
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/usr/local/lib/python3.11',
    '/usr/local/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: no codec search functions registered: can't find encoding

Current thread 0x76f1e040 (most recent call first):
  <no Python frame>

Sorry if it’s staring me in the face, but I don’t see it!

Thanks

2

From experience you need to ensure you have a bunch of dependencies installed:

sudo apt install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline-dev \
  libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev \
  zlib1g-dev libudev-dev unixodbc-dev libpq-dev libwebp-dev libopenjp2-7-dev libjpeg-dev \
  libtiff5-dev libfreetype6-dev libc6-dev libffi-dev libbluetooth-dev libtirpc-dev libjemalloc-dev cargo
sudo ldconfig

If you use MariaDB then install libmariadb-dev, and PostgreSQL needs libpq-dev.

Then install pyenv and use to install Python 3.11 and build your venv.

3

Thank you very much for the response.

I tried following it but right at the end, pyenv is not giving me the confirmation it has installed Python but rather it is rebooting with no error message. Running ‘python --version’ still shows the 3.9.x release. The same seems to happen when I install (altinstall) Python 3.11.x manually, which perhaps is not what it should do but I’ve never installed Python before. Are there any logs I could look at to see why it’s failing? This is a vanilla Raspbian Bullseye install and then immediately installing the dependencies and Python so I can’t understand why it would just fail…

4

pyenv installs a new Python alongside the existing one (you really should read the pyenv docs).

pyenv install 3.11.4
pyenv virtualenv 3.11.4 venv-3.11.4
pyenv activate venv-3.11.4
pip install --upgrade homeassistant

Then your systemd unit file will look something like:

[Unit]
Description=Home Assistant
After=network-online.target

[Service]
Type=simple
User=homeassistant
ExecStart=/home/homeassistant/.pyenv/versions/venv-3.11.4/bin/hass -c "/home/homeassistant/.homeassistant" --log-rotate-days 1

[Install]
WantedBy=multi-user.target
5

Thanks for helping me. I did read some of the docs, but Python is new to me so it’s a lot of information to take in.

I don’t believe it installed correctly because after the reboot, i can run Python3 but Python3.11 gives a ‘command not found’ error. Running ‘pyenv virtualenv 3.11.4 venv-3.11.4’ also gave an error “pyenv : unknown command virtualenv”.

I then ran through the manual altinstall process again and was able to run Python3.11 but now get the same error as before, “Could not find platform dependent libraries <exec_prefix>”.

I’ve tried other versions of Python 3.11 so I guess I’ll try some other releases of Raspbian OS, perhaps there’s a problem there.

6

Is there a reason then you’re not just using Docker?

The only reason somebody should use a venv is because they’re at home with them already.

7

Yes, I use HA to call pilight bash scripts which I could never figure out in HA OS or Docker because I couldn’t get them to talk to the host machine in order to call the scripts. I’d be happy to try building either if I could get raw RF to work

8

What’s the full error then, with context - use a code share site if it’s long.

9

Thank you, the exact steps I have followed to get to this point are as follows :

Installed Raspbian Bullseye on an 8Gb card, flashed using Balena Etcher and Raspberry Pi Imager.
Set a static IP
$ sudo apt-get update
$ sudo apt-get -y install build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev
$ sudo apt install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libudev-dev unixodbc-dev libpq-dev libwebp-dev libopenjp2-7-dev libjpeg-dev libtiff5-dev libfreetype6-dev libc6-dev libffi-dev libbluetooth-dev libtirpc-dev libjemalloc-dev cargo
$ wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz
$ tar -zxvf Python-3.11.4.tgz
$ cd Python-3.11.4
$ ./configure --enable-optimizations
$ sudo make altinstall
$ sudo apt-get install -y python3 python3-dev python3-venv python3-pip bluez libffi-dev libssl-dev libjpeg-dev zlib1g-dev autoconf build-essential libopenjp2-7 libtiff5 libturbojpeg0-dev tzdata ffmpeg liblapack3 liblapack-dev libatlas-base-dev
$ sudo useradd -rm homeassistant -G dialout,gpio,i2c
$ sudo mkdir /srv/homeassistant
$ sudo chown homeassistant:homeassistant /srv/homeassistant
$ sudo -u homeassistant -H -s
$ cd /srv/homeassistant
$ python3.11 -m venv .

And the error from that last command is :

Could not find platform dependent libraries <exec_prefix>
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/usr/local/lib/python3.11/venv/__init__.py", line 10, in <module>
    import subprocess
  File "/usr/local/lib/python3.11/subprocess.py", line 104, in <module>
    from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'

I am aware some of the apt-get entries have duplicates, I’ve taken them from 3 different sources (the Python upgrade process here, the HA installation guide and your kind input too.

10

I wouldn’t install python packages with apt-get after “make altinstall”, this might provide something which was built for an older version of python?
if one of the packages is required, I would install them prior to make altinstall

Armin

11

That’s the entire error?

12

Ok, I mostly followed that on a spare Pi3 running Bullseye. I used pyenv instead of manually installing Python, but otherwise the same.

pyenv install 3.11.4
pyenv virtualenv 3.11.4 venv-3.11.4

No errors at that point :man_shrugging:

13

Thank you for going to the trouble of trying it for me. Seems I need to try changing the OS / card / Pi to see what is causing the issue

1 Like
14

Well it turns out the Pi3 I was using seemed to have an issue when compiling Python. The restart seemed strange to me but having not compiled Python before, I couldn’t be sure it wasn’t supposed to restart. Running “Python3.11 --version” told me 3.11.4 was installed but it really wasn’t. Switching to another Pi3 I had lying around and the compile finishes without restarting, and I am able to get beyond the error message I was seeing and create the virtual environment.

The next issue I seem to be encountering is when trying to install wheel when inside the venv :

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

I have tried all manner of options, installing OpenSSL (apparently 1.1.1 is already present) and compiling Python with option

./configure --enable-optimizations --with-openssl=/usr

but the issue persists.

Thinking that perhaps HA Core is now a bit beyond my abilities, I started looking at getting HA Operating System working and running pilight on a dedicated second Pi to mitigate my reasons for not simply using HAOS. I can’t even get that to work, although my pilight web service is working fine the fresh HAOS install refuses to see it :

Error during setup of component pilight
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 288, in _async_setup_component
    result = await task
             ^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/pilight/__init__.py", line 77, in setup
    pilight_client = pilight.Client(host=host, port=port)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pilight/pilight.py", line 93, in __init__
    answer_1 = json.loads(self.receive_socket.recv(1024).decode())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At this rate I will probably just run a 2022 release of HA Core forever.

15

Well to thankfully draw this to a close, I persisted with the Python 3.11 install with OpenSSL for as long as I could stand it before admitting defeat and deciding I need to learn more about how it works. I returned to the more desirable solution which I failed to achieve a couple of years back, which was to build HA Operating System and get it to run Pilight as a service. Initially I was able to get HAOS to talk to a separate dedicated Pi running Pilight which was good enough, but after fighting a bit more I finally managed to get Pilight to run on the same Pi as HAOS and I’m now happy to move away from HA Core. Thank you for the help.

16

Since you’re running HA inside a Python VENV, your startup must setup the necessary environment for the VENV.
Install instructions have us do a lot of steps within the VENV environment, including installing homeassistant itself. That all happens inside the VENV space, but your systemd unit file isn’t setting up that environment before launching python/hass.

i.e. in the systemd unit file, you have to do the same thing as you do in a shell when you (while in the VENV directory) type ‘source bin/activate’ to activate the VENV environment variables you need.

Unless your systemctl unit file causes this setup step to run, your python and HA get executed in the system’s default environment, and not in the VENV it’s supposed to be using.

To remedy that, in my case I just have the systemd unit file (e.g. homeassistant.service) exec a bash shell script. That script does the ‘source bin/activate’ before then launching ‘python3 hass’