Upgrading HomeAssistant Core in a Python VE

Hi,

Installing or upgrading HASS to the latest & greatest in a Python Virtual Environment, may be tricky. So I thought I’d create a post after I’ve struggled through it enough times now.

Hopefully this helps other members.

In summary there is a couple of steps involved, but from a High-Level, this is what’s required:

  1. Ensure you have a supported Python version for the latest build. (Currently for HASS 2023.10, the earliest supported version of Python is 3.10 - This guide has been updated for Python 3.10.9)
  2. If required, Upgrade Python on your platform (this guide is for Debian)
  3. Freeze your current Python dependencies / packages for HASS & Integrations
  4. Create a backup of your old HASS environment
  5. Activate a new Python VE on the later version of Python
  6. Rebuild HASS to run in the new Python virtual environment
  7. Upgrade HASS after its running from the new Python environment
  8. Check for other breaking changes (For example the SQL Lite version).

So, in summary here is what I did to upgrade my HASS core deployment on Ubuntu 18.04 to the latest:

A) Log into the shell.
B) Check / Upgrade your current python version:

python3 --version

If you have a Python version that is supported (3.10.9 at the time of writing this) or above then skip to step G) otherwise continue to upgrade Python as follow:

Download the required Python Version (this is for version 3.10.9):

cd /
sudo wget https://www.python.org/ftp/python/3.10.9/Python-3.10.9.tgz

Unpack it:

sudo tar -xzf Python-3.10.9.tgz

Make & Install it:

cd Python-3.10.9
./configure
make -j 4
sudo make install

C) Freeze current Python packages

This will grab the list of current Packages used in your HASS environment, and save them to a “requirements.txt” file which you will use later on to install all the required packages in your new Python VE):

cd /home/homeassistant/.homeassistant
sudo -u homeassistant -H -s
source /srv/homeassistant/bin/activate
pip3 freeze –local > requirements.txt
exit

D) Backup your existing HASS folder

If you’ve previously done this, you will already have a folder called srv/homeassistantold. Either remove the old homeassistantold directory with the command “sudo rm -r homeassistantold” or when you run the move (mv) command below, specify another directory (e.g. homeassistantold-versionx)

To create a backup, we simply create a copy of the “/srv/homeassistant” folder, if you want to restore to a previous “backup”, you would copy the backup (e.g. /srv/homeassistantold) back to its original folder (/srv/homeassistant) using the cp command.

To create the backup do the following:

Stop your HASS instance:

sudo systemctl stop [email protected]

Go to the SRV folder, and copy the homeassistant program directory to your backup folder:

cd /srv
sudo mv homeassistant homeassistantold

Create a new homeassistant folder, which will run on the new version of Python:

sudo mkdir homeassistant

Set ownership of the folder for the homeassistant account:

sudo chown -R homeassistant:homeassistant homeassistant
sudo -u homeassistant -H -s

E) Install Python & component dependencies for the new Python Version.

During this step - depending on your previous install and 3’rd party integrations - some dependencies might fail due to compatibility of integrations / components with the Python 3.10 environment (which you will note as errors that pop up when you run the “pip3 install -r requirements.txt” command).

If this is the case, then either remove the dependency from your requirements.txt file (sudo nano requirements.txt) in which case the components that depends on the removed item will likely fail, or you could resolve each of the failures by “googling” the error, and applying a workaround by changing versions etc in the requirements.txt file, until you’re able to run the last command (pip3 install -r requirements.txt) without any errors.

An Example of this would be an integration that I had which requires “backports.zoneinfo” version 0.2.1 that is no longer supported on Python 3.10. According to this thread: python - ERROR: Could not build wheels for backports.zoneinfo, which is required to install pyproject.toml-based projects - Stack Overflow , I had to replace the “backports.zoneinfo==0.2.1” in the requirements.txt file with backports.zoneinfo;python_version<“3.9” as per the workaround described in the article.

Another example where editing the requirements.txt file was required for me on Python 3.10 was for the ruamel.yaml version 0.15.100 dependency, which needed to be changed to 0.16.11, as per this article: ruamel dependency broken with setuptools>=50 · Issue #39987 · home-assistant/core · GitHub

This can be frustrating, but just work through the errors one by one, until you get an error free pip install of the requirements.txt file.

You might also have to wait for some components to be updated by the developer after a new Python version, so you might need to exclude a component, until its been updated by the developer to work on your Python version, and circle back to this component/integration when you want to re-enable it after a new version has been released.

For the time being, do not attempt to go with Python 3.11, there are too many components that will fail, just stick with what is required e.g. 3.10

To start off, we’ll first add Python to the newly created “srv/homeassistant” directory:

cd /srv/homeassistant
sudo python3.10 -m venv .

Then continue by setting the user on the Python folders created by the previous command:

cd /srv/homeassistant/lib
sudo -u homeassistant -H -s

Activate the Python venv:

source /srv/homeassistant/bin/activate

And now proceed to Install the requirements (noting you might have to resolve some errors, and run the last command a couple of times after editing your requirements.txt file)

cd /home/homeassistant/.homeassistant
pip3 install --upgrade pip
pip install --upgrade pip
pip3 install -r requirements.txt

NOTE) If you get an Error about not having access to some Python Libraries (e.g. ERROR: Could not install packages due to an Environment Error: [Errno 13] Permission denied: '/srv/homeassistant/lib/python3.10/site-packages/six.py’) Then do the following (which recursively sets the correct owner - the homeassistant account we’re using to execute the pip install command with - on the directory the installer is trying to access):

cd /srv
sudo chown -R homeassistant:homeassistant homeassistant

After you’ve set the owner using the chown command, you can go back to the HASS config directory, and re-run the pip install command again as per:

cd /home/homeassistant/.homeassistant
pip3 install -r requirements.txt

F) Finalise / Activate your old HASS Install on the new Python version:

Now that all the requirements have been installed for Python V3.10.9 , you can go ahead and install HASS (still the current / old version, just running on the newer Python VE)

If you want to, you can skip straight ahead to step G) from here, however I would recommend rather seeing if your old version of HASS will run without issues on the newer version of Python.

If you’re playing it safe, and dont skip to G), then install homeassistant by running:

pip3 install homeassistant

If everything went ahead without errors / issues then you can start your HASS instance by running:

sudo systemctl start [email protected]

You can also monitor the HASS startup through viewing the log file with the following command:

sudo nano /home/homeassistant/.homeassistant/home-assistant.log

Recorder might start rebuilding DB’s at this point, and this may take some time, just monitor the progress and wait for HASS to become fully available to confirm all is good, if you’re happy / confident, then proceed with upgrading in the Next step.

G) Upgrade HASS to the Latest version:

At this point, HASS should be running on a supported version of Python with all your Python Requirements / components / dependencies installed & loaded.

To proceed with the upgrade do the following:

Run the following command to activate your Python instance:

source /srv/homeassistant/bin/activate

Stop the Homeassistant service

sudo systemctl stop [email protected]

Before we upgrade, there are several sub-components which needs to be upgraded for later versions of HASS, here is a few that you should run before attempting the latest build:

pip3 install --upgrade hass-nabucasa
pip install --use-deprecated=legacy-resolver cryptography==38.0.3
pip3 install pyOpenSSL --upgrade

Once these are done you can finally Upgrade HASS:

pip3 install --upgrade homeassistant

Start home assistant by running:

sudo systemctl start [email protected]

Once all of this is done, HASS should be the latest version, Python should be a supported version, and all should be good.

Once you have reached this point, wait for any recorder/db checks & rebuilds, and wait until HASS becomes available again, you can monitor the progress in the log file again by running:

sudo nano /home/homeassistant/.homeassistant/home-assistant.log

HOWEVER…

If you view the log and you see an Error that reads: "Version 3.22.0 of SQLite is not supported; minimum supported version is 3.31.0. " then an upgrade of SQL lite is required

To fix this, do the following:
H) Start a Fresh SSH session, & Install & configure SQL Lite by:

sudo systemctl stop [email protected]
wget https://sqlite.org/2021/sqlite-autoconf-3360000.tar.gz
tar -xvf sqlite-autoconf-3360000.tar.gz
./configure
make
sudo make install

Once the supported version of SQL lite is downloaded & installed, you will probably still have the error when trying to start Home Assistant, reason for this is because the SQL Lite install happens to the default location (/usr/local/lib/), and is not in a place where HomeAssistant expects it to be (/usr/lib/xxxx).

To correct this, you first have to determine your system type (For Example, a normal x86/x64 PC will be x86_64-linux-gnu, and a Raspberry PI it would be arm-linux-gnueabihf).
Once you know this you can copy the files from the default install to the folder HASS expects.

For a x86 or x64 Ubuntu system:

sudo cp /usr/local/lib/*sql* /usr/lib/x86_64-linux-gnu/

For a Raspberry Pi ARM system:

sudo cp /usr/local/lib/*sql* /usr/lib/arm-linux-gnueabihf/

Once copied across, we need to set the permissions on the above directories.
For a x86 or x64 Ubuntu system:

sudo chmod a+x /usr/lib/x86_64-linux-gnu/*sql*

For a Raspberry Pi ARM system:

sudo chmod a+x /usr/lib/arm-linux-gnueabihf/*sql*

Once this is done, you can Start the HASS Instance:

sudo systemctl start [email protected]

Once again, monitor the startup and you should be good to go, however recorded might take some time to rebuild the DB, depending on its size.

sudo nano /home/homeassistant/.homeassistant/home-assistant.log

Hopefully this helps, or at least gives you an understanding of the processes required to run HASS on a supported version of Python. I’ve tried to be as descriptive as possible, to convey an understanding of what is required, rather than just giving you a bunch of commands.

If you’re still having issues, reply to this thread and I’ll try to see if I can help.

5 Likes

I tried to follow your instructions for updating Python. I’m trying to update Python 3.10 because 3.9 is not supported with HA 2023.2.
Until now I was not able to upgrade Python.
My questions:

  • Step E: Which directory do you mean in the second line? /srv/homeassistant of /home/homeassistant ?
  • Step F: After executing the last line I got the following error: “ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: ‘/srv/homeassistant/lib/python3.10/site-packages/voluptuous’ Check the permissions.”

Who can give me answers to the above questions?

Thanks in advance.

Hi André

I have updated the guide now for Python 3.10.9, and tried to be a bit more descriptive of the directories, processes etc.

To answer your questions, the directory is /srv/homeassistant
For step F, I’ve added more descriptions, but if you get a permission error, just follow the instructions given in the “NOTE” section in step E, or you could try running the install command with “sudo” first and then re-running it again with the normal “pip3 install -r requirements.txt” command.

Great post ! Thank you !
No chance with “sudo python3.10 -m venv .” at step E
because I don’t know the homeassisant password but “python3.10 -m venv .” did the job. Then same thing with “sudo -u homeassistant -H -s” just after. I omit it.

Now my HASS is running with Pyhton 3.10.9 and next HA upgrade won’t be an issue.

cheers

It’s a pleasure!

1 Like

follow your instructions for updating Python

Thanks for the instruction. Manage to upgrage Python but get an error at the following instruction in section E

Error log

  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      /srv/homeassistant/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
        warnings.warn(msg, warning_class)
      /srv/homeassistant/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      WARNING: The wheel package is not available.
      WARNING: The wheel package is not available.
        error: subprocess-exited-with-error

        × python setup.py bdist_wheel did not run successfully.
        │ exit code: 1
        ╰─> [8 lines of output]
            /srv/homeassistant/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
              warnings.warn(msg, warning_class)
            usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
               or: setup.py --help [cmd1 cmd2 ...]
               or: setup.py --help-commands
               or: setup.py cmd --help

            error: invalid command 'bdist_wheel'
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for cffi
      ERROR: Failed to build one or more wheels
 

Any suggestions? Thanks

Solved the issue, problem was related to wheel
following action resolved the issue

python3 -m pip install wheel

Android 13, chrooted in ubuntu jammy

I just deleted folder and install HA core normally, took couple of minutes.

sudo -u homeassistant -H -s
cd /srv/homeassistant

rm -r *

python3 -m venv .
source bin/activate

python3 -m pip install wheel
pip3 install homeassistant

Great was about to respond with that :slight_smile:

If anyone has problems with Bluetooth after upgrading to Python 3.10 try this solution here. Short version

sudo setcap 'cap_net_raw,cap_net_admin+eip' /usr/local/bin/python3.10
sudo systemctl restart [email protected]