Upgrading HomeAssistant Core in a Python VE

Hi,

I just want to preface this, as I’ve seen a couple of queries in the comments which I dont know how to address… this Upgrade / My Environment, is with HomeAssistant, running on an Ubuntu Server (a propper server, which also runs other workloads like Plex, and Web Services). Theoretically this process should work for you, if you have something similar configured, e.g. an Ubuntu Server of Flavour xxx, and you’re running HASS in a Python VE.

Time for me to Upgrade my HomeAssistance instance again (28 June 2024)
This revision is now for Python version 3.12, and to Upgrade to HomeAssistant version 2024.6

Before you Continue, pleasee read the comments below. Alot of people are doing the Python Upgrade withing the reqquirements.txt file. This would be better for some, perhaps even most users. I did inlcude it, as I have alot of other components and dependancies I do not want to fix / update. The requirements.txt is the easiest way for me to ensure that everything is covered, without treating my upgrade as a newe install. I do strongly recommend that if you do not have a very complicated setup, to follow the guidance in the comments below, and just exclude it.


Installing or upgrading HASS to the latest & greatest in a Python Virtual Environment, may be tricky. So I thought I’d create a post after I’ve struggled through it enough times now.

Hopefully this helps other members.

In summary there is a couple of steps involved, but from a High-Level, this is what’s required:

  1. Ensure you have a supported Python version for the latest build. (Currently for HASS 2024.60, the earliest supported version of Python is 3.12 - This guide has been updated for Python 3.12)
  2. If required, Upgrade Python on your platform (this guide is for Debian)
  3. Freeze your current Python dependencies / packages for HASS & Integrations
  4. Create a backup of your old HASS environment
  5. Activate a new Python VE on the later version of Python
  6. Rebuild HASS to run in the new Python virtual environment
  7. Upgrade HASS after its running from the new Python environment
  8. Check for other breaking changes (For example the SQL Lite version).

So, in summary here is what I did to upgrade my HASS core deployment on Ubuntu 18.04 to the latest:

A) Log into the shell.
B) Check / Upgrade your current python version:

python3 --version

If you have a Python version that is supported (3.10.9 at the time of writing this) or above then skip to step G) otherwise continue to upgrade Python as follow:

Download the required Python Version (this is for version 3.10.9):

cd /
sudo wget https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tgz

Unpack it:

sudo tar -xzf Python-3.12.0.tgz

Make & Install it:

cd Python-3.12.0
./configure
make -j 4
sudo make install

I also had to ensure that Python Verion 3.11 was completely removed by:

sudo apt update && sudo apt remove --purge python3.11 && sudo apt install python3.12

After This I verivied that Python V3.12 was actually running with re-running the command:

python3 --version

C) Freeze current Python packages

This will grab the list of current Packages used in your HASS environment, and save them to a “requirements.txt” file which you will use later on to install all the required packages in your new Python VE):

cd /home/homeassistant/.homeassistant
sudo -u homeassistant -H -s
source /srv/homeassistant/bin/activate
pip3 freeze –local > requirements-2024.06.txt
exit

D) Backup your existing HASS folder

If you’ve previously done this, you will already have a folder called srv/homeassistantold. Either remove the old homeassistantold directory with the command “sudo rm -r homeassistantold” or when you run the move (mv) command below, specify another directory (e.g. homeassistantold-versionx)

To create a backup, we simply create a copy of the “/srv/homeassistant” folder, if you want to restore to a previous “backup”, you would copy the backup (e.g. /srv/homeassistantold) back to its original folder (/srv/homeassistant) using the cp command.

To create the backup do the following:

Stop your HASS instance:

sudo systemctl stop [email protected]

Go to the SRV folder, and copy the homeassistant program directory to your backup folder:

cd /srv
sudo mv homeassistant homeassistantold202406

Create a new homeassistant folder, which will run on the new version of Python:

sudo mkdir homeassistant

Set ownership of the folder for the homeassistant account:

sudo chown -R homeassistant:homeassistant homeassistant
sudo -u homeassistant -H -s

E) Install Python & component dependencies for the new Python Version.

During this step - depending on your previous install and 3’rd party integrations - some dependencies might fail due to compatibility of integrations / components with the Python 3.11 environment (which you will note as errors that pop up when you run the “pip3 install -r requirements.txt” command).

If this is the case, then either remove the dependency from your requirements.txt file (sudo nano requirements.txt) in which case the components that depends on the removed item will likely fail, or you could resolve each of the failures by “googling” the error, and applying a workaround by changing versions etc in the requirements.txt file, until you’re able to run the last command (pip3 install -r requirements.txt) without any errors.

An Example of this would be an integration that I had which requires “backports.zoneinfo” version 0.2.1 that is no longer supported on Python 3.10. According to this thread: python - ERROR: Could not build wheels for backports.zoneinfo, which is required to install pyproject.toml-based projects - Stack Overflow , I had to replace the “backports.zoneinfo==0.2.1” in the requirements.txt file with backports.zoneinfo;python_version<“3.9” as per the workaround described in the article.

Another example where editing the requirements.txt file was required for me on Python 3.10 was for the ruamel.yaml version 0.15.100 dependency, which needed to be changed to 0.16.11, as per this article: ruamel dependency broken with setuptools>=50 · Issue #39987 · home-assistant/core · GitHub

This can be frustrating, but just work through the errors one by one, until you get an error free pip install of the requirements.txt file.

You might also have to wait for some components to be updated by the developer after a new Python version, so you might need to exclude a component, until its been updated by the developer to work on your Python version, and circle back to this component/integration when you want to re-enable it after a new version has been released.

For the time being, do not attempt to go with Python 3.11, there are too many components that will fail, just stick with what is required e.g. 3.10

To start off, we’ll first add Python to the newly created “srv/homeassistant” directory:

cd /srv/homeassistant
sudo python3.12 -m venv .

Then continue by setting the user on the Python folders created by the previous command:

cd /srv/homeassistant/lib
sudo -u homeassistant -H -s

Activate the Python venv:

source /srv/homeassistant/bin/activate

And now proceed to Install the requirements (noting you might have to resolve some errors, and run the last command a couple of times after editing your requirements.txt file)

cd /home/homeassistant/.homeassistant
pip3 install --upgrade pip
pip install --upgrade pip
pip3 install -r requirements-2024.06.txt

NOTE) If you get an Error about not having access to some Python Libraries (e.g. ERROR: Could not install packages due to an Environment Error: [Errno 13] Permission denied: '/srv/homeassistant/lib/python3.10/site-packages/six.py’) Then do the following (which recursively sets the correct owner - the homeassistant account we’re using to execute the pip install command with - on the directory the installer is trying to access):

cd /srv
sudo chown -R homeassistant:homeassistant homeassistant

After you’ve set the owner using the chown command, you can go back to the HASS config directory, and re-run the pip install command again as per:

cd /home/homeassistant/.homeassistant
pip3 install -r requirements-2024.06.txt

If you still get errors at this point, you could also run the pip3 install with sudo, and then set the permissions manually.

cd /home/homeassistant/.homeassistant
sudo pip3 install -r requirements-2024.06.txt
cd /srv
sudo chown -R homeassistant:homeassistant homeassistant

I know it seems redundant running it twice, but this has worked for me in the past.
Also one thing to note the libxml12-dev and libxslt-dev components are notrious difficult to install. I Found this link usefull: Question #700282 “libxml2-dev : Depends: libicu-dev but it is no...” : Questions : libxml2 package : Ubuntu

F) Finalise / Activate your old HASS Install on the new Python version:

Now that all the requirements have been installed for Python V3.10.9 , you can go ahead and install HASS (still the current / old version, just running on the newer Python VE)

If you want to, you can skip straight ahead to step G) from here, however I would recommend rather seeing if your old version of HASS will run without issues on the newer version of Python.

If you’re playing it safe, and dont skip to G), then install homeassistant by running:

pip3 install homeassistant

If everything went ahead without errors / issues then you can start your HASS instance by running:

sudo systemctl start [email protected]

You can also monitor the HASS startup through viewing the log file with the following command:

sudo nano /home/homeassistant/.homeassistant/home-assistant.log

Recorder might start rebuilding DB’s at this point, and this may take some time, just monitor the progress and wait for HASS to become fully available to confirm all is good, if you’re happy / confident, then proceed with upgrading in the Next step.

G) Upgrade HASS to the Latest version:

At this point, HASS should be running on a supported version of Python with all your Python Requirements / components / dependencies installed & loaded.

To proceed with the upgrade do the following:

Run the following command to activate your Python instance:

source /srv/homeassistant/bin/activate

Stop the Homeassistant service

sudo systemctl stop [email protected]

Before we upgrade, there are several sub-components which needs to be upgraded for later versions of HASS, here is a few that you should run before attempting the latest build: (Note you should Upgrade hass-nabucasa, then pyOpenSSL, then cryptography, if you get errors here, re-run these commands so that all conflicts are resolved, before continuing, by changing the order you install them in)

pip3 install --upgrade hass-nabucasa
pip3 install pyOpenSSL --upgrade
pip install --use-deprecated=legacy-resolver cryptography==42.0.0

Once these are done you can finally Upgrade HASS:

pip3 install --upgrade homeassistant

After I confirmed the downloaded version of 2024.06 I Start home assistant by running:

sudo systemctl start [email protected]

And Et Voila! HASS is Upgraded.

Once all of this is done, HASS should be the latest version, Python should be a supported version, and all should be good.

Once you have reached this point, wait for any recorder/db checks & rebuilds, and wait until HASS becomes available again, you can monitor the progress in the log file again by running:

sudo nano /home/homeassistant/.homeassistant/home-assistant.log

HOWEVER…

If you view the log and you see an Error that reads: "Version 3.22.0 of SQLite is not supported; minimum supported version is 3.31.0. " then an upgrade of SQL lite is required

To fix this, do the following:
H) Start a Fresh SSH session, & Install & configure SQL Lite by:

sudo systemctl stop [email protected]
wget https://sqlite.org/2021/sqlite-autoconf-3360000.tar.gz
tar -xvf sqlite-autoconf-3360000.tar.gz
./configure
make
sudo make install

Once the supported version of SQL lite is downloaded & installed, you will probably still have the error when trying to start Home Assistant, reason for this is because the SQL Lite install happens to the default location (/usr/local/lib/), and is not in a place where HomeAssistant expects it to be (/usr/lib/xxxx).

To correct this, you first have to determine your system type (For Example, a normal x86/x64 PC will be x86_64-linux-gnu, and a Raspberry PI it would be arm-linux-gnueabihf).
Once you know this you can copy the files from the default install to the folder HASS expects.

For a x86 or x64 Ubuntu system:

sudo cp /usr/local/lib/*sql* /usr/lib/x86_64-linux-gnu/

For a Raspberry Pi ARM system:

sudo cp /usr/local/lib/*sql* /usr/lib/arm-linux-gnueabihf/

Once copied across, we need to set the permissions on the above directories.
For a x86 or x64 Ubuntu system:

sudo chmod a+x /usr/lib/x86_64-linux-gnu/*sql*

For a Raspberry Pi ARM system:

sudo chmod a+x /usr/lib/arm-linux-gnueabihf/*sql*

Once this is done, you can Start the HASS Instance:

sudo systemctl start [email protected]

Once again, monitor the startup and you should be good to go, however recorded might take some time to rebuild the DB, depending on its size.

sudo nano /home/homeassistant/.homeassistant/home-assistant.log

Hopefully this helps, or at least gives you an understanding of the processes required to run HASS on a supported version of Python. I’ve tried to be as descriptive as possible, to convey an understanding of what is required, rather than just giving you a bunch of commands.

If you’re still having issues, reply to this thread and I’ll try to see if I can help.

6 Likes

I tried to follow your instructions for updating Python. I’m trying to update Python 3.10 because 3.9 is not supported with HA 2023.2.
Until now I was not able to upgrade Python.
My questions:

  • Step E: Which directory do you mean in the second line? /srv/homeassistant of /home/homeassistant ?
  • Step F: After executing the last line I got the following error: “ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: ‘/srv/homeassistant/lib/python3.10/site-packages/voluptuous’ Check the permissions.”

Who can give me answers to the above questions?

Thanks in advance.

Hi André

I have updated the guide now for Python 3.10.9, and tried to be a bit more descriptive of the directories, processes etc.

To answer your questions, the directory is /srv/homeassistant
For step F, I’ve added more descriptions, but if you get a permission error, just follow the instructions given in the “NOTE” section in step E, or you could try running the install command with “sudo” first and then re-running it again with the normal “pip3 install -r requirements.txt” command.

Great post ! Thank you !
No chance with “sudo python3.10 -m venv .” at step E
because I don’t know the homeassisant password but “python3.10 -m venv .” did the job. Then same thing with “sudo -u homeassistant -H -s” just after. I omit it.

Now my HASS is running with Pyhton 3.10.9 and next HA upgrade won’t be an issue.

cheers

1 Like

It’s a pleasure!

1 Like

follow your instructions for updating Python

Thanks for the instruction. Manage to upgrage Python but get an error at the following instruction in section E

Error log

  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      /srv/homeassistant/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
        warnings.warn(msg, warning_class)
      /srv/homeassistant/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      WARNING: The wheel package is not available.
      WARNING: The wheel package is not available.
        error: subprocess-exited-with-error

        × python setup.py bdist_wheel did not run successfully.
        │ exit code: 1
        ╰─> [8 lines of output]
            /srv/homeassistant/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
              warnings.warn(msg, warning_class)
            usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
               or: setup.py --help [cmd1 cmd2 ...]
               or: setup.py --help-commands
               or: setup.py cmd --help

            error: invalid command 'bdist_wheel'
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for cffi
      ERROR: Failed to build one or more wheels
 

Any suggestions? Thanks

Solved the issue, problem was related to wheel
following action resolved the issue

python3 -m pip install wheel

Android 13, chrooted in ubuntu jammy

I just deleted folder and install HA core normally, took couple of minutes.

sudo -u homeassistant -H -s
cd /srv/homeassistant

rm -r *

python3 -m venv .
source bin/activate

python3 -m pip install wheel
pip3 install homeassistant

Great was about to respond with that :slight_smile:

If anyone has problems with Bluetooth after upgrading to Python 3.10 try this solution here. Short version

sudo setcap 'cap_net_raw,cap_net_admin+eip' /usr/local/bin/python3.10
sudo systemctl restart home-assistant@homeassistant

Sorry I might sound stupid but how do I update Python if I am using the HAOS running on a VM on Proxmox?

image

Am I missing something ? The base OS I guess is Debian/Ubuntu but how can I run other commands rather than the ones the CLI offers ?

thanks

This guide is not for HAOS but for HA Core.
In your case Python is updated by the OS.

I would like to suggest to stop EVERY possible service (HA, as well) before running “make”, if you are an old Rpi (like mine 3B+) or it will take a very looooooong time!

Thanks for trying to shed some light on the worst mess I’ve seen in 43 years since I started with an Apple ][!

  1. Is home-assistant (with -) above a typo?
  2. Edit: You use sudo with python3 -m venv . The docs don’t.
  3. Thanks for the suffice-with-3.10 tip. Was about to use 3.11. Edit: Will simply KILL it. How about trying with pyenv?

Later
G.

Hi Maarten,

This unfortunately is the most complex part of the upgrade. Each persons HomeAssistant Integrations are different, so the requirements are different.

What is happening here is the PIP installer is trying to install all the requirements / prerequisites which are unique to your environment & dependant on integrations / components you have configured in HASS.

You might need to try and resolve those individually, or worst case, you might need to edit your requirements.txt file, and remove the items which PIP has a problem with installing.

One item that might be easily resolvable for instance is the “license_file parameter is deprecated, use license_files instead” where you might just need to update the requirements.txt file to use “license_files” instead of “license_file”.

Hi, I’m running HA with a Python VEnv too and have done multiple Python-Updates ( 3.9, 3.10, 3.11) since I started with HA.
My recommendation is NOT to use the “requirements.txt”-approach.
After downloading and installing Python, creating a new VEnv for the new version like you do describe in the guide I always handled it like a new install of HA and installed HA with pip as described in documentation, then changed the service to point to the new folder and started HA. On startup ( which takes quite a while in this case) it pulls all missing requirements, including those for custom components.
Advantage is that you don’t carry over any dependencies from the old VEnv and the old Python-version

Of course this approach also might produce problems
For cryptography a manual install of rustc was required ( believe this was with Python 3.9), otherwise Python was unable to compile the package
Manually installed python libraries have to be installed again. For example I’m using a Infrared Receiver and the “lirc”-package requires manual installation from source into each new environment

my 2 cents
Armin

Thanks for a great guide @Pieter_Rossouw ! Below is a quick guide that describe what I did to go from Python 3.11 to 3.12.

Assumptions

  • HA is run as root. I personally run it in a LXC container (Proxmox). See additional steps provided in the first post in this thread for steps required to use a different user.
  • The Python runtime previously installed using the deadsnakes PPA.
  • Packages already installed: apt install wget build-essential libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev python3-pip

Install new python version

apt install python3.12 python3.12-dev python3.12-venv

Stop HA

systemctl stop home-assistant@root

Write package requirements to requirements.txt

cd /root/.homeassistant
source /srv/homeassistant/bin/activate
pip3 freeze –local > requirements.txt
exit

Backup existing HA runtime

cd /srv
mv homeassistant homeassistantold

Set up new runtime

cd /srv
mkdir homeassistant
cd /srv/homeassistant
python3.12 -m venv .

Activate python env

source /srv/homeassistant/bin/activate

Install dependencies

cd /root/.homeassistant
pip3 install --upgrade pip
pip install --upgrade pip
pip3 install -r requirements.txt

Install HA

pip3 install homeassistant

Start it again

systemctl start home-assistant@root

1 Like

I just upgraded this and did the same … no requirements.txt. Just backup the old install and reinstall from scratch. Assuming you have the home-assistant service (for stop and start) …

sudo apt update
sudo apt install python3.12
sudo apt install python3.12-venv
sudo apt install python3.12-dev
sudo systemctl stop home-assistant@homeassistant
cd /srv
sudo mv homeassistant homeassistant-3.11
sudo python3.12 -m venv homeassistant
sudo chown -R homeassistant:homeassistant homeassistant/
pip3 install wheel
cd homeassistant
sudo -u homeassistant -H -s
source bin/activate
pip3 install --upgrade homeassistant
pip3 install --upgrade home-assistant-frontend
exit
sudo systemctl start home-assistant@homeassistant

Worked a charm. Yes you patiently wait for all the python to compile and such (5 minutes or so) but up and running and faster than before

Hello, I’m having this problem… how to solve it?
thanks

root@raspberrypi:/home/lucavasini# sudo apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian-security bookworm-security InRelease
Hit:3 http://deb.debian.org/debian bookworm-updates InRelease
Hit:4 http://archive.raspberrypi.com/debian bookworm InRelease
Hit:5 https://download.docker.com/linux/debian bookworm InRelease
Ign:6 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu bookworm InRelease
Err:7 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu bookworm Release
  404  Not Found [IP: 185.125.190.80 443]