Python 3.6 upgrade of a virtualenv

Configuration
1

Everyone has been telling us the 3.6 upgrade for python is going to be required soon. I started running into some errors with some other things because I hadn’t upgraded, so I went ahead and figured I would spend the weekend making this work. Luckily so far, its been relatively painless. Basically there isn’t a good way to upgrade a python virtualenv. You really just have to blow the old one away and start over. But before you have a heart attack, that’s not to say you are blowing away your HomeAssistant install, just the virtualenvironment directory. So here is what I did, and to my suprise, I didn’t lose anything.

ha python 3.6 upgrade

BACKUP EVERYTHING first If possible make a disk image to a thumb drive or something.

first you will need to install the following (thanks @thundergreen)

sudo apt-get install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev

Then follow the instructions here for installing python3.6


The first answer is the one that I followed. Basically:
deactivate any virtualenv you may be in. Then from the pi home directory, 
wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tgz
tar xzvf Python-3.6.0.tgz
cd Python-3.6.0/
./configure
make
sudo make install

After this you should have python 2.7, python 3.4 and python 3.6 installed.

re-activate the current home assistant virtual environment
source <path to virtualenv>/bin/activate

with the home assistant virtual environment active
from the home assistant home directory

pip3 freeze —local > requirements.txt

This generates a file with the versions of all the packages you have installed. It can be used later to re-install all those packages.

Check this file and make sure it looks ok

deactivate the ha virtual environment

deactivate

Delete your virtual environment. In my case it was /srv/homeassistant I think the standard install puts it in /srv/homeassistant/homeassistant_venv or something like that.

use python3.6 to create the new virtual environment

python3.6 -m venv /srv/homeassistant

use the path to where you want your virtualenv to be.

activate the new virtual environment

source /srv/homeassistant/bin/activate

Again, use the path to your virtualenv.

with the new virtual environment active do the following checks.

python —version
python3 —version
pip3 —version

all three should indicate python3.6

go to the home assistant home directory

cd /home/homeassistant

Now we use the requirements.txt file we captured earlier.
pip3 install -r requirements.txt

This should install all the packages that were installed in the old virtual environment.

The openzwave items gave me a problem installing. I commented out the following items. Try installing with them in, you may have better luck than me.

libopenzwave==0.3.2 
openzwave==0.3.2
pyozwman==0.3.2
pyozwweb==0.3.2

If it fails on any of these, just comment it out, and run the pip3 install -r requirements.txt again. It will try to re-install the ones it has already done, but it doesn’t do any harm.

It may take a few attempts commenting out the openzwave items to get through, but eventually the package collection will finish and installation will start.

everything seems to have installed ok at least no errors

if you moved your virtual environment (I did by mistake), edit the /etc/systemd/systems/[email protected] file to correct the virtualenv directories

if you had to change the services file
sudo systemctl daemon-reload

to start homeassistant
sudo systemctl start [email protected]

Lets see if it’s running
ps -ef | grep home

see if python3.6 is running hass

my browser came up but i had invalid components for zwave, discovery and zeroconf so I’m going to try installing them manually

pip3 install zeroconf
this seems to have taken care of discovery too.

z-wave gave me a problem. So, with the virtualenv active

sudo apt-get install libudev-dev

then

pip3 install —upgrade python_openzwave

it took a while, but just let it run. Restarted HA and everything is there.

At this point, it may be a good thing to do an uninstall/install on HomeAssistant, just like you would to pull down the latest upgrade. The pip -r install earlier should have pulled down all the packages but it won’t hurt to do the upgrade.

please leave a comment if any of this doesn’t work for you or if you find a better way.

19 Likes
2

I had also delete dependency folder in hass home folder.

3

Hi @turboc thanks for the walkthrough!
Might be helpful if no updated image of hassbian comes out in time :slight_smile:

Is there a reason for choosing python 3.6 over 3.6.2, though?

Jochen

4

I think when I originally started on this, 3.6.2 was still a candidate and 3.6.1 hadn’t been released for long. Upgrading a minor release like that though shouldn’t be a problem that will impact the virtual environment though. It should be able to be done in place. At least that’s what I’ve read.

5

come to think of it I did that too. I did an uninstall on HA and then did a re-install (just like a normal upgrade). But I did it trying to get openzwave to load automatically so it didn’t occur to me that I was re-installing those apps as well. Good point.

6

Chip, shouldn’t this be in “Share Your Projects”? Seems appropriate for a how to like this.

1 Like
7

At the end of configuring 3.6.1 (which is the latest stable 3.6.x version) I get the following message:

If you want a release build with all optimizations active (LTO, PGO, etc),
please run ./configure --enable-optimizations

Optimizations sound good to me so perhaps this should be used instead of a plain ./configure.

Also, I have been testing a bit with pyenv which makes the whole switching/upgrading part a little easier in the future as well. Simply tell pyenv to install version X.Y.Z and it installs it locally (and not adding it to /usr/bin etc). Afterwards you can simply create a fresh virtualenv with that particular version.

Also, with pyenv local you can transparently use the virtualenv, making it even easier to upgrade.
For instance, if Python 3.7.0 is released it would be as easy as:

  1. pyenv install 3.7.0
  2. pyenv local 3.7.0

Re-run pip to correct any version-specific dependencies and that’s it. At least, it seems that way.

8

I am going to bump up to 3.6 also. But I am afraid ths will break my openzwave component? Does aboe described procedure keep openzwave alive? :slight_smile:

I will just install python 3.6 for the moment till z wave is going well. I saw a post in which it was explained manually instal openzwave but i lost link :frowning: this might help here … will research

9

Note The pyvenv script has been deprecated as of Python 3.6 in favor of using python3 -m venv to help prevent any potential confusion as to which Python interpreter a virtual environment will be based on.

https://docs.python.org/3/library/venv.html

10

What I ended up doing to get openzwave doing was install python_openzwave from inside the virtual environment. It is a pip wrapper to handle the install. I used the default which seems to be 0.4.0. Interestingly their site says that is the beta version, but that’s what the default install did. So I don’t know if the readme is out of date, or the default install pulled down the right thing.

Like I said at the beginning. BACKUP EVERYTHING. I did an image backup of my sd card prior to trying this just in case I had to fall back. This worked for my install. I don’t know how it will behave for anyone elses.

11

Yes, but that is for pyvenv not pyenv :wink: Its something different.

12

Sorry, misread.

13

well… i’ll wait monday to give it a try as my hass is backuped each sunday night thanks so far!

14

It worked for me, thanks! But for my setup it was necessary to delete the deps/-Folder so HA sets up all dependencies again.

15

Also zwave ?

16

It’s new since .45 that zwave is being installed by pip right? So normally it should work fine . Did u remove the packages from requirement.txt ?

A step by step would be nice indeed

17

I didn`t have zwave in my setup. And why a step by step? At the top of this thread is a guide

18

thought u would have z wave … that`s why

19

I ended up commenting the openzwave related items out of the requirements.txt as I re-ran it and they errored out. It doesn’t hurt to run the requirements.txt step multiple times since its erroring when it’s downloading not installing, and even if it were installing, it would see the existing packages and just skip over them.

This basically is a step by step without numbers and with poor formatting I guess. The problem is, everyone’s install can be a little different so telling someone to update this file may be appropriate for me, but it may not for you.

20

The guide just installs the python interpreter. The rest of it is how to deal with it with HA and virtualenv.