HA core: numpy is installed but i get no module named numpy when pip install libtiff

FreeBSD 13.2
HA core
Python 3.11

My setup has been working fine for a while but after a power outtage, I wasn’t able to start HA core properly (the system has been restarted multiple time since my last update, so this is strange).

So I get error from image_upload which would require libtiff.so.5. So I tried installing libtiff with pip. I don’t know if that would solve the problem.

So I tried pip install libtiff with pip, pip3, pip3.9,pip3.11 and it says I need numpy which is supposed to be installed. So I’ve installed numpy with all the above pip command. I still cannot install libtiff.

This is the output of pip3.11 install libtiff:

Collecting libtiff
  Using cached libtiff-0.4.2.tar.gz (129 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
Collecting libtiff
  Using cached libtiff-0.4.2.tar.gz (129 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "/hass/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/hass/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/hass/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-_a57xvve/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-_a57xvve/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-_a57xvve/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-_a57xvve/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 77, in <module>
      ModuleNotFoundError: No module named 'numpy'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Corrupt SD after your power outage?

Are you running HA Core in a venv?

Yes I,m running HA core in venv.

I don’t think anything has corrupted since my other hosted app works fine. That would be very specific data corruption.

I believe it may have something to do with python 3.9 upgraded to 3.11?

I’m not sure how python is managing its different version dependencies, but it seems HA uses 3.9 and 3.11. Is that a problem?

I’ve been able to install libtiff with:
python3 -m pip install numpy --upgrade
python3 -m pip install libtiff

But that doesn’t solve the problem.

This is the error I get:
Setup failed for 'image_upload': Unable to import component: Shared object "libtiff.so.5" not found, required by "_imaging.cpython-311.so"

Did you make a typo here? It cannot be both. HA supports 3.11 and up. Did you also upgrade your Python?

HA determines its dependencies, but something like numpy will depend on underlying C and Fortran libs, which are provided by the OS.

Maybe your distro needs an upgrade.

I’ve updated to freebsd 14, I still get the same error.

I don’t know what’s with libtiff.so.5

I’m using Python 3.11. I was just wondering if it could be some kind of error related to updating from 3.9 to 3.11.

How did you upgrade Python? Did you upgrade the system Python or did you do a separate installation (typically built from sources)?

Upgraded by installing python 3.11 with pkg manager.

This is my general upgrade setup:

  • Upgrade

    • Install python pkg: pkg install pythonXXX pyXXX-sqlite3 pyxxx-pillow
    • Source activate in homeassistant venv dir: . bin/activate
    • Get requirements: bin/pip3 freeze --local > requirements.txt ## forget about that step
    • Install venv pythonXX.XX -m venv .
    • As homeassistant user: . bin/activate
    • Upgrade
      pip3 install --upgrade pip
      pip install --upgrade pip
      # Remove homeassistant and cryptography from requirements first. next step is useless
      pip3 install -r requirements.txt
      
    • Fix problems then install home assistant: pip3 install homeassistant
    • Outside venv (run as root):
      git clone https://github.com/numpy/numpy.git numpy-git
      cd numpy-git
      git checkout v1.26.0
      git cherry-pick 040ed2d
      git submodule update --init
      cd ..
      
    • In venv as homeassistant: pip install numpy-git/
    • pip3 install numpy
    • `pip3 install git+https://github.com/rhasspy/webrtc-noise-gain.git

This needs to be updated

You should only upgrade the system Python to a version directly supported by your distro. I’m unfortunately not very familiar with package management on FreeBSD, being a Debian user myself. But, what I’m getting at is that you should generally speaking only update Python with your distro’s upgrade and updates. If you install a specific Python version like above, it might not be a version your distro would’ve upgraded to, causing all kinds of dependency issues.

Your creating of the venv and all that looks fine. I’d recommend that you revert to the last known working Python version (the 3.9.x version you had) and then do a side-by-side Python install of 3.11 (or even better, 3.12) by building from sources.

Why are you cherry picking numpy?

I’m cherry picking numpy for a previous bug, Unfortunately, I did not keep notes of the actual bug but running HA on FreeBSD isn’t a smooth experience.

python 3.11 is currently the latest supported python on freebsd

Sorry I cannot be of more help.

Try to to downgrade your Python again, at least as a test to see if it resolves the issue.

Otherwise, all I can say is that an .so is a dynamically linked library, so you might be missing the symbols or something. You could try to install the dev package for libtiff. This will (should) provide the header files that might help the linker to resolve this.

Also check the respective changelogs for numpy and libtiff to see if there’s a specific version of libtiff that works with the version of numpy you installed.

EDIT: We have some issues with the major numpy v2 release at work. Not sure if that’s what you upgraded to.

I’ve been able to solve it by simply moving the hass venv folder to backup and create a new venv up to date.

So when you upgraded Python, did you not create a new venv as you stated here?

No, I did not create a new venv for updating