Manual development environment setup glitch on Linux

WillS · January 19, 2023, 11:08pm

All my Python development has been in Jupyter Notebooks, though I need an integration for my new Darfon solar inverter, so I’m pitching in to make one. I am following the procedure to set up the dev environment from the documentation, and have hit a snag;

Current development environment is Ubuntu Studio ~~18.04~~ 20.04

I installed the core Linux dependencies without a hitch or error (had set it up earlier, but reran it just to capture the output to show here)

will@will-G5-5590:~/core$ sudo apt-get install python3-pip python3-dev python3-venv autoconf libssl-dev libxml2-dev libxslt1-dev libjpeg-dev libffi-dev libudev-dev zlib1g-dev pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libavresample-dev libavfilter-dev ffmpeg
Reading package lists… Done
Building dependency tree
Reading state information… Done
autoconf is already the newest version (2.69-11.1).
libffi-dev is already the newest version (3.3-4).
libjpeg-dev is already the newest version (8c-2ubuntu8).
pkg-config is already the newest version (0.29.1-0ubuntu4).
python3-dev is already the newest version (3.8.2-0ubuntu2).
python3-venv is already the newest version (3.8.2-0ubuntu2).
libssl-dev is already the newest version (1.1.1f-1ubuntu2.16).
libudev-dev is already the newest version (245.4-4ubuntu3.19).
libxml2-dev is already the newest version (2.9.10+dfsg-5ubuntu0.20.04.5).
libxslt1-dev is already the newest version (1.1.34-4ubuntu0.20.04.1).
zlib1g-dev is already the newest version (1:1.2.11.dfsg-2ubuntu1.5).
ffmpeg is already the newest version (7:4.2.7-0ubuntu0.1).
libavcodec-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libavdevice-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libavfilter-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libavformat-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libavresample-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libavutil-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libswscale-dev is already the newest version (7:4.2.7-0ubuntu0.1).
python3-pip is already the newest version (20.0.2-5ubuntu1.6).
The following packages were automatically installed and are no longer required:
libfwupdplugin1 libllvm11 libllvm11:i386 shim
Use ‘sudo apt autoremove’ to remove them.
0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.

I set up the local repository without a htich;

will@will-G5-5590:~$ git clone GitHub - skyemoor/core: 🏡 Open source home automation that puts local control and privacy first.
Cloning into ‘core’…
remote: Enumerating objects: 714880, done.
remote: Counting objects: 100% (1050/1050), done.
remote: Compressing objects: 100% (799/799), done.
remote: Total 714880 (delta 275), reused 994 (delta 250), pack-reused 713830
Receiving objects: 100% (714880/714880), 404.47 MiB | 1.22 MiB/s, done.
Resolving deltas: 100% (511441/511441), done.
Updating files: 100% (30297/30297), done.
will@will-G5-5590:~$ cd core
will@will-G5-5590:~/core$ git remote add upstream GitHub - home-assistant/core: 🏡 Open source home automation that puts local control and privacy first.

Running the setup script was where the problem arose;

will@will-G5-5590:~/core$ script/setup
Copy ./.vscode/settings.default.json to ./.vscode/settings.json.
Installing development dependencies…
Collecting wheel
Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel
Successfully installed wheel-0.38.4

Usage:
/home/will/core/venv/bin/python3 -m pip install [options] [package-index-options] …
/home/will/core/venv/bin/python3 -m pip install [options] -r [package-index-options] …
/home/will/core/venv/bin/python3 -m pip install [options] [-e] …
/home/will/core/venv/bin/python3 -m pip install [options] [-e] …
/home/will/core/venv/bin/python3 -m pip install [options] <archive url/path> …

no such option: --use-deprecated

This appears to be the point where the problem occurred. In the setup script, this line of code is;

python3 -m pip install -e . --constraint homeassistant/package_constraints.txt --use-deprecated=legacy-resolverT

I carried on just to see if this would still work, and ran the venv command, though the following command came back “file not found”

hass -c config

Sure enough, a search of the entire core subdirectories showed no “hass” file of any kind, not surprising as that was to be installed in the line that failed in the setup script.

I followed everything to a T, though the script seems to have a line that cannot be executed. Is there anything I need to correct in the setup script, or did I make a mistake (the latter of which I admit is the most likely)?

WillS · January 20, 2023, 8:37pm

Found one mistake of mine and an update that was helpful;

While I had updated to python 3.9, I didn’t go through the update-alternatives process, something foreign to me in my Anaconda work. So I deleted the repository and files, and reforked the HA repository.

However, a cached wheel held up matters, so I upgrade my pip to be able to easily purge the wheel.

Now everything looks to be going smoothly, though recently it’s taken 8 minutes to load rokuecp and now 45 minutes to load ibeacon.ble and it’s still not done…

Michel · January 21, 2023, 12:35pm

FYI just running script/setup without being in a virtual env will use the system default Python version which is probably not your installed 3.9 (don’t change system default Python, things will break).

To make sure the correct Python version is used by the script first create (or activate) a virtual env with that version and only after that run script/setup.

Creating a virtual env for a specific version is something like this from the top of my head python3.9 -m venv venv

WillS · January 21, 2023, 9:04pm

Thanks!

Have success (Successfully installed homeassistant-2023.2.0.dev0) up to the point that that after

hass -c config

and setting up service, began loading components. Loaded 9 successfully until Bluetooth. Now all I get are warnings about “socket connection lost”, “Bad file descriptor”, and “could not be reset due to timeout”

…
2023-01-21 11:52:44.792 INFO (SyncWorker_5) [homeassistant.loader] Loaded esphome from homeassistant.components.esphome
2023-01-21 11:52:44.796 INFO (SyncWorker_2) [homeassistant.loader] Loaded spotify from homeassistant.components.spotify
2023-01-21 11:52:45.455 INFO (SyncWorker_4) [homeassistant.loader] Loaded homekit_controller from homeassistant.components.homekit_controller
2023-01-21 11:52:45.468 INFO (SyncWorker_3) [homeassistant.loader] Loaded bluetooth_adapters from homeassistant.components.bluetooth_adapters
2023-01-21 11:52:45.474 INFO (SyncWorker_2) [homeassistant.loader] Loaded shelly from homeassistant.components.shelly
2023-01-21 11:52:45.474 INFO (SyncWorker_5) [homeassistant.loader] Loaded ruuvi_gateway from homeassistant.components.ruuvi_gateway
2023-01-21 11:52:45.484 INFO (SyncWorker_4) [homeassistant.loader] Loaded roku from homeassistant.components.roku
2023-01-21 11:52:45.486 INFO (MainThread) [homeassistant.setup] Setting up bluetooth_adapters
2023-01-21 11:52:45.486 INFO (MainThread) [homeassistant.setup] Setup of domain bluetooth_adapters took 0.0 seconds
2023-01-21 11:52:45.664 INFO (SyncWorker_5) [homeassistant.loader] Loaded upnp from homeassistant.components.upnp
2023-01-21 11:53:02.368 INFO (SyncWorker_1) [homeassistant.loader] Loaded intellifire from homeassistant.components.intellifire
2023-01-21 12:19:14.535 INFO (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (7C:B2:7D:6F:31:FE): Bluetooth scanner has gone quiet for 90s, restarting
2023-01-21 12:21:14.540 INFO (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (7C:B2:7D:6F:31:FE): Bluetooth scanner has gone quiet for 90s, restarting
2023-01-21 12:21:19.647 WARNING (MainThread) [bluetooth_auto_recovery.recover] Bluetooth adapter hci0 [7C:B2:7D:6F:31:FE] could not be reset due to timeout
2023-01-21 12:21:19.649 INFO (MainThread) [bluetooth_auto_recovery.recover] hci0 permission denied to /dev/bus/usb/001/003 while attempting USB reset: [Errno 13] Permission denied: ‘/dev/bus/usb/001/003’
2023-01-21 12:21:19.650 WARNING (MainThread) [bluetooth_auto_recovery.recover] Bluetooth management socket connection lost: [Errno 9] Bad file descriptor
2023-01-21 12:23:14.544 INFO (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (7C:B2:7D:6F:31:FE): Bluetooth scanner has gone quiet for 90s, restarting
2023-01-21 12:23:19.554 WARNING (MainThread) [bluetooth_auto_recovery.recover] Bluetooth adapter hci0 [7C:B2:7D:6F:31:FE] could not be reset due to timeout
2023-01-21 12:23:19.556 INFO (MainThread) [bluetooth_auto_recovery.recover] hci0 permission denied to /dev/bus/usb/001/003 while attempting USB reset: [Errno 13] Permission denied: ‘/dev/bus/usb/001/003’
2023-01-21 12:23:19.556 WARNING (MainThread) [bluetooth_auto_recovery.recover] Bluetooth management socket connection lost: [Errno 9] Bad file descriptor
2023-01-21 12:25:14.547 INFO (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (7C:B2:7D:6F:31:FE): Bluetooth scanner has gone quiet for 90s, restarting
2023-01-21 12:25:14.549 ERROR (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (7C:B2:7D:6F:31:FE): Error stopping scanner: [org.bluez.Error.Failed] No discovery started
2023-01-21 12:25:19.568 WARNING (MainThread) [bluetooth_auto_recovery.recover] Bluetooth adapter hci0 [7C:B2:7D:6F:31:FE] could not be reset due to timeout

Can I just remove the bluetooth components and site packages and run again? I have no BT devices or comms of any kind in HA.

Michel · January 22, 2023, 12:15pm

Don’t know if disabling bluetooth would work, but there might be something else after that.

You mention Ubuntu Studio 18.04, not entirely sure what the “Studio” part is, but if it is running Ubuntu 18.04 your environment might just be too old.

Maybe upgrading to a more recent Ubuntu version or running the devcontainer are better options.

WillS · January 22, 2023, 1:44pm

Good point and thank you again. Checking my Ubuntu Studio version it turns out I had updated it to 20.04 at one point. The update to 22.04 is slightly more complicated, though I believe I’ll need to undertake that as well to keep current with HA environment updates. I just this morning upgraded from 16G to 32G memory, so may return to running Docker. Had planned on using Spyder as my development environment, which is a change from Jupyter notebooks where I’ve been coding in python up to this point. I surmise going the HACS route is the usual route for integration starting points, or for ones that likely won’t be utilized by others, is that essentially accurate?

Michel · January 22, 2023, 7:51pm

Usage is not neccesarily a reason I guess as according to Integrations | Home Assistant Analytics there are almost 200 integrations with <= 10 users (that opted in for analytics)

HACS or even just a HACS compatible repo is a nice way to get started I think, You can just play around and try stuff when you have time. Adding things to Core requires review and that can take a long time (weeks, months)

Personally I keep my custom integrations in a HACS compatible repo and when I think the integration is mature enough and useful to other people I submit it for the HACS default list, so it is a bit easier to find and I think I am going to keep it at that. But that is just me, plenty of other people do submit their custom components to core.

Michel · January 22, 2023, 8:33pm

BTW Python 3.9 support has just been dropped from dev so Python 3.10 is now minimum.

WillS · January 27, 2023, 11:26am

I have updated my laptop to Ubuntu Studio 22.04 with python 3.10.4, and still have the issue with bluetooth causing the hass configuration script to stop. I can’t find this issue in the community posts, so I tried eliminating bluetooth site components and component directories, and removing any lines or defs about bluetooth from various yaml and .py files, though the more I dug, it appears that bluetooth is still in hundreds of locations.

To recap, I went through the full process of starting from scratch on setting up the development environment.

The only change I made was substituting libsvresample for libavresample as the latter was for older os versions, and the documentation said to switch to libsvresample.

Everything went fine until the same place during;

hass -c config

The same sticking point occurred, though without any errors concerning bluetooth;

…
2023-01-26 22:23:20.147 INFO (SyncWorker_3) [homeassistant.loader] Loaded shelly from homeassistant.components.shelly
2023-01-26 22:23:20.147 INFO (SyncWorker_5) [homeassistant.loader] Loaded ruuvi_gateway from homeassistant.components.ruuvi_gateway
2023-01-26 22:23:20.150 INFO (MainThread) [homeassistant.setup] Setting up bluetooth_adapters
2023-01-26 22:23:20.151 INFO (MainThread) [homeassistant.setup] Setup of domain bluetooth_adapters took 0.0 seconds
2023-01-26 22:23:20.230 INFO (SyncWorker_1) [homeassistant.loader] Loaded ipp from homeassistant.components.ipp
2023-01-26 22:23:20.389 INFO (SyncWorker_4) [homeassistant.loader] Loaded dlna_dms from homeassistant.components.dlna_dms
2023-01-26 22:23:20.397 INFO (SyncWorker_5) [homeassistant.loader] Loaded upnp from homeassistant.components.upnp
2023-01-26 22:23:20.426 INFO (SyncWorker_1) [homeassistant.loader] Loaded roku from homeassistant.components.roku
2023-01-26 22:23:30.535 INFO (SyncWorker_3) [homeassistant.loader] Loaded intellifire from homeassistant.components.intellifire

My Dell G5 laptop bluetooth is functioning, and I confirmed by pairing my phone and sending a file to it.

I can’t find any other people having this issue in the forum. Is there any practical way around this? I really need to create an integration for my Darfon solar inverter, as there is no HA integration, and their data is only accessible via a company portal login, which I coded to extract with Selenium.