Starting development - problem with tox

I am struggling with tox testing. Tried to run it on dev branch without any changes and have errors. I am new to python and git development so I would like to check if I understand homeassistant development setup correctly. These are steps I did according how I understood info on web:

  1. Fork homeassistant to my github

  2. Creating venv

     ~ python3 -m vevn hass-dev-venv
    
  3. Clone my forked homeassistant

     ~ cd work/git
     ~ git clone https://github.com/MY_USERNAME/home-assistant.git
    
  4. Activate venv

     ~ cd ../../hass-dev-env
     ~ source bin/activate
    
  5. From active vevn call setup/script

     (hass-dev-venv)~ cd ../work/git/home-assistant/
     (hass-dev-venv)~ script/setup
    
  6. Install tox in venv

     (hass-dev-venv)~ pip3 isntall tox
    
  7. Now when I try to run tox:

     (hass-dev-venv)~ tox -e py35
    
  8. I have following error summary. I would expect that clear dev branch should run without problems so is there anything I am doing wrong?

ERROR: py35: could not install deps [-r/home/USER/work/git/home-assistant/requirements_all.txt, -r/home/USER/work/git/home-assistant/requirements_test.txt]; v = InvocationError(’/home/USER/work/git/home-assistant/.tox/py35/bin/pip install -r/home/USER/work/git/home-assistant/requirements_all.txt -r/home/USER/work/git/home-assistant/requirements_test.txt (see /home/USER/work/git/home-assistant/.tox/py35/log/py35-1.log)’, 1)

Your clone needs to be inside the venv. I guess that something like this should do the job

$ git clone https://github.com/MY_USERNAME/home-assistant.git
$ python3 -m venv home-assistant
$ cd home-assistant
$ source bin/activate
...
2 Likes

Thanks for answer it helped.

I experienced other problems to make tox working so I will put solution that worked for me here for reference.

  1. Missing some dependencies - had to install following

     sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev libjpeg8-dev zlib1g-dev
    
  2. On target device with debian and python 3.4 creating venv doesnt work so I use solution mentioned in http://askubuntu.com/questions/488529/pyvenv-3-4-error-returned-non-zero-exit-status-1

     sudo apt-get install python3.4-venv
     pyvenv-3.4 --without-pip hass-dev-venv
     source hass-dev-venv/bin/activate
     curl https://bootstrap.pypa.io/get-pip.py | python
     deactivate
     source hass-dev-venv/bin/activate

Whoaw, I have been stuck for the whole afternoon before seeing this.

Would it be a good idea to add it to the documentation ?

Hi !

I got the same problem again with your method, can’t test my dev locally :cry:

So here are the steps :

And I got those shiny logs :

    [...]
    running build_ext
    running config
    pkg-config is required for building PyAV
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/pollet/dev/home-assistant/.tox/cov/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/9g/3vtyd8sd2wl0rw5p7dj9sgs00000gn/T/pip-install-do8gwofo/av/setup.py'"'"'; __file__='"'"'/private/var/folders/9g/3vtyd8sd2wl0rw5p7dj9sgs00000gn/T/pip-install-do8gwofo/av/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/9g/3vtyd8sd2wl0rw5p7dj9sgs00000gn/T/pip-record-rd7qozar/install-record.txt --single-version-externally-managed --compile --install-headers /Users/pollet/dev/home-assistant/.tox/cov/bin/../include/site/python3.7/av Check the logs for full command output.

======================================================= log end =======================================================
ERROR: could not install deps [-r/Users/pollet/dev/home-assistant/requirements_test_all.txt, -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt]; v = InvocationError('/Users/pollet/dev/home-assistant/.tox/cov/bin/python -m pip install -r/Users/pollet/dev/home-assistant/requirements_test_all.txt -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt', 1)
_______________________________________________________ summary _______________________________________________________
ERROR:   py36: could not install deps [-r/Users/pollet/dev/home-assistant/requirements_test_all.txt, -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt]; v = InvocationError('/Users/pollet/dev/home-assistant/.tox/py36/bin/python -m pip install -r/Users/pollet/dev/home-assistant/requirements_test_all.txt -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt', 1)
ERROR:   py37: could not install deps [-r/Users/pollet/dev/home-assistant/requirements_test_all.txt, -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt]; v = InvocationError('/Users/pollet/dev/home-assistant/.tox/py37/bin/python -m pip install -r/Users/pollet/dev/home-assistant/requirements_test_all.txt -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt', 1)
ERROR:   py38: could not install deps [-r/Users/pollet/dev/home-assistant/requirements_test_all.txt, -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt]; v = InvocationError('/Users/pollet/dev/home-assistant/.tox/py38/bin/python -m pip install -r/Users/pollet/dev/home-assistant/requirements_test_all.txt -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt', 1)
  lint: commands succeeded
ERROR:   pylint: could not install deps [-r/Users/pollet/dev/home-assistant/requirements_all.txt, -r/Users/pollet/dev/home-assistant/requirements_test.txt, -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt]; v = InvocationError('/Users/pollet/dev/home-assistant/.tox/pylint/bin/python -m pip install -r/Users/pollet/dev/home-assistant/requirements_all.txt -r/Users/pollet/dev/home-assistant/requirements_test.txt -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt', 1)
  typing: commands succeeded
ERROR:   cov: could not install deps [-r/Users/pollet/dev/home-assistant/requirements_test_all.txt, -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt]; v = InvocationError('/Users/pollet/dev/home-assistant/.tox/cov/bin/python -m pip install -r/Users/pollet/dev/home-assistant/requirements_test_all.txt -c/Users/pollet/dev/home-assistant/homeassistant/package_constraints.txt', 1)

Using :

  • macOS 10.14 (almost fresh install)
  • Python 3.7.4
  • pip 19.2.3 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
  • tox 3.9.0 imported from /usr/local/lib/python3.7/site-packages/tox/init.py
  • autoconf (GNU Autoconf) 2.69
  • ffmpeg version 4.2.1

Can someone help me ?

Thanks

Fresh CentOS 7.7 install. Here’s my setup documentation, almost entirely in line with version 0.100.0 documentation:

########################################
#                                      #
#   First-time setup                   #
#                                      #
########################################

# 1. Install development tools
sudo yum groupinstall -y "Development Tools"

# 2. Install Software Collections Repository
sudo yum install -y centos-release-scl

# 3. Install Python 3.6, python36-devel, pip3, mysql-devel and gcc
sudo yum install -y rh-python36 python36-devel python36-pip mysql-devel gcc

# 4. Clone fork of Home Assistant Git repo
mkdir -p ~/git
cd ~/git
git clone https://github.com/NobleKangaroo/home-assistant.git

# 5. Add upstream repo
cd ~/git/home-assistant
git remote add upstream https://github.com/home-assistant/home-assistant.git

# 6. Create and activate virtual environment
cd ~/git/home-assistant
python3 -m venv venv
. ./venv/bin/activate

# 7. Run the setup script
cd ~/git/home-assistant
./script/setup


########################################
#                                      #
#   Get into development mode          #
#                                      #
########################################

# 1. Activate the virtual environment
cd ~/git/home-assistant
. ./venv/bin/activate

# 2. Create a new branch to hold changes
git checkout -b some-feature

# 3. Make some code changes
cd ~/git/home-assistant
vi ...

# 4. Test them with tox and black
cd ~/git/home-assistant
tox
black --verbose --fast ./homeassistant

# 5. Commit changes
cd ~/git/home-assistant
git add .
git commit -m "Add some feature"

# 6. Push changes back to fork
cd ~/git/home-assistant
git push origin HEAD

# 7. Create a new pull request on Github


########################################
#                                      #
#   Catching up with reality           #
#                                      #
########################################

# 1. Fetch and rebase until all changes are resolved
cd ~/git/home-assistant/
git fetch upstream dev  # to pull the latest changes into a local dev branch
git rebase upstream/dev # to put those changes into your feature branch before your changes

# 2. `git status` shows your the file with the conflict; edit the file and resolve the lines between `<<<< | >>>>`

# 3. Add the modified file: `git add <file>` or `git add .`

# 4. Continue rebase: `git rebase --continue`

# 5. Repeat until all conflicts are resolved

# 6. After rebasing branch, get the fork up to date with local branch
# run this from feature branch
git push origin --force

# 7. Then run `git pull --rebase upstream dev`

Attempting to run tox, I’m presented with a whole gamut of errors, including dependencies that couldn’t be installed, commands that are missing (dot, for example), gcc failing, and tox being generally unhappy about the process.

I’m fairly familiar with Git and would love to get some commits pushed out, but this tox stuff is driving me nuts! I’m still looking into it; might have to just install additional packages to fill the gaps. But advice would be truly appreciated.
 

- NobleKangaroo

Ended up switching over to a fresh Debian Buster install, followed these steps and got through tox and black.

########################################
#                                      #
#   First-time setup                   #
#                                      #
########################################

# 1. Install  dependencies
sudo apt-get -y install python3-pip python3-dev python3-venv
sudo apt-get -y install autoconf libssl-dev libxml2-dev libxslt1-dev libjpeg-dev libffi-dev libudev-dev zlib1g-dev pkg-config
sudo apt-get -y install libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libavresample-dev libavfilter-dev

# 2. Install Git
sudo apt-get -y install git

# 3. Clone fork of Home Assistant Git repo
mkdir -p ~/git && cd ~/git
git clone https://github.com/NobleKangaroo/home-assistant.git

# 4. Add upstream repo
cd ~/git/home-assistant
git remote add upstream https://github.com/home-assistant/home-assistant.git

# 5. Create and activate virtual environment
cd ~/git/home-assistant
python3 -m venv venv
. ./venv/bin/activate

# 6. Install wheel, flake8 and pylint
pip3 install wheel flake8 pylint

# 7. Run the setup script
cd ~/git/home-assistant
./script/setup


########################################
#                                      #
#   Get into development mode          #
#                                      #
########################################

# 1. Activate the virtual environment
cd ~/git/home-assistant
. ./venv/bin/activate

# 2. Create a new branch to hold changes
git checkout -b some-feature

# 3. Make some code changes
cd ~/git/home-assistant
vi ...

# 3. Re-format with black
cd ~/git/home-assistant
black --verbose --fast ./homeassistant

# 4. Test with flake8 and pylint
cd ~/git/home-assistant
flake8 [file]
pylint [file]

# 5. Test with tox
cd ~/git/home-assistant
tox [file]

# 6. Commit changes
cd ~/git/home-assistant
git add .
git commit -m "Add some feature"

# 7. Push changes back to fork
cd ~/git/home-assistant
git push origin HEAD

# 8. Create a new pull request on Github


########################################
#                                      #
#   Catching up with reality           #
#                                      #
########################################

# 1. Fetch and rebase until all changes are resolved
cd ~/git/home-assistant/
git fetch upstream dev  # to pull the latest changes into a local dev branch
git rebase upstream/dev # to put those changes into your feature branch before your changes

# 2. `git status` shows your the file with the conflict; edit the file and resolve the lines between `<<<< | >>>>`

# 3. Add the modified file: `git add <file>` or `git add .`

# 4. Continue rebase: `git rebase --continue`

# 5. Repeat until all conflicts are resolved

# 6. After rebasing branch, get the fork up to date with local branch
# run this from feature branch
git push origin --force

# 7. Then run `git pull --rebase upstream dev`

Actually, I’m testing outside of Tox now and it’s working pretty well.

The downside is that you should pip install [module_name] on your machine.

trying to follow guidelines here:https://developers.home-assistant.io/docs/en/development_guidelines.html

thing is I cant even use pip. how then to proceed with https://github.com/psf/black

$ pip install black
-bash: pip: command not found

now what…?

Are you on Mac OS ?

Use pip3 :blush:

Actually, what I’ve said earlier is a bit false :

You CAN pip install [module_name] on the venv, whithout pip3 :stuck_out_tongue:

Then you can test your component using those kinds of commands (ones I use often) :

pylint homeassistant/components/icloud/
pylint tests/components/icloud/
py.test tests/components/icloud/[specific_optional_file.py]
isort homeassistant/components/icloud/*.py (optional, will be triggered at pre-commit, since 12/12/2019 https://github.com/home-assistant/home-assistant/pull/29739)
isort tests/components/icloud/*.py (optional, will be triggered at pre-commit, since 12/12/2019 https://github.com/home-assistant/home-assistant/pull/29739)
black homeassistant (optional, will be triggered at pre-commit)
black tests (optional, will be triggered at pre-commit)

I also have problems with tox.
I use macos, I set up my environment,

$ brew install python3 autoconf
$ brew install ffmpeg
$ git clone https://github.com/YOUR_GIT_USERNAME/core.git
$ cd core
$ git remote add upstream https://github.com/home-assistant/core.git
$ python3.7 -m venv venv
$ source venv/bin/activate
$ script/setup
$ hass #let this run a while
$ ctrl c
$ git checkout -b some-feature
# I didn't make any changes
$ tox

Then I get tox errors
This are the first lines

(venv) iMac-5k:core dev$ tox
/Users/dev/Dev/HA-Dev/core/venv/lib/python3.7/site-packages/tox/config/__init__.py:595: UserWarning: conflicting basepython version (set 3.7, should be 3.8) for env 'py38';resolve conflict or set ignore_basepython_conflict
  proposed_version, implied_version, testenv_config.envname
/Users/dev/Dev/HA-Dev/core/venv/lib/python3.7/site-packages/tox/config/__init__.py:595: UserWarning: conflicting basepython version (set 3.7, should be 3.6) for env 'py36';resolve conflict or set ignore_basepython_conflict
  proposed_version, implied_version, testenv_config.envname
  py36: commands succeeded
ERROR:   py37: commands failed
  py38: commands succeeded
  lint: commands succeeded
ERROR:   pylint: could not install deps [-r/Users/dev/Dev/HA-Dev/core/requirements_all.txt, -r/Users/dev/Dev/HA-Dev/core/requirements_test.txt, -c/Users/dev/Dev/HA-Dev/core/homeassistant/package_constraints.txt]; v = InvocationError('/Users/dev/Dev/HA-Dev/core/.tox/pylint/bin/python -m pip install -r/Users/dev/Dev/HA-Dev/core/requirements_all.txt -r/Users/dev/Dev/HA-Dev/core/requirements_test.txt -c/Users/dev/Dev/HA-Dev/core/homeassistant/package_constraints.txt', 1)
  typing: commands succeeded
ERROR:   cov: commands failed

Then

(venv) iMac-5k:core dev$ pip install tox-venv
(venv) iMac-5k:core dev$ tox -r
/Users/dev/Dev/HA-Dev/core/venv/lib/python3.7/site-packages/tox/config/__init__.py:595: UserWarning: conflicting basepython version (set 3.7, should be 3.8) for env 'py38';resolve conflict or set ignore_basepython_conflict
  proposed_version, implied_version, testenv_config.envname
/Users/dev/Dev/HA-Dev/core/venv/lib/python3.7/site-packages/tox/config/__init__.py:595: UserWarning: conflicting basepython version (set 3.7, should be 3.6) for env 'py36';resolve conflict or set ignore_basepython_conflict
  proposed_version, implied_version, testenv_config.envname
ERROR:   py36: commands failed
  py37: commands succeeded
ERROR:   py38: commands failed
  lint: commands succeeded
ERROR:   pylint: could not install deps [-r/Users/dev/Dev/HA-Dev/core/requirements_all.txt, -r/Users/dev/Dev/HA-Dev/core/requirements_test.txt, -c/Users/dev/Dev/HA-Dev/core/homeassistant/package_constraints.txt]; v = InvocationError('/Users/dev/Dev/HA-Dev/core/.tox/pylint/bin/python -m pip install -r/Users/dev/Dev/HA-Dev/core/requirements_all.txt -r/Users/dev/Dev/HA-Dev/core/requirements_test.txt -c/Users/dev/Dev/HA-Dev/core/homeassistant/package_constraints.txt', 1)
ERROR:   typing: commands failed
  cov: commands succeeded

So I don’t get it, I didn’t make any changes to the code and testing will fail, how can I make changes and let testing succeed?

Dealing with tox, after some error solved by following this thread, now I get:

ERROR: Could not build wheels for DTLSSocket which use PEP 517 and cannot be installed directly

any idea?

I also have the same kind of problems, I have follow step by step the developer instructions without any success. Is something missing in the documentations or why is that many of us are having the same kind of errors with tox?

Hi, I’m new here, I have also tried to no avail. to setup a devel environment and run tox without errors :

  1. I followed TO THE LETTER the instructions to run devcontainer : HA starts on CLEAN DEV branch. but I can’t get tox to work

  2. out of idea, I started my wsl2 ubuntu 20.04 :

  • I’m on CLEAN DEV git clone, no modification AT ALL
  • I installed dependencies I could find here and there
  • I ran script/setup and venv/bin/activate.

and yet … I still get lots of failing checks by running “tox -e py38”. 7 red “F” that I suppose are failled checks.

this is ridiculous ! has ANYONE succesfully run tox locally on the dev branch ?!? I’m trying to fix a bug I reported, but reaaaaally, that is discouraging and frustrating … we need clear steps to follow ! If even dev branch does not pass tests, how are WE supposed to get them passed, before doing a P.R. ???

here are some extracts from the failures :

the failures, on all the test envs I have tried (python 3.8) :

FAILED tests/components/homekit/test_type_cameras.py::test_camera_stream_source_configured[pyloop] - KeyError: UUID('3303d503-17cc-469a-b672-92436a71a2f6')
FAILED tests/components/homekit/test_type_cameras.py::test_camera_stream_source_found[pyloop] - KeyError: UUID('3303d503-17cc-469a-b672-92436a71a2f6')
FAILED tests/components/homekit/test_type_cameras.py::test_camera_stream_source_configured_and_copy_codec[pyloop] - KeyError: UUID('3303d503-17cc-469a-b672-92436a71a2f6')
FAILED tests/components/stream/test_recorder.py::test_record_stream_audio[pyloop] - Failed: Timeout >9.0s
FAILED tests/components/telegram/test_notify.py::test_reload_notify[pyloop] - ModuleNotFoundError: No module named 'telegram'
FAILED tests/util/test_package.py::test_check_package_version_does_not_match - AssertionError: assert not True
ERROR tests/components/shopping_list/test_intent.py::test_recent_items_intent[pyloop] - RuntimeError: Event loop is closed
self = <accessory display_name='Camera' services=['AccessoryInformation', 'Microphone', 'CameraRTPStreamManagement', 'CameraRTPStreamManagement', 'CameraRTPStreamManagement']>
session_id = UUID('3303d503-17cc-469a-b672-92436a71a2f6')

    @callback
    def _async_stop_ffmpeg_watch(self, session_id):
        """Cleanup a streaming session after stopping."""
>       if FFMPEG_WATCHER not in self.sessions[session_id]:
E       KeyError: UUID('3303d503-17cc-469a-b672-92436a71a2f6')

homeassistant/components/homekit/type_cameras.py:436: KeyError

multiples ffmpeg errors on “stream unexpected stop” or keyerror

DEBUG:homeassistant.components.homekit.type_cameras:FFmpeg output settings: -map 0:v:0 -an -c:v libx264 -profile:v high -tune zerolatency -pix_fmt yuv420p -r 30 -b:v 299k -bufsize 1196k -maxrate 299k -payload_type 99 -ssrc 1334673 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params zdPmNLWeI86DtLJHvVLI6YPvqhVeeiLsNtrAgbgL srtp://192.168.208.5:51246?rtcpport=51246&localrtcpport=51246&pkt_size=1316 -map 0:a:0 -vn -c:a libopus -application lowdelay -ac 1 -ar 24k -b:a 24k -bufsize 96k -payload_type 110 -ssrc 8646279 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params shnETgfD+7xUQ8zRdsaytY11wu6CO73IJ+RZVJpU srtp://192.168.208.5:51108?rtcpport=51108&localrtcpport=51108&pkt_size=188
INFO:homeassistant.components.homekit.type_cameras:[3303d503-17cc-469a-b672-92436a71a2f6] Started stream process - PID 1
WARNING:homeassistant.components.homekit.type_cameras:Streaming process ended unexpectedly - PID 1
ERROR:pyhap.camera:[3303d503-17cc-469a-b672-92436a71a2f6] Failed to start/reconfigure stream, deleting session.

a telegram bot error :

_______________________________________________________________________________________ test_reload_notify[pyloop] ________________________________________________________________________________________
[gw0] linux -- Python 3.8.5 /home/arnaud/core/.tox/py38/bin/python

thing = <module 'homeassistant.components' from '/home/arnaud/core/homeassistant/components/__init__.py'>, comp = 'telegram_bot', import_path = 'homeassistant.components.telegram_bot'

    def _dot_lookup(thing, comp, import_path):
        try:
>           return getattr(thing, comp)
E           AttributeError: module 'homeassistant.components' has no attribute 'telegram_bot'

/usr/lib/python3.8/unittest/mock.py:1215: AttributeError

During handling of the above exception, another exception occurred:

hass = <homeassistant.core.HomeAssistant object at 0x7fbca5b31ee0>

>   ???

a mysterious record stream timeout :

____________________________________________________________________________________ test_record_stream_audio[pyloop] _____________________________________________________________________________________
[gw6] linux -- Python 3.8.5 /home/arnaud/core/.tox/py38/bin/python

hass = <homeassistant.core.HomeAssistant object at 0x7fa1dd4140a0>, hass_client = <function hass_client.<locals>.auth_client at 0x7fa1dd3caee0>
stream_worker_sync = <tests.components.stream.conftest.WorkerSync object at 0x7fa1dd0f8160>, record_worker_sync = <tests.components.stream.test_recorder.SaveRecordWorkerSync object at 0x7fa1dd6e94f0>

>   ???

/workspaces/core/tests/components/stream/test_recorder.py:235:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

container_format = 'mov', audio_codec = None

    def generate_h264_video(container_format="mp4", audio_codec=None):
        """
        Generate a test video.

        See: http://docs.mikeboers.com/pyav/develop/cookbook/numpy.html
        """
.....

            frame = av.VideoFrame.from_ndarray(img, format="rgb24")
>           for packet in stream.encode(frame):
E           Failed: Timeout >9.0s

tests/components/stream/common.py:71: Failed

some cryptic “deprecated default configuration check” :

______________________________________________________________________________________ test_deprecated_with_default _______________________________________________________________________________________
[gw2] linux -- Python 3.8.5 /home/arnaud/core/.tox/py38/bin/python

caplog = <_pytest.logging.LogCaptureFixture object at 0x7f82b85f8be0>
schema = <Schema({'venus': <function boolean at 0x7f8312eb3670>, 'mars': <function boolean at 0x7f8312eb3670>, 'jupiter': <function boolean at 0x7f8312eb3670>}, extra=PREVENT_EXTRA, required=False) object at 0x7f82b85f8790>

>   ???
E   AssertionError: assert 'homeassistan...ig_validation' == 'tests.helper...ig_validation'
E     - tests.helpers.test_config_validation
E     ? ^   ^         -----
E     + homeassistant.helpers.config_validation
E     ? ^^^ ++++  ^^^

/workspaces/core/tests/helpers/test_config_validation.py:720: AssertionError
------------------------------------------------------------------------------------------ Captured stderr call -------------------------------------------------------------------------------------------
WARNING:homeassistant.helpers.config_validation:The 'mars' option is deprecated, please remove it from your configuration
-------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------
WARNING  homeassistant.helpers.config_validation:config_validation.py:747 The 'mars' option is deprecated, please remove it from your configuration
________________________________________________________________________________ test_check_package_version_does_not_match ________________________________________________________________________________
[gw1] linux -- Python 3.8.5 /home/arnaud/core/.tox/py38/bin/python

>   ???
E   AssertionError: assert not True
E    +  where True = <function is_installed at 0x7f1de8937040>('homeassistant>=999.999.999')
E    +    where <function is_installed at 0x7f1de8937040> = package.is_installed

/workspaces/core/tests/util/test_package.py:257: AssertionError

I’m about to giving up :frowning:

This warning seems to come from homeassistant/components/homekit/type_cameras.py

That function checks whether ffmpeg is still running using this function in homeassistant/components/homekit/util.py:

During testing, the ffmpeg process is replaced by unittest.mock.MagicMock. The problem seems to be this:

+ >>> import os
+ >>> import unittest.mock
+ >>> int(unittest.mock.MagicMock())
1
+ >>> os.kill(unittest.mock.MagicMock(), 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 1] Operation not permitted
+ >>> os.kill(1, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 1] Operation not permitted

It checks whether a process is running by trying to kill it with os.kill. However, PID1 is the init system and thus cannot be killed by a normal user. When the python process is running as root (which might be the case in CI on Github), everything works:

$ sudo python3
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.kill(1,0)
>>> 

With this workaround, the camera-related tests pass:

def pid_is_alive(pid) -> bool:
    """Check to see if a process is alive."""
    if str(type(pid)) == "<class 'unittest.mock.MagicMock'>":
        return True
    try:
        os.kill(pid, 0)
        return True
    except OSError:
        pass
    return False

Maybe I should report this issue on Github.

I am having problems with other tests, too. I even found some tests that would fail during the batch testing, but pass when I run them individually with pytest.
A really strange example is this:

(venv) $ pytest -n auto -p no:sugar tests/util/test_package.py
============================================================================= test session starts ==============================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/ondra/source/repos/home-assistant-core, configfile: pyproject.toml
plugins: xdist-2.2.1, rerunfailures-10.1, cov-2.12.1, timeout-1.4.2, test-groups-1.0.3, requests-mock-1.9.2, mock-3.6.1, forked-1.3.0, aiohttp-0.3.0
gw0 [13] / gw1 [13]
............F                                                                                                                                                            [100%]
=================================================================================== FAILURES ===================================================================================
__________________________________________________________________ test_check_package_version_does_not_match ___________________________________________________________________
[gw0] linux -- Python 3.8.10 /home/ondra/source/repos/home-assistant-core/venv/bin/python3

    def test_check_package_version_does_not_match():
        """Test for version mismatch."""
        installed_package = list(pkg_resources.working_set)[0].project_name
        assert not package.is_installed(f"{installed_package}==999.999.999")
>       assert not package.is_installed(f"{installed_package}>=999.999.999")
E       AssertionError: assert not True
E        +  where True = <function is_installed at 0x7f386d3888b0>('homeassistant>=999.999.999')
E        +    where <function is_installed at 0x7f386d3888b0> = package.is_installed

tests/util/test_package.py:258: AssertionError
=========================================================================== short test summary info ============================================================================
FAILED tests/util/test_package.py::test_check_package_version_does_not_match - AssertionError: assert not True
========================================================================= 1 failed, 12 passed in 2.17s =========================================================================
(venv) $ pytest -p no:sugar tests/util/test_package.py 
============================================================================= test session starts ==============================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/ondra/source/repos/home-assistant-core, configfile: pyproject.toml
plugins: xdist-2.2.1, rerunfailures-10.1, cov-2.12.1, timeout-1.4.2, test-groups-1.0.3, requests-mock-1.9.2, mock-3.6.1, forked-1.3.0, aiohttp-0.3.0
collected 13 items                                                                                                                                                             

tests/util/test_package.py .............                                                                                                                                 [100%]

============================================================================== 13 passed in 0.15s ==============================================================================

It only fails if I run pytest with multiple processes (-n auto).

Is there anyone who has solved these issues?