Can't test my new sensor using Pylint outside of Tox - import-error

gregtd · July 2, 2019, 4:27pm

I have setup a development system on Ubuntu under Windows 10. I read the docs and followed the steps. My new component is under: homeassistant/components/rainforest and it contains:

init.py
manifest.json
sensor.py

I ran the ‘python3 -m script.gen_requirements_all’ which added:

homeassistant.components.rainforest_eagle
eagle200_reader==0.1.4

To the ‘requirements_all.txt’ file.

I updated the ‘.coveragerc’ file to exclude my platform as I didn’t create any tests and as well it using an external library to communicate with a device.

homeassistant/components/rainforest/sensor.py

When I run the ‘tox’ command when it gets to pylint I see the external eagle200_reader library in the output.

pylint installdeps: -r/home/home/home-assistant/requirements_all.txt, -r/home/home/home-assistant/requirements_test.txt, -c/home/home/home-assistant/homeassistant/package_constraints.txt
....
,eagle200-reader==0.1.4,
.....

The summary of the tox command has a few errors but none are for an import

***** ERROR
Tests are leaving files behind. Please update the tests to avoid writing any files:
homeassistant/components/rainforest_eagle/__init__.py
homeassistant/components/rainforest_eagle/manifest.json
homeassistant/components/rainforest_eagle/sensor.py

ERROR: InvocationError for command /home/home/home-assistant/script/check_dirty (exited with code 1)
_________________________________________________ summary __________________________________________________
ERROR:   py35: commands failed
ERROR:   py36: commands failed
ERROR:   py37: commands failed
ERROR:   py38: commands failed
  lint: commands succeeded
  pylint: commands succeeded
ERROR:   typing: commands failed
ERROR:   cov: commands failed
(venv) home@DiningRoomPC:~/home-assistant$

When I run pylint outside of tox I get the import error. I ran the pip3 install as how it says in the online docs

pip3 install -r requirements_test_all.txt -c homeassistant/package_constraints.txt

But the output does not list it getting the eagle200_reader library. Is this because I’m using the requirements_test_all.txt and not the requirements_all.txt file?

pylint homeassistant/components/rainforest_eagle/sensor.py
************* Module homeassistant.components.rainforest_eagle.sensor
homeassistant/components/rainforest_eagle/sensor.py:83:8: E0401: Unable to import 'eagle200_reader' (import-error)

------------------------------------------------------------------
Your code has been rated at 8.65/10 (previous run: 8.65/10, +0.00)

Any pointers would be helpful! As well the sensor code I’m using is not yet posted to HA github repository, but I am copying it from my personal repository https://github.com/gtdiehl/rainforest_eagle if you want to see the 3 files I mentioned earlier.

Again thank you for any help
Greg

pnbruckner · July 2, 2019, 4:37pm

What do you get if you run:

git status

Also, have you tried:

script/lint

gregtd · July 2, 2019, 6:04pm

@pnbruckner

$ git status
On branch rainforest-eagle
Changes not staged for commit:
 (use "git add <file>..." to update what will be committed)
 (use "git checkout -- <file>..." to discard changes in working directory)

       modified:   .coveragerc
       modified:   CODEOWNERS
       modified:   requirements_all.txt

Untracked files:
 (use "git add <file>..." to include in what will be committed)

       hbmqtt_pub pip3.6 rflinkproxy tox-quickstart
       homeassistant/components/rainforest_eagle/
       pbr
       pbr python3 tls.py
       t rst2html.py uvc
       t2html4.py vapid

no changes added to commit (use "git add" and/or "git commit -a")

I tried script/lint but it says no files have changed

$ script/lint
fatal: Not a valid object name upstream/dev
=================================================
=                FILES CHANGED                  =
=================================================
No python file changed. Rather use: tox -e lint

(venv) home@DiningRoomPC:~/home-assistant$ script/lint homeassistant/components/rainforest_eagle/sensor.py
fatal: Not a valid object name upstream/dev
=================================================
=                FILES CHANGED                  =
=================================================
No python file changed. Rather use: tox -e lint

Here is the output of my git upstream

$ git remote show upstream
* remote upstream
  Fetch URL: https://github.com/home-assistant/home-assistant.git
  Push  URL: https://github.com/home-assistant/home-assistant.git
  HEAD branch: dev
  Remote branches:
    add-tesla-components-to-entity-registry new (next fetch will store in remotes/upstream)
    awarecan-patch-1                        new (next fetch will store in remotes/upstream)
    awarecan-patch-2                        new (next fetch will store in remotes/upstream)
    climate-1.0                             new (next fetch will store in remotes/upstream)
    climate-1.0_ambiclimate                 new (next fetch will store in remotes/upstream)
    dev                                     new (next fetch will store in remotes/upstream)
    disable-auto-generated-groups           new (next fetch will store in remotes/upstream)
    fix-tahoma-horiz-awning                 new (next fetch will store in remotes/upstream)
    frenck-2019-0045                        new (next fetch will store in remotes/upstream)
    han_solo                                new (next fetch will store in remotes/upstream)
    history-opti                            new (next fetch will store in remotes/upstream)
    master                                  new (next fetch will store in remotes/upstream)
    mobile_app-async-entry-events           new (next fetch will store in remotes/upstream)
    mobile_app-delete-and-unload            new (next fetch will store in remotes/upstream)
    mobile_app-reorganize-integration       new (next fetch will store in remotes/upstream)
    nest-account-link                       new (next fetch will store in remotes/upstream)
    rc                                      new (next fetch will store in remotes/upstream)
    remove-elevation                        new (next fetch will store in remotes/upstream)
    traceback-iqvia                         new (next fetch will store in remotes/upstream)
  Local ref configured for 'git push':
    dev pushes to dev (local out of date)

pnbruckner · July 2, 2019, 6:09pm

I think you need to use git add to add the new files to the index, then try script/lint again. E.g.,

git add -- homeassistant/components/rainforest_eagle/*
script/lint

gregtd · July 2, 2019, 6:30pm

@pnbruckner
I added the new files, but script/lint still says no changed files even though git status now shows the 3 files to be committed.

$ script/lint
fatal: Not a valid object name upstream/dev
=================================================
=                FILES CHANGED                  =
=================================================
No python file changed. Rather use: tox -e lint

(venv) home@DiningRoomPC:~/home-assistant$ git status
On branch rainforest-eagle
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        new file:   homeassistant/components/rainforest_eagle/__init__.py
        new file:   homeassistant/components/rainforest_eagle/manifest.json
        new file:   homeassistant/components/rainforest_eagle/sensor.py

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   .coveragerc
        modified:   CODEOWNERS
        modified:   requirements_all.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        hbmqtt_pub pip3.6 rflinkproxy tox-quickstart
        pbr
        pbr python3 tls.py
        t rst2html.py uvc
        t2html4.py vapid

I’m wondering if it is due to the fatal failure from script/lint?

$ script/lint
fatal: Not a valid object name upstream/dev
...

pnbruckner · July 2, 2019, 6:32pm

What does git branch -a -vv show?

gregtd · July 2, 2019, 6:36pm

@pnbruckner

$ git branch -a -vv
  dev                                                    23dd644f4 [origin/dev] Update IDs for rename node/value (#24646)
* rainforest-eagle                                       23dd644f4 Update IDs for rename node/value (#24646)
  remotes/origin/HEAD                                    -> origin/dev
  remotes/origin/add-tesla-components-to-entity-registry 3b6594491 THIS ONE TOO
  remotes/origin/awarecan-patch-1                        3741e79de Remove validate ftom
  remotes/origin/awarecan-patch-2                        b2db0aff8 Frontend should depend on config
  remotes/origin/climate-1.0                             f7c2997fd Fix PRESET can be None
  remotes/origin/dev                                     23dd644f4 Update IDs for rename node/value (#24646)
  remotes/origin/disable-auto-generated-groups           c84459334 Remove passing in entity_added_callback
  remotes/origin/fix-tahoma-horiz-awning                 b3b09dabd Do not reverse open/close calls
  remotes/origin/frenck-2019-0045                        fb40b6bac :shirt: Fixes too many blank lines
  remotes/origin/han_solo                                59e54a069 han solo
  remotes/origin/history-opti                            bab6a8c4b Make single entity query easier
  remotes/origin/master                                  9cf43dd8f Merge pull request #24839 from home-assistant/rc
  remotes/origin/mobile_app-async-entry-events           93cdfad2a Initial support for async_remove_entry and async_unload_entry in mobile_app
  remotes/origin/mobile_app-delete-and-unload            66271d4ab Add new webhook command to delete the registration
  remotes/origin/mobile_app-reorganize-integration       2d9afbe25 Rename the entity variable to entity_config
  remotes/origin/nest-account-link                       5efbe119f WIP
  remotes/origin/rc                                      21c2e8da6 Bumped version to 0.95.4
  remotes/origin/remove-elevation                        26b718b15 Remove calling of elevation
  remotes/origin/traceback-iqvia                         2c0cf1abd Suppress traceback

It appears it’s pointing to origin/dev rather than upstream/dev? Do I have to change it? How would I do that?

pnbruckner · July 2, 2019, 6:44pm

Looks like you don’t have the upstream remote set. What does this show:

git remote -v

It should show both your github fork as origin, and the HA github repo as upstream. E.g., mine shows:

origin  [email protected]:pnbruckner/home-assistant.git (fetch)
origin  [email protected]:pnbruckner/home-assistant.git (push)
upstream        https://github.com/home-assistant/home-assistant.git (fetch)
upstream        https://github.com/home-assistant/home-assistant.git (push)

If not, then you just need to add the upstream remote. (You must have missed that step.)

git remote add upstream https://github.com/home-assistant/home-assistant.git

gregtd · July 2, 2019, 6:53pm

The command shows the upstream

$ git remote -v
origin  https://github.com/gtdiehl/home-assistant.git (fetch)
origin  https://github.com/gtdiehl/home-assistant.git (push)
upstream        https://github.com/home-assistant/home-assistant.git (fetch)
upstream        https://github.com/home-assistant/home-assistant.git (push)

I tried to re-run the git remote add command just to see what would happen, and it failed because it said it already existed

$ git remote add upstream https://github.com/home-assistant/home-assistant.git
fatal: remote upstream already exists.

The origin is using a slightly different path than yours but it is still pointing to my fork, really odd

I might start over, but running tox initially takes like 3 hours!

pnbruckner · July 2, 2019, 7:07pm

Hmm, ok.

Did you run?

script/setup

Also, if you just want to do the pylint step, you can do:

tox -r -e pylint

(You don’t need the -r switch if you’ve already used it and you haven’t changed the requirements files.)

gregtd · July 2, 2019, 7:28pm

Yeah I ran script/setup command when I initially setup the environment. Do I have to run it each time I activate the virtual environment?

When I run tox -e pylint I get a success message but really no indication if my new sensor was tested. I only see that the external library eagle200_reader is listed in the output with all the other modules and version numbers. That’s why I ran pylint homeassistant/components/rainforest_eagle/sensor.py outside of tox to know I was testing my sensor, as well I thought it would be much quicker to run against one file rather than the whole repository of components.

$ tox -e pylint
/home/home/home-assistant/venv/lib/python3.6/site-packages/tox/config/__init__.py:578: UserWarning: conflicting basepython version (set 36, should be 35) for env 'py35';resolve conflict or set ignore_basepython_conflict
  proposed_version, implied_version, testenv_config.envname
/home/home/home-assistant/venv/lib/python3.6/site-packages/tox/config/__init__.py:578: UserWarning: conflicting basepython version (set 36, should be 37) for env 'py37';resolve conflict or set ignore_basepython_conflict
  proposed_version, implied_version, testenv_config.envname
/home/home/home-assistant/venv/lib/python3.6/site-packages/tox/config/__init__.py:578: UserWarning: conflicting basepython version (set 36, should be 38) for env 'py38';resolve conflict or set ignore_basepython_conflict
  proposed_version, implied_version, testenv_config.envname
GLOB sdist-make: /home/home/home-assistant/setup.py
pylint inst-nodeps: /home/home/home-assistant/.tox/.tmp/package/1/homeassistant-0.96.0.dev0.zip
pylint installed: <snip> ,eagle200-reader==0.1.4, <snip>
pylint run-test-pre: PYTHONHASHSEED='1640573658'
pylint run-test: commands[0] | pylint homeassistant
 
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

_______________________________________________________ summary ________________________________________________________
  pylint: commands succeeded
  congratulations :)

I appreciate your quick responses!

Do you think the above output tested my sensor?

pnbruckner · July 2, 2019, 7:31pm

I guess the easiest way to find out is to break it. Just a an extra blank line between functions or something. That should be enough for pylint to complain (if it’s actually parsing your files.)

pnbruckner · July 2, 2019, 7:34pm

Regarding script/setup, this is the habit I’ve gotten into, right or wrong…

When I do a checkout that might be “significantly” different, or I’ve updated requirements, etc., I do this:

git clean -fdx -e venv
script/setup
tox -r
pip3 install -r requirements_test_all.txt -c homeassistant/package_constraints.txt

This assumes the virtual environment is in a sub-folder named venv (which it seems yours is, too.)

gregtd · July 2, 2019, 7:40pm

@pnbruckner
Thank you. I think I’ll try and break the sensor.py first. Then try the other commands, if I still have problems.

pnbruckner · July 2, 2019, 7:49pm

Actually, of those last two steps, I don’t always do them both, immediately. E.g., I’ll only do the last one if I want to test outside of tox. And for the tox -r option, I tend to use it with -e. E.g., tox -r -e py36, or tox -r -e pylint. Sometimes I’ll run all the tests with it (which is what tox -r does), but not often.

gregtd · July 3, 2019, 12:44am

@pnbruckner
Again thank you for all your help! I don’t know what the problem was, but after I started over my development environment is now able to run pylint outside of tox against my sensor code

(venv) home@DiningRoomPC:~/home-assistant$ pylint homeassistant/components/rainforest_eagle/__init__.py
(venv) home@DiningRoomPC:~/home-assistant$ pylint homeassistant/components/rainforest_eagle/sensor.py

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 8.65/10, +1.35)