Announcement: AppDaemon 3.0 beta 2

aimc · January 27, 2018, 7:54pm

Hello all - after some good feedback and a little work, I bring you the second beta of AppDaemon 3.0.

If you are upgrading to beta2 directly from a 2.x release please review the breaking changes documented here.

Not a lot has changed, this has mostly been cleanup and bugfixes. There are a few minor breaking changes, but nothing earth shattering.

Give it a go and let me know what you think!

3.0.0b2

Features

Make int args in appdaemon.yaml a little more robust
Improve handling for missing app files
Module loading enhancements
Moved from requests to aiohttp client for better async behavior
Added thread monitoring for worker threads
Give more informative error message if AppDaemon can’t locate a valid config dir

Fixes

Fixed a bug that could cause multiple apps.yaml changes or additions to be ignored
Fixed a bug causing listen_state() callbacks with duration set to fire immediately
Pinned yarl library to fix an issue with Docker build
Fixed a couple of potential event loop hold ups
Fixed a bug in password security for HADashboard service and state calls
Changes to apps.yaml now also force a reload of dependent modules
exclude_dirs now applies to yaml files as well as python files
Fixed broken icon on HADashboard logon screen
Fixed a bug preventing the media title from showing in the media player

Breaking Changes

App modules not listed in an apps.yaml file will no longer be loaded. Python modules may still be imported directly if they are in a directory in which other apps reside.
cert_path is deprecated. With the replacement of requests with aiohttp, it is now sufficient to set cert_verify to False to use a self signed certificate.
Initial dashboard loads may be slower on less powerful hardware when using password authentication. Updating after the initial load is unaffected.

frenck · January 27, 2018, 7:55pm

Nice timing… was just finishing the Hass.io add-on for beta 1

aimc · January 27, 2018, 7:55pm

LOL - now you can go straight to b2 and benefit from the fixes

frenck · January 27, 2018, 7:55pm

I will

carbuthn · January 27, 2018, 8:11pm

@aimc and @frenck thanks for all of the hard work, it is appreciated. Looking forward to using the new version.

Seth_Daemen · January 28, 2018, 6:19pm

appdaemon.yaml file is not filling correctly in docker.

Then I get error getting state check java console.
Then when I check the java console I see it goes to http://192.168.1.12:5050/state/light.woonkamer_lamp_level I think the link has to be port number 8123

aimc · January 28, 2018, 6:22pm

I’m not sure what you mean by this?

No, the port on the URL should be the port for HADashboard, not HASS.

Seth_Daemen · January 28, 2018, 6:35pm

I downloaded now the latest dev image and It is working after clearing browser cache

frenck · January 28, 2018, 7:42pm

The Hassio add-on for AppDaemon3 is done, nevertheless, I ran into an issue with the Hass.io Supervisor causing issues with the v3 version of AppDaemon.

The issue has been pinpointed and the fix will be released soon. Afterward, the add-on can be put out to the public.

aimc · January 28, 2018, 7:43pm

Thanks for your hard work on this @frenk!

keithh666 · January 28, 2018, 11:01pm

Hi @aimc,

Trying to install the beta 2 of appdaemon gives the following on win 10…

C:\WINDOWS\system32>pip3 install --upgrade --pre appdaemon
Collecting appdaemon
  Downloading appdaemon-3.0.0b2-py3-none-any.whl (5.9MB)
    100% |████████████████████████████████| 5.9MB 206kB/s
Requirement already up-to-date: aiohttp==2.3.7 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: voluptuous in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: websocket-client in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: daemonize in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: Jinja2>=2.9.5 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: aiohttp-jinja2 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: iso8601 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: yarl<1.0,>=0.11 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: bcrypt in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: pyyaml in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: requests>=2.6.0 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: astral in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: configparser in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: sseclient in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: feedparser in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Requirement already up-to-date: async in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from appdaemon)
Collecting multidict>=3.0.0 (from aiohttp==2.3.7->appdaemon)
  Downloading multidict-4.1.0-cp36-cp36m-win_amd64.whl (194kB)
    100% |████████████████████████████████| 194kB 2.2MB/s
Requirement already up-to-date: async-timeout>=1.2.0 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from aiohttp==2.3.7->appdaemon)
Requirement already up-to-date: chardet in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from aiohttp==2.3.7->appdaemon)
Requirement already up-to-date: six in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from websocket-client->appdaemon)
Requirement already up-to-date: MarkupSafe>=0.23 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from Jinja2>=2.9.5->appdaemon)
Requirement already up-to-date: idna>=2.0 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from yarl<1.0,>=0.11->appdaemon)
Requirement already up-to-date: cffi>=1.1 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from bcrypt->appdaemon)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from requests>=2.6.0->appdaemon)
Requirement already up-to-date: certifi>=2017.4.17 in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from requests>=2.6.0->appdaemon)
Requirement already up-to-date: pytz in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from astral->appdaemon)
Requirement already up-to-date: pycparser in c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages (from cffi>=1.1->bcrypt->appdaemon)
Installing collected packages: appdaemon, multidict
  Found existing installation: appdaemon 3.0.0b1
    Uninstalling appdaemon-3.0.0b1:
      Successfully uninstalled appdaemon-3.0.0b1
  Found existing installation: multidict 4.0.0
    Uninstalling multidict-4.0.0:
      Successfully uninstalled multidict-4.0.0
Exception:
Traceback (most recent call last):
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\MyPc\\AppData\\Local\\Temp\\pip-ldwhcrv0-uninstall\\users\\mypc\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\multidict\\_istr.cp36-win_amd64.pyd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\req\req_set.py", line 795, in install
    requirement.commit_uninstall()
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\req\req_install.py", line 767, in commit_uninstall
    self.uninstalled.commit()
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\req\req_uninstall.py", line 142, in commit
    rmtree(self.save_dir)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\_vendor\six.py", line 686, in reraise
    raise value
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\utils\__init__.py", line 102, in rmtree
    onerror=rmtree_errorhandler)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 6 more times]
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\users\mypc\appdata\local\programs\python\python36\lib\site-packages\pip\utils\__init__.py", line 114, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\MyPc\\AppData\\Local\\Temp\\pip-ldwhcrv0-uninstall\\users\\mypc\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\multidict\\_istr.cp36-win_amd64.pyd'

C:\WINDOWS\system32>

keithh666 · January 28, 2018, 11:03pm

Doing it a 2nd time seems to fix the issue

tschmidty · January 29, 2018, 6:24am

Working well for me, nice. Yaml doesn’t register an error and .py logs but doesn’t throw an exception.

One question, can I use secrets in apps.yaml files? I tried putting it in hass secrets as well as in the apps directory and still get an error like this

Jan 28 17:41:38 home appdaemon[3491]:   File "/srv/homeassistant/lib/python3.6/site-packages/appdaemon/utils.py", line 104, in _secret_yaml                                      Jan 28 17:41:38 home appdaemon[3491]:     raise ValueError("{} not found in secrets file".format(node.value))
Jan 28 17:41:38 home appdaemon[3491]: ValueError: ask_wolfram_api_key not found in secrets file

I can poke at the code but wasn’t sure if I should be expected t be able to use that outside the appdaemon.yaml config file itself.

tschmidty · January 29, 2018, 6:53am

One other quick note, dependencies don’t seem to reload quite the way I would expect. I have a module that is a dependency for a number of other modules, but changes to it don’t reload the dependencies.

OK, and again this is me being a pain and doing a lot with your code, but I absolutely love the auto reloading and think it makes a great framework for developing things. And I apologizing for just raising the issue and not proposing a solution, banging on hassio and docker this weekend and haven’t had a chance to poke.

This is on startup

Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.238096 INFO AppDaemon: Loading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_core.py                   Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.239495 INFO AppDaemon: Loading Object jarvis_core using class jarvis_core from module jarvis_core
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.247819 WARNING AppDaemon: No app description found for: /home/homeassistant/.homeassistant/apps/jarvis/data/fetch_netfl
ix_ids.py - ignoring                                                                                                                                                             Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.247980 INFO AppDaemon: Loading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_music/jarvis_music.py
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.249129 INFO AppDaemon: Loading Object jarvis_music using class jarvis_music from module jarvis_music                   Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.249467 INFO AppDaemon: Loading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_lights/jarvis_lights.py
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.250431 INFO AppDaemon: Loading Object jarvis_lights using class jarvis_lights from module jarvis_lights                Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.255852 INFO jarvis_core: jarvis_register_intent: lightsTurnOnSet
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.257316 INFO jarvis_core: jarvis_register_intent: lightsTurnOff                                                         Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.258643 INFO jarvis_core: jarvis_register_intent: lightsTurnUp
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.259934 INFO jarvis_core: jarvis_register_intent: lightsTurnDown
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.260112 INFO AppDaemon: Loading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_weather/jarvis_weather.py
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.261108 INFO AppDaemon: Loading Object jarvis_weather using class jarvis_weather from module jarvis_weather
Jan 28 15:36:17 home appdaemon[3491]: 2018-01-28 15:36:17.261440 INFO AppDaemon: Loading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_roku_tv/jarvis_roku_tv.py

In this case sub apps call a function in jarvis_core to register intents. But if I later make a change to jarvis_core it doesn’t reload the dependent modules. Not normally an issue, but since I reloaded jarvis core I lost the intents I registered from the sub modules.

If i change the apps.yaml it does reload them, but doesn’t actually run the initialize function again.

Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.457545 INFO AppDaemon: /home/homeassistant/.homeassistant/apps/jarvis/jarvis.yaml added or modified
Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.576233 INFO AppDaemon: Reloading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_timer/jarvis_timer.py
Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.585897 INFO AppDaemon: App 'jarvis_thermostat' added - running
Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.587968 INFO AppDaemon: Reloading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_thermostat/jarvis_thermostat.py
Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.589378 INFO AppDaemon: App 'jarvis_roku_tv' added - running
Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.591457 INFO AppDaemon: Reloading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_roku_tv/jarvis_roku_tv.py
Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.605489 INFO AppDaemon: App 'jarvis_ask_wolfram' added - running
Jan 29 01:50:04 home appdaemon[3491]: 2018-01-29 01:50:04.607529 INFO AppDaemon: Reloading Module: /home/homeassistant/.homeassistant/apps/jarvis/jarvis_ask_wolfram/jarvis_ask_wolfram.py

I should have some time this week to poke at code if you don’t get a chance to look, I hate to be that guy who just complains without offering a solution!

ReneTode · January 29, 2018, 11:42am

that guy is me
i also did complain about the reloading
if i change 1 app instance it seems to reload all instances (or at least in the log)
1 have 1 app with over 50 instances, i dont want them all to be reset if i change just 1
I am sure Andrew will catch the problem, but some help never hurts

aimc · January 29, 2018, 1:18pm

The reloading should work ok @tschmidty do you have your dependencies configured correvtly?

And no, you can;t put secrets in an apps.yaml file currenyly.

@ReneTode - that is working as designed currently - it may be an unintended side effect of some other changes I made, so I’ll need to dig into the code to double check that. This is an evolving area so it will get some more love before the beta is done.

ReneTode · January 29, 2018, 1:52pm

im glad that you will give it some more love, because it isnt really as it can/should/must be

if i create a yaml with 2 instances from an app i get this in the log:

1 of the instances added - running
py file (module) reloading
all other already existing instances reloaded.

the behaviour from AD 2 was better.
in the case i added 2 instances from an app to the apps.yaml i got:

both instances loaded
the module (which didnt change) was not reloaded
all other instances were uneffected

i am still not sure if its just a logging problem, but i did add the behaviour to my doc file

i must correct that:
the actuall behaviour is now:

first instance added - running
py file (module) reloading
all other already existing instances reloaded.
next instance added - running
py file (module) reloading
all other already existing instances reloaded.
next instance added - running
py file (module) reloading
all other already existing instances reloaded.
etc.

tschmidty · January 29, 2018, 2:07pm

I thought I did

jarvis_core:
  module: jarvis_core
  class: jarvis_core

jarvis_weather:
  module: jarvis_weather
  class: jarvis_weather
  dependencies: jarvis_core

tschmidty · January 29, 2018, 6:19pm

Hmm, so I started thinking about plugins more since I could see them being very useful, but would really require a little tweaking to work the way I am thinking.

One thing I think would be very cool is to make one app be able to work with different plugins which would obviously require changing the way imports are done in the apps.

Bit of a grand idea but, my thinking is I’d like to implement a standalone plugin (really just an mqtt listener for my purposes) and probably an OpenHab plugin, and my app would be able to work with all of them (not necessarily at the same time!). Any thoughts?

aimc · January 29, 2018, 7:07pm

MQTT support as a great idea for a plug-in. You have to have 2 apps though as each app can only have one parent object. Apps can be in the same file though. Also apps using different instances of the same plug-in can work together using namespaces. I envisaged apps cooperatively sharing access to dissimilar plugins using get_app()