HA automation in Python from a developer's POV

[molto_b (Mike Pagel)]

The challenge

HA allows various ways of scripting and automation. I am still blown away of how much stuff can be done in the HA application itself, be it the UI, an embedded editor for YAMLs or extensions like NodeRed.

If you’re proficient with Jinja templating and YAML you’ll get things done. I am. But is that a development experience I am completely happy with?

Not quite, as I keep comparing this to a native development environment. So questions like this might come up:

• I can discover many options in the UI, but eventually, I’ll end up using YAML configs. How do I easily discover APIs and constructs without code completion (or browsing docs all the time)?
• How can I cleanly structure reuse? Yes, we can create and call (utility) scripts, services and automations, but I believe all these items live at the same (top) level. How do I ensure I’ll still find stuff? And blueprints are not more than gists I clone. That’s hard to maintain.
• More complex logic might deserve some (automated) test code beyond live rollout and running through the house to check.
• Other quality tools (black, linters, …) might be interesting.
• How can I debug? Can I set a breakpoint to inspect the data that is passed to my automation? The tracing feature for automations is terrific but it’s not a debugger.
• Maybe I want to use my own IDE?

As a Python developer I looked at the options to script automations in Python, but the above kept nagging me. I wanted a real developer experience.

Python automation options

I looked at these:

• The built-in Python scripts integration is a good first start, but quite limited in features.
• PyScript is much more capable and a very cool project. It’s only a subset of Python though and one of the things it lacks is support for classes (hope I did read this correctly). I’d like to use classes for organization of my logic and not wholy rely on function composition.
• AppDaemon is real Python and does have reuse in mind. As an externally running application it is limited in terms of what it can do within HA, but on the other hand, it is decoupled. And that allows us to run it from anywhere… IDE, I’m coming!

There were more options that crossed my path, but none of them seemed to be in use very much, so I’ll leave them out.

My setup

AppDaemon (AD) is a stand-alone application written in Python. It talks to HA over a web (socket) API. This allows running an instance of AD on your local development machine. In fact, you can run one AD within your production setup and connect with another development instance to the same HA, if you like.

The setup of the development machine is:

1. Create a virtual environment and install the appdaemon package. Immediately, you are able to call appdaemon -h to get the CLI help page.
2. Install your preferred development tools into that venv.
3. AD is looking for configuration to load, typically from a folder called config, create that as part of your working area. It usually contains this content:
   • The appdaemon.yaml file. It will be almost identical to the one on the production system, just the own http interface should be set to http://127.0.0.1:5050.
   • An apps folder. This is where your AD apps will be created.
4. I use git not only for versioning, but also to (literally) pull updates onto the production system. Watch out to not overwrite anything but AD apps. I use a Github repo as shared repository.
5. In the IDE/editor or your choice, consider creating run/debug configurations.
6. Take a look at the -D option of calling AD, it allows you see your module’s debug log messages in the console.

Having this in place, start reading the AD docs. For me the best initial page was Writing AppDaemon Apps. It explains most of the concepts to get a good first grasp.

And now, start coding, set breakpoints, go to definitions of AD methods to learn, how they do things etc.

To get the best experience (at least in PyCharm), I am using the asynchronous approach (all callbacks are declared as async def). Besides providing good code completion this allows for very simple sequence programming, as asynchronous sleep can be used.

Here are some impressions of the development experience:

---

Prepared run configuration

---

Logs including debug output of automation code

---

Inspecting a breakpoint in a callback

---

Code completion

Wrapping up

I hope this can be useful for some developers. It would have helped me getting over that initial difficulty of choosing a Python development solution. Let me know how you do development of automations and of course help me correcting false statements or assumptions I might have made.

[dubhunter (Will Mason)]

@molto_b you are a genius. This is a total game changer.

I finally (after too long) decided to sit down and get AppDaemon working tonight. While working on my first “real” app, I immediately wanted my debugger. A few failed google searches later, I found your post. A little tinkering and I just hit my first break point.

Thanks for your post!!

[zzafarr (ZB)]

@molto_b How do you debug with AppDaemon?

Also, is there a way (easy ) to write the Unit Tests ?

Thanks!

[molto_b (Mike Pagel)]

I’m sorry I missed your question until now!

Yes, totally, you can write tests. I personally do not test (hence mock) the interaction with AD. I am mainly interested in validating internal logic, i.e. my own code. Here I also have control over whether async is used or not, possibly making my life easier as well.

Hope this gives some insight.

[brazoayeye]

thanks for informations!

I installed the HASS add on, copied the appdaemon.yaml from there but it’s very small and missing all connection informations

---
appdaemon:
  latitude: 42.779189
  longitude: 6.599431
  elevation: 2
  time_zone: Europe/Amsterdam
  plugins:
    HASS:
      type: hass
http:
  url: http://127.0.0.1:5050
admin:
api:
hadashboard:

If I stop the plugin and I try
venv/bin/python3.11 -m appdaemon -D DEBUG -c .
i can’t connect because I haven’t specified connection informations.

Do you know how I can extract the informations from the plugin?
On the other side do you have a guide to understand needed parameters?

After that, can you send me the launch.json for vscode or a link explaining how to setup it for the debug?

Thanks

[molto_b (Mike Pagel)]

Under appdaemon I you’re missing where it should connect. The http section is for appdeamon’s HTTP interface, but appdaemon wants to talk to HA and you need to tell it where.

Here is my appdaemon YAML, I replaced values with angular bracketed expressions, where you need to add your data:

appdaemon:
  # this is important to get right to let HA/AD work with sunrise/sunset:
  latitude: <...>
  longitude: <...>
  elevation: <...>
  time_zone: <...>
  plugins:
    HASS:
      type: hass
      # you might need to ignore cert warnings, or switch the URL to http:
      # cert_verify: false
      ha_url: https://<url or IP where HA listens>
      token: <long lived token created in HA>
  <more config irrelevant here>
  # this might help if you find the kwargs handling in AD weird, see AD docs:
  use_dictionary_unpacking: true

<...>

And this would be the entry in my launch.json:

        {
            "name": "appdaemon",
            "type": "python",
            "request": "launch",
            "module": "appdaemon",
            "args": [
                "--config",
                "config",
                "-m",
                "radiator_guestbath",
                "DEBUG",
            ],
            "cwd": "${workspaceFolder}/../..",
            "console": "integratedTerminal",
            "justMyCode": false,
        },

And finally, this is the folder structure (files partly omitted) I am working in and referenced by the launch config:

[brazoayeye]

Thanks, now I can connecto to hass but i’m unable to launch apps.

appdaemon.yaml

appdaemon:
  latitude: 42.379189
  longitude: 4.899431
  elevation: 2
  time_zone: Europe/Amsterdam
  plugins:
    HASS:
      type: hass
      ha_url: https://asdasd.duckdns.org:8123
      token: "s sdds"
http:
  url: http://127.0.0.1:6060
admin:
api:
hadashboard:

apps/apps.yaml

hello_world:
  module: hello
  class: HelloWorld

apps/hello.py

import appdaemon.plugins.hass.hassapi as hass

class HelloWorld(hass.Hass):
  def initialize(self):
    self.log("Hello from AppDaemon")

result:

$ venv/bin/python3 -m appdaemon -c .
2024-05-07 16:44:41.305760 INFO AppDaemon: AppDaemon Version 4.4.2 starting
2024-05-07 16:44:41.306268 INFO AppDaemon: Python version is 3.11.9
2024-05-07 16:44:41.306358 INFO AppDaemon: Configuration read from: ./appdaemon.yaml
2024-05-07 16:44:41.306517 INFO AppDaemon: Added log: AppDaemon
2024-05-07 16:44:41.306600 INFO AppDaemon: Added log: Error
2024-05-07 16:44:41.306751 INFO AppDaemon: Added log: Access
2024-05-07 16:44:41.306825 INFO AppDaemon: Added log: Diag
2024-05-07 16:44:41.324518 INFO AppDaemon: Loading Plugin HASS using class HassPlugin from module hassplugin
2024-05-07 16:44:41.337477 INFO HASS: HASS Plugin Initializing
2024-05-07 16:44:41.338042 INFO HASS: HASS Plugin initialization complete
2024-05-07 16:44:41.338288 INFO AppDaemon: Initializing HTTP
2024-05-07 16:44:41.338541 INFO AppDaemon: Using 'ws' for event stream
2024-05-07 16:44:41.339990 INFO AppDaemon: Starting API
2024-05-07 16:44:41.341121 INFO AppDaemon: Starting Admin Interface
2024-05-07 16:44:41.341325 INFO AppDaemon: Starting Dashboards
2024-05-07 16:44:41.460840 INFO HASS: Connected to Home Assistant 2024.5.2
2024-05-07 16:44:41.464629 INFO AppDaemon: Starting Apps with 0 workers and 0 pins
2024-05-07 16:44:41.467460 INFO AppDaemon: Running on port 6060
2024-05-07 16:44:41.722923 INFO HASS: Evaluating startup conditions
2024-05-07 16:44:41.757039 INFO HASS: Startup condition met: hass state=RUNNING
2024-05-07 16:44:41.757188 INFO HASS: All startup conditions met
2024-05-07 16:44:41.819297 INFO AppDaemon: Got initial state from namespace default
2024-05-07 16:44:43.504259 INFO AppDaemon: Scheduler running in realtime
2024-05-07 16:44:43.506431 WARNING AppDaemon: No app description found for: ./apps/hello.py - ignoring
2024-05-07 16:44:43.506935 INFO AppDaemon: App initialization complete
^C2024-05-07 16:44:49.817210 INFO AppDaemon: Keyboard interrupt
2024-05-07 16:44:49.817472 INFO AppDaemon: AppDaemon is shutting down
2024-05-07 16:44:49.876364 INFO HASS: Disconnecting from Home Assistant
2024-05-07 16:44:50.555384 INFO AppDaemon: Removing module ./apps/hello.py
2024-05-07 16:44:50.555700 INFO AppDaemon: Shutting down webserver
2024-05-07 16:44:50.556109 INFO AppDaemon: Saving all namespaces
2024-05-07 16:44:50.556244 INFO AppDaemon: AppDaemon is stopped.
2024-05-07 16:43:50.166582 INFO AppDaemon: AppDaemon is stopped.

Why isn’t it running?
From the hass plugin → log i can see “Hello from AppDaemon”

Thanks

[brazoayeye]

The problem was caused by -c . parameter.

I have to use absolute path

appdaemon doesn't load apps when the config path is relative

[serpent1]

Hello, how do you have your run/debug config setup for pycharm?

Running the app scripts doesn’t trigger the breakpoints, which makes sense as they’re just classes and need to be triggered by the appdaemon, but the app daemon is an exe so that can’t be run directly, so I’m not really sure how to setup pycharm’s debugger to catch the breakpoints?

[molto_b (Mike Pagel)]

To any developers who are looking for ways of advanced automation and found this article, I’d like to give a quick recap of having spent 2,5 years with AppDaemon now.

tl;dr: I just got rid of AppDaemon in my setup.

(Unfortunately I can no longer edit the original post to put the update there.)

The gist of it is:

• AD is still a very powerful platform.
• AD has all the advantages of local development and debugging listed above.

But, and that’s a bigger “but” than I wanted to believe initially:

• AD updates have wrecked my setup a number of times by introducing breaking changes, without larger announcements, major version updates or migration help.
• AD does not support device triggers AFAIK (please correct me, if it does). This wrecked my setup for Z2M-based buttons, when Z2M made their move away from legacy action triggers. The lack of the feature required to write an MQTT wrapper app etc.
• While AD apps may use async (just like HA and AD do internally), the code base is currently broken, when trying to configure this.
• This break was introduced by a (IMHO) major refactoring of the startup sequence of apps, done in a minor version update to 4.5.12
• The codebase of AD is typed for non-async apps, leading to numerous typing issues when using async apps.
• In my setup (HAOS on Proxmox on a MiniPC), AD’s appears to lag compared to other solutions. But take that with a grain of salt, My mood might have influence that, I did not really measure.

My apps can still not be ported to the current AD version because of the breaking change without major refactoring, as e.g. 4.5.12 prevents entities to be read in the initialize method.

So all in all I am no longer in the mood to run after AD changes or issues. Instead of refactoring my AD apps I moved completely away from it and uninstalled the add-on today.

Since I still like to use Python automations I reevaluated today’s options and eventually settled for PyScript, in combination with Blueprints and deployments with Ansible.

PyScript is still not perfect, has it quirks, but it is a much lighter dependency (just an integration) and appears to be much closer to HA (you are running async within the HA loop), actually feels snappy, and also just gets the job done.

I hope I am not in a similar situation as with AD any time soon

Let me know if you want more details on the current approach, then I’ll write up another article, but that takes a bit more thought than this reply.

[deanpm70]

Thanks for this update Mike. I would have gone charging down the AD route had you not. Have had a few months to work with PyScript, how do you feel about it now?

[entdgc (David Cheshire)]

Thanks for this summary. I had a little dabble with AD but preferred the simplicity of using pyscript. It generally works well but one thing I find a little difficult is debugging. What is your debugging work flow? Is there any easy way to set breakpoints in pyscript?

[molto_b (Mike Pagel)]

I am still using pyscript. But I agree with the comment from @entdgc Breakpoints are not easily possible, as pyscript is not Python in terms of interpreter, so none of the standard tooling will work.

Also I kind of revisited the state of built-in stuff with blueprints and packages and my typical setup for non-trivial logic is to put that in pyscript and trigger that pyscript module from a blueprint-based automation. In the pyscript code you need to maintain some dict from "automation name/id/..." passed in as an argument to instance state. Feels a bit like doing OO in C:

class ClimateState:
    automation_id: str
    radiator_id: str
    ...

state_by_instance: dict[str, ClimateState] = {}

@service
def climate_control(
    automation_id: str,
    radiator_id: str,
    ...
):
    # Get or create state for this automation instance
    if not (climate_state := state_by_instance.get(automation_id)):
        climate_state = ClimateState()
        state_by_instance[automation_id] = climate_state

        climate_state.automation_id = automation_id
        climate_state.radiator_id = radiator_id
        ...

    # work with client state vars...

But honestly, once that hurdle is taken, I am happy with it. Especially since AI now knows the pattern...