Check out my CI deployment pipeline

After a conversation about it on Reddit, I decided to give deploying my config via Gitlab CI another try. This time I set up a local runner, which removed the biggest issue I had with my earlier attempt (the speed of Gitlab’s shared runners).

I based most of my config on Frenck’s awesome configuration, which has some pretty thorough checks.

I’ve written up my experiences below:

https://webworxshop.com/continuous-integration-deployment-for-home-assistant-with-gitlab-ci/

Please let me know what you think. Any feedback and improvements will be greatly received.

7 Likes

Thank you, trying to work this out myself. Much appreciated - was trying to follow a post from a year ago (https://about.gitlab.com/2018/08/02/using-the-gitlab-ci-slash-cd-for-smart-home-configuration-management/) to do this, but I think using your method will be better for me.

I will try to follow it and let you know.

1 Like

Glad you enjoyed it. Let me know how it goes.

I’m a bit new to this - whats the advantage (benefit if you will) of going Gitlab over Travis?

I’m afraid I really have no idea, since I’ve never used Travis! Gitlab is a full git hosting solution (more fully featured that Github, although you can use GH repos with the CI). Gitlab is also largely Open Source (Travis and GH are not).

1 Like

Great stuff. Glad CI/CD in the HASS world is getting traction. Thanks for sharing!

Here’s my journey with HASS-related CI (and a very simplistic CD). In order to keep my code private, I use Gitea as a self-hosted Git service and Drone as my CI platform. Although syntax/etc. is different from Travis, GitLab, etc., the same concepts apply – if nothing else, the scope of what I’m testing might be of interest.

What I Test

AppDaemon: I run various tools across my AppDaemon apps to ensure that the code is linted, Black-ified, and type-checked. Additionally, although it’s not there today, I’m slowly working on incorporating the AppDaemon Test Framework, as well, so that unit tests will be a part of CI.

ESPHome: I run config checks against both beta and latest versions of ESPHome. I also run a YAML linter on my configuration files.

Home Assistant: I run config checks against dev, rc, and latest versions of HASS. I also run a YAML linter on my configuration files.

NGINX: I run both a config check and static analysis against my NGINX config (used for a reverse proxy to HASS, AppDaemon, etc.) – this ensures that the config (a) is correct and (b) adheres to good standards. I also run a config check against Fail2Ban, which I use to firewall off IPs that try to access NGINX in an unauthorized fashion.

Shell Scripts: I run a static analysis check (using shellcheck) on all of my Bash scripts to ensure that they conform to good standards.

A Drone run looks like this:

When everything completes, I get a Slack notification:

16%20AM

Drone Configuration File

Like all of the other modern CI tools, Drone uses a configuration file to define the pipelines, steps, etc. Here’s mine:

.drone.yml

---
kind: pipeline
name: AppDaemon

trigger:
  branch:
    - master
    - feature/*

steps:
  - name: Linting & Typing
    image: python:3
    commands:
      - pip3 install --upgrade pip black flake8 mypy pylint yamllint
      - black --check --fast ./appdaemon/settings/apps/
      - flake8 ./appdaemon/settings/apps/
      - mypy --ignore-missing-imports ./appdaemon/settings/apps/
      - pylint --rcfile ./appdaemon/settings/pylintrc appdaemon/settings/apps/
      - yamllint ./appdaemon/settings/

---
kind: pipeline
name: ESPHome

trigger:
  branch:
    - master
    - feature/*

steps:
  - name: "Config Check: Beta"
    image: esphome/esphome:beta
    pull: always
    failure: ignore
    commands:
      - "for file in $(find /drone/src/esphome -type f -name \"*.yaml\" -not \
         -name \"secrets.yaml\"); do esphome \"$file\" config; done"
  - name: "Config Check: Latest"
    image: esphome/esphome:latest
    pull: always
    commands:
      - "for file in $(find /drone/src/esphome -type f -name \"*.yaml\" -not \
         -name \"secrets.yaml\"); do esphome \"$file\" config; done"
  - name: Linting
    image: python:3
    commands:
      - pip3 install --upgrade pip yamllint
      - yamllint ./esphome/

---
kind: pipeline
name: Home Assistant

trigger:
  branch:
    - master
    - feature/*

steps:
  - name: "Config Check: Dev"
    image: homeassistant/home-assistant:dev
    pull: always
    failure: ignore
    commands:
      - cd /usr/src/homeassistant
      - "python -m homeassistant -c /drone/src/home-assistant/settings \
         --script check_config"
  - name: "Config Check: Latest"
    image: homeassistant/home-assistant:latest
    pull: always
    commands:
      - cd /usr/src/app
      - "python -m homeassistant -c /drone/src/home-assistant/settings \
         --script check_config"
  - name: "Config Check: RC"
    image: homeassistant/home-assistant:rc
    pull: always
    commands:
      - cd /usr/src/homeassistant
      - "python -m homeassistant -c /drone/src/home-assistant/settings \
         --script check_config"
  - name: Linting
    image: python:3
    commands:
      - pip3 install --upgrade pip yamllint
      - yamllint ./home-assistant/settings/

---
kind: pipeline
name: NGINX

trigger:
  branch:
    - master
    - feature/*

steps:
  - name: "Config Check: NGINX"
    image: alpine:3.10.1
    commands:
      - apk update && apk add --no-cache nginx
      - ln -s /drone/src/nginx/settings/nginx /etc/nginx
      - nginx -t -c /etc/nginx/nginx.conf -g 'pid /tmp/nginx.pid; daemon off;'
  - name: "Config Check: Fail2Ban"
    image: alpine:3.10.1
    commands:
      - apk update && apk add --no-cache fail2ban
      - ln -s ./nginx/settings/fail2ban /etc/fail2ban
      - touch /var/log/messages
      - "mkdir -p /var/log/nginx && touch /var/log/nginx/access.log \
         && touch /var/log/nginx/error.log"
      - fail2ban-client -t
  - name: "Static Analysis: NGINX"
    image: yandex/gixy:latest
    pull: always
    commands:
      - ln -s /drone/src/nginx/settings/nginx /etc/nginx
      - gixy /etc/nginx/nginx.conf

---
kind: pipeline
name: Shell Scripts

trigger:
  branch:
    - master
    - feature/*

steps:
  - name: "Static Analysis: bin/*"
    image: koalaman/shellcheck-alpine:stable
    pull: always
    commands:
      - shellcheck --version
      - shellcheck ./bin/*

---
kind: pipeline
name: Wrap-up

trigger:
  branch:
    - master
    - feature/*
  status:
    - failure
    - success

depends_on:
  - AppDaemon
  - ESPHome
  - Home Assistant
  - NGINX
  - Shell Scripts

steps:
  - name: Send Notification
    image: plugins/slack
    settings:
      webhook:
        from_secret: slack_webhook
      channel:
        from_secret: slack_channel_name
      template: >
        {{#success build.status}}
          `{{repo.name}}/{{build.branch}}`: Build #{{build.number}} successful
        {{else}}
          `{{repo.name}}/{{build.branch}}`: Build #{{build.number}} failed
        {{/success}}
    when:
      status:
        - failure
        - success

Deployment

I don’t particularly want my Drone server – a Linode VPS – to be able to punch through my home network and modify the NUC that runs HASS, AppDaemon, etc. Therefore, I chose a simpler method: I run a regular Bash script on the NUC that checks for newly-successful Drone jobs and, upon finding one, deploys everything via docker-compose.

bin/build (the script that builds the entire stack)

#!/bin/bash
set -euxo pipefail

REPO_PATH="$( dirname "$( cd "$(dirname "$0")" ; pwd -P )" )"

function copy_config_data() {
  volume_name=$1
  settings_path=$2

  docker volume create "$volume_name"

  docker run --rm \
    -v "$settings_path":/src \
    -v "$volume_name":/data \
    -w /src \
    busybox \
    cp -a . /data
}

# Update AppDaemon configs:
copy_config_data "hub_appdaemon-config" "$REPO_PATH"/appdaemon/settings

# Update Glances configs:
copy_config_data "hub_glances-config" "$REPO_PATH"/glances/settings

# Update Home Assistant configs:
copy_config_data "hub_hass-config" "$REPO_PATH"/home-assistant/settings

# Re-build the containers if necessary:
docker-compose -f "$REPO_PATH"/docker-compose.yml build

# Tear the stack down and clean up loose containers and images:
docker-compose -f "$REPO_PATH"/docker-compose.yml down --remove-orphans
docker container prune -f
docker image prune -f

# Bring up all "background" services (i.e., those services that can come up before the
# big three: HASS, AppDaemon, and NGINX):
docker-compose -f "$REPO_PATH"/docker-compose.yml up -d \
    --scale appdaemon=0 \
    --scale hass=0 \
    --scale nginx=0

# Bring up HASS, AppDaemon, and NGINX with delays in-between:
sleep 5
docker-compose -f "$REPO_PATH"/docker-compose.yml up -d hass
sleep 5
docker-compose -f "$REPO_PATH"/docker-compose.yml up -d appdaemon
sleep 5
docker-compose -f "$REPO_PATH"/docker-compose.yml up -d nginx

# Do a final cleanup of unused networks and volumes:
docker network prune -f
docker volume prune -f

bin/deploy (the script that runs at regular intervals and, upon finding a successful Drone job, calls bin/build)

#!/bin/bash
set -euxo pipefail

# Variables to edit when copying this deployment script to other repos:
BRANCH="master"
REPO_NAME="hub"

# Variables that shouldn't need editing:
API_BASE="https://my-drone-server"
API_LAST_BUILD="api/repos/bachya/$REPO_NAME/builds"
API_TOKEN="token123"
API_USER_AGENT="User-Agent: Paw/3.1.8 (Macintosh; OS X/10.14.6) GCDHTTPRequest"

SLACK_HOOK_URL="https://hooks.slack.com/services/..."
COLOR_BLUE="#1539e0"
COLOR_GREEN="good"
COLOR_RED="danger"
COLOR_YELLOW="warning"

LAST_BUILD_FILEPATH="/tmp/${REPO_NAME}_last_drone_build"
BIN_PATH="$( dirname "$( cd "$(dirname "$0")" ; pwd -P )" )"


function fetch_latest_build_number() {
    curl \
        -s "$API_BASE/$API_LAST_BUILD" \
        -H "$API_USER_AGENT" \
        -H "Authorization: Bearer $API_TOKEN" \
        | jq 'map(select(.status == "success" and .target == "'$BRANCH'"))[0].number'
}

function remote_git() {
    git --git-dir "$BIN_PATH"/.git --work-tree="$BIN_PATH" "$@"
}

function send_slack_message() {
   local color="$2"
   local text="$1"
   local message
   message="\`$REPO_NAME/$BRANCH\`: $text"

   curl \
       -X "POST" \
     -H 'Content-Type: application/json' \
     -d $'{ "attachments": [ { "color": "'"$color"'", "text": "'"$message"'" } ] }' \
    "$SLACK_HOOK_URL"
}

function should_update() {
    local latest_build_number
    local saved_build_number

    # If the last build file doesn't exist, create it:
    [ ! -f "$LAST_BUILD_FILEPATH" ] && touch "$LAST_BUILD_FILEPATH"

    latest_build_number=$(fetch_latest_build_number)
    saved_build_number=$(cat "$LAST_BUILD_FILEPATH")

    # If the build number is the same as the last one retrieved, don't trigger
    # a build:
    if [ "$latest_build_number" == "$saved_build_number" ]; then
        echo "false"
        return
    fi

    echo "$latest_build_number" > "$LAST_BUILD_FILEPATH"

    echo "true"
}

function update() {
    local git_pull_result
    local local_branch
    local saved_build_number

    # If the repo has a branch other than the target one checked out, don't trigger
    # a build:
    local_branch="$(remote_git rev-parse --abbrev-ref HEAD)"
    if [ "$local_branch" != "$BRANCH" ]; then
        send_slack_message \
            "Refusing to deploy to local branch: $local_branch" \
            "$COLOR_YELLOW"
        echo "false"
        return
    fi

    git_pull_result="$(remote_git pull origin master > /dev/null 2>&1)"
    if [ "$git_pull_result" == "1" ]; then
        send_slack_message \
            "Deployment failed due to failed Git pull on target host" \
            "$COLOR_RED"
        return
    fi

    saved_build_number=$(cat "$LAST_BUILD_FILEPATH")

    send_slack_message \
        "Deployment for build #$saved_build_number starting..." \
        "$COLOR_BLUE"

    "$BIN_PATH"/bin/build

    send_slack_message \
        "Deployment for build #$saved_build_number completed" \
        "$COLOR_GREEN"
}


dirty=$(should_update)
if [ "$dirty" == "true" ]; then
    update > /dev/null 2>&1
    echo -n "Deployment complete"
else
    echo -n "No deployment required"
fi

The Whole Thing

For reference and interest, my entire repo can be found here: https://github.com/bachya/smart-home

8 Likes

One benefit is that Travis is only free for public repos. GitLab’s built-in CI is available for private/self-hosted repos.

Wow, thanks for the detailed breakdown. That’s pretty awesome! I’d also love to see CI/CD get more traction in the HA world. It’s already made me more relaxed about changing my config - now it’s just “git push and chill” rather than worrying if I’m going to break something!

How are you finding Drone for CI? I already have a local Gitea server for repos that I don’t want in the cloud and for an extra backup. I really wanted to self-host Gitlab, but the resource requirements are prohibitive. I should probably have another look at Drone.

I also didn’t know about the AppDaemon test framework - I was thinking I might have to build my own, but this looks awesome! Thanks so much.

1 Like

I’m really loving Drone. I agree that hosted GitLab CI took too many resources; I imagine it has more core features, but Drone offsets that with an excellent plugin architecture and ecosystem that extends it nearly infinitely.

Thanks, that’s cool. I’ll definitely be taking a better look.

anyone tried the azure ci? I have not seen example here on the forum.

Not personally, but HASS itself uses Azure CI: https://github.com/home-assistant/home-assistant/blob/16fff16082b42f3b1d672c005737d137a8360478/azure-pipelines-ci.yml.

I didn’t quite get there yet @robconnolly - I haven’t forgotten. For feedback purposes I got stuck setting up the runner on an unRAID server and ran out of time. Will get back to you soon.

I’ve just made a massive update to my CI setup, with automated Docker deployment, ESPHome pipeline (including automated OTA updates) and a pipeline for AppDaemon integrating Appdaemontest framework:

https://webworxshop.com/continuous-integration-for-home-assistant-esphome-and-appdaemon/

Thanks to @bachya for the inspiration on the AppDaemon stuff!

2 Likes

Great stuff, @robconnolly! Love it. You’ve helped me realize that the “CD” portion of my setup – wherein my NUC has a regular cron job to check for new builds – was well-intentioned, but wonky.

I still don’t want to punch an SSH hole from my Drone server (a cloud VPS) to my home automation NUC, but you’ve inspired to consider some sort of public webhook (with a Drone build key) that, upon getting hit, will trigger a build. All that to say: thank you for your write-up; it helped me think more critically!

1 Like

I’m really glad you enjoyed it and that it made you think how to improve your own setup. I had thought about the webhook approach for my initial deployment, but the SSH approach just worked out easier for me in the end. It also means I’m not dependent on HASS actually being running to deploy my code, which might be useful in a real emergency. Also, I’m not really sure I could do the Docker deployment stuff, since that really needs to run on the Docker host.

I see your point about not wanting to give SSH access to your cloud VM. If it’s any help, it’s possible to lock down the commands that can be run via a given SSH key by specifying a command=.... string before the key in your authorized_keys file.

There’s also a whole load of other arguments you can supply to lock things down further. See https://serverfault.com/questions/142997/what-options-can-be-put-into-a-ssh-authorized-keys-file for a good run down.

1 Like

Great point – with keys-only auth, allowing only specific commands, and fail2ban, I suppose there isn’t a huge reason to worry about exposing SSH. :+1:

Yep, SSH is probably the most secure thing you could have exposed! Don’t forget to disable root login and only allow key based auth. If you like you can always run it on a random high port too, which significantly reduces the volume of script based login attempts.

1 Like

A quick update on why CI/CD is nice. Today, I pushed some new changes to my HASS and noticed that the piplines testing the rc and dev images was failing:

A quick look at the appropriate file showed that, indeed, the configuration schema had been changed: https://github.com/home-assistant/home-assistant/pull/28830

If I didn’t have my pipeline in place, my changes would have deployed and I would never have been the wiser that something had broken.

2 Likes

That’s awesome and exactly the kind of thing it’s supposed to catch!