I had a decently complex Jinja macro that I wanted to write some test for. I probably could have done something in Python, but I started working through a route to get it working fully within Home Assistant, and it worked out.
It’s not perfect and it’s a bit ugly, but it works. One downside is there’s no real way to catch template errors, but I have a global error handler that reads the log file and reports errors it finds, so those tend to get caught even if I’m not tailing the log file.
Example Macro and Test Suite
custom_templates/math.jinja:
{% macro add(a, b) %}
{{ a + b }}
{% endmacro %}
custom_templates/math_tests.yaml:
- macro: add
description: adding should work well
params:
- 2
- 3
result: 5
And a file gets generated:
{# automatically generated test file. do not edit. #}
{% from 'tests_templates.jinja' import assertion %}
{% from 'math.jinja' import add %}
{% macro run_suite() %}
[
{{ assertion(add(2, 3)|int, 5, 'add(2, 3)|int', 'adding should work well') }}
]
{% endmacro %}
Script
This starts the automation that handles actually running the results. It fires
an event to get that process started and waits on the result. It runs nicely
from the Developer Tools Action screen.
icon: mdi:test-tube
alias: Run Custom Template Test Suite
mode: restart
description: ""
fields:
report_success:
default: true
description: >
Set to `false` to prevent notifying of a successful test run.
selector:
boolean: {}
reload_automations:
default: true
description: >
Set to `false` to skip the reloading of automations. Note that this may
result in test files not being updating properly since the test
configuration is pulled in through the automation system.
selector:
boolean: {}
sequence:
- if:
- condition: template
value_template: >
{{ iif(reload_automations|default(true)) }}
then:
- action: automation.reload
- event: templates_test_suite.run
event_data:
reload_templates: true
report_success: >
{{ report_success|default(true) }}
- wait_for_trigger:
- platform: event
event_type: templates_test_suite.complete
timeout:
minutes: 1
- variables:
test_results: >
{{
{
'timeout': wait.remaining == 0,
'failures': wait.trigger.event.data if wait.remaining > 0 else none,
}
}}
- stop: done
response_variable: test_results
Automation
This actually writes out the test files and then runs the tests. It’s an
automation so that it can run automatically at midnight each night as well.
It needs to be defined at automations/somdir/somename.yaml or you need to
change the path within it so that the !include works properly.
Each test suite is loaded from a .yaml file within custom_templates. A
file is then created by calling the create_test_file macro and the content
of that file is passed off to a shell script to write the test Jinja file into
the custom_templates directory.
It then writes out another file to run all tests and imports & runs them.
id: templates_manage_test_suite
alias: Manage Templates Test Suite
mode: queued
max: 10
description: ""
trigger:
- platform: homeassistant
event: start
- platform: event
event_type:
- templates_test_suite.run
- platform: time
at: "00:00:00"
condition: []
action:
- variables:
notify_devices: mobile_app_my_device
test_suites: !include_dir_named ../../custom_templates
report_success: >
{{
trigger.platform == 'event' and
trigger.event.data.report_success|default(false)
}}
reload_templates: >
{{
trigger.platform == 'event' and
trigger.event.data.reload_templates|default(false)
}}
# reload templates if requested before writing test files to ensure the tests
# are being written with the most recent templates.
- if:
- condition: template
value_template: >
{{ reload_templates }}
then:
- action: homeassistant.reload_custom_templates
- alias: "Write individaul test files"
repeat:
for_each: "{{ test_suites.keys()|list }}"
sequence:
- variables:
content: >
{% from 'tests_templates.jinja' import create_test_file %}
{{
create_test_file(
repeat.item|regex_replace('_tests$', ''),
test_suites[repeat.item]
)
}}
- action: shell_command.write_template_tests
data:
encoded_json: >
{{ {
'filename': repeat.item,
'content': content,
}|to_json|base64_encode }}
response_variable: command_response
- if:
- condition: template
value_template: >
{{ command_response['returncode'] != 0 }}
then:
- action: persistent_notification.create
data:
title: Failure to write individual test file
message: >
{{ command_response['stderr'] }}
- alias: "Generate full run test file"
variables:
content: >
{% from 'tests_templates.jinja' import create_full_run_file %}
{{
create_full_run_file(
test_suites.keys()
|map('regex_replace', '_tests$', '')
|list
)
}}
- alias: "Write full run test file"
action: shell_command.write_template_tests
data:
encoded_json: >
{{ {
'filename': 'test_all',
'content': content,
}|to_json|base64_encode }}
response_variable: command_response
- if:
- condition: template
value_template: >
{{ command_response['returncode'] != 0 }}
then:
- action: persistent_notification.create
data:
title: Failure to write full run test file
message: >
{{ command_response['stderr'] }}
# reload templates if requested after writing test files to ensure the tests
# are being run with the just written templates.
- if:
- condition: template
value_template: >
{{ reload_templates }}
then:
- action: homeassistant.reload_custom_templates
- alias: "Run tests"
variables:
test_results: >
{% from 'test_all.jinja' import run_all %}
{{ run_all() }}
- alias: "Find failed tests"
variables:
failed_tests: >
{{ test_results
|selectattr('0', 'eq', false)
|list }}
- alias: "Report test failures"
if:
- condition: template
value_template: >
{{ failed_tests|length > 0 }}
then:
- alias: "Variables: title, message"
variables:
title: Custom Jinja2 template test failures
message: >
{{ failed_tests|length }} failed tests:
{% for failure in failed_tests|map(attribute='1')|list %}
- {{ failure }}
{% endfor %}
Rerun the tests via `script.run_custom_template_test_suite` to ensure
all tests are regenerated and reloaded properly.
- action: persistent_notification.create
data:
title: '{{ title }}'
message: '{{ message }}'
notification_id: custom_templates_test_failures__fwd_disabled
- action: notify.{{ notify_devices }}
data:
title: '{{ title }}'
message: '{{ message }}'
data:
color: '#b30925'
channel: Alerts
importance: high
priority: high
ttl: 0
notification_icon: mdi:test-tube
tag: custom-templates-test-results
- alias: "Report test success"
if:
- condition: template
value_template: >
{{ report_success and failed_tests|length == 0 }}
then:
- action: notify.{{ notify_devices }}
data:
title: All custom Jinja2 template tests passed
message: >
{{ test_results|length }} tests passed successfully.
data:
color: '#42c2f5'
channel: Alerts
importance: high
priority: high
ttl: 0
notification_icon: mdi:test-tube
tag: custom-templates-test-results
- alias: "Emit about completing"
event: templates_test_suite.complete
event_data:
failed_tests: >
{{ failed_tests|map(attribute='1')|list }}
configuration.yaml
automation manual: !include_dir_list automations
shell_command:
write_template_tests: ./cmds/write-template-tests.sh '{{ encoded_json }}'
cmds/write-template-tests.sh
This file also gets marked as executable. Base64 encoding was needed to get
quoted values to the script properly.
#!/usr/bin/env bash
set -e
cd "$(dirname "$0")"
json=$(echo ${1} | base64 -d)
filename=$(echo $json | jq -r .filename)
content=$(echo $json | jq -r .content)
file="../custom_templates/${filename}.jinja"
orig="../custom_templates/${filename}.jinja.orig"
touch "${file}"
cp "${file}" "${orig}"
# echo "args are ${@}"
cat > "${file}" <<EOF
{# automatically generated test file. do not edit. #}
${content}
EOF
diff "${orig}" "${file}" || true
rm "${orig}"
custom_templates/tests_templates.jinja
This one is a bit ugly, but it writes out test files and a runner file to run
all tests. So it’s written in Jinja and outputs Jinja. It also contains an
assertion helper.
{% macro create_test_file(suitename, tests) %}
{% set file = suitename + '.jinja' %}
{% set ns = namespace(str='') %}
{% set ns.str = ns.str + '{% from \'tests_templates.jinja\' import assertion %}\n' %}
{% for macro in tests|map(attribute='macro')|list|unique %}
{% set ns.str = ns.str + '{% from \'' + file + '\' import ' + macro + ' %}\n' %}
{% endfor %}
{% set helpers = tests
|map(attribute='result_config')
|map('default', {})
|map(attribute='preprocess')
|select('defined')
|list
|unique %}
{% for helper in helpers %}
{% set ns.str = ns.str + '{% from \'' +
suitename + '_tests_helpers.jinja\' import ' +
helper + ' %}\n' %}
{% endfor %}
{% for helper in helpers %}
{% set ns.str = ns.str ~ 'iterating for the helper\n' %}
{% set ns.str = ns.str + helper + '\n' %}
{% endfor %}
{% set ns.str = ns.str + '\n' %}
{% set ns.str = ns.str + '{% macro run_suite() %}\n' %}
{% set ns.str = ns.str + '[\n' %}
{% for test in tests %}
{% set description = test.get('description') or '' %}
{% set params_config = test.get('params_config', {}) %}
{% set result_config = test.get('result_config', {}) %}
{% set result =
'none|string' if test.result == none else
'\' + test.result + \'' if test.result is string else
test.result %}
{% set result_typefilter =
'|int' if result is integer else
'|float' if result is float else
'|trim' if result is string else
'|from_json' if result is mapping else '' %}
{% set result_preprocess_wrapper = result_config.get('preprocess') %}
{# create params list as json objects which is a nice way to quote #}
{% set ns2 = namespace(params=[]) %}
{% for param in test.params %}
{% set ns2.params = ns2.params + [
param|to_json|string|regex_replace('null', 'none')
] %}
{% endfor %}
{# build invocation to call macro and obtain result object #}
{% set invocation = test.macro + '(' + ns2.params|join(', ' ) + ')' +
result_typefilter %}
{% if result_preprocess_wrapper %}
{% set invocation = result_preprocess_wrapper + '(' + invocation + ')' +
result_typefilter %}
{% endif %}
{% set params_description =
params_config.get('description') or
ns2.params|join(', ' )|regex_replace('\'', '\\\'') %}
{% set call_description =
test.macro + '(' + params_description + ')' + result_typefilter %}
{% set ns.str = ns.str + ' {{ assertion(' +
invocation + ', ' +
result|string + ', ' +
'\'' + call_description + '\', ' +
'\'' + description + '\') }}' + ('' if loop.last else ',') + '\n' %}
{% endfor %}
{% set ns.str = ns.str + ']\n' %}
{% set ns.str = ns.str + '{% endmacro %}\n' %}
{{ ns.str }}
{% endmacro %}
{% macro create_full_run_file(suitenames) %}
{% set ns = namespace(str='') %}
{% for suitename in suitenames %}
{% set file = suitename + '_tests.jinja' %}
{% set ns.str = ns.str + '{% import \'' + file + '\' as ' + suitename + ' %}\n' %}
{% endfor %}
{% set ns.str = ns.str + '\n' %}
{% set ns.str = ns.str + '{% macro run_all() %}\n' %}
{% set ns.str = ns.str + ' {% set all = {\n' %}
{% for suitename in suitenames %}
{% set ns.str = ns.str + ' \'' + suitename + '\': ' + suitename + '.run_suite,\n' %}
{% endfor %}
{% set ns.str = ns.str + ' } %}\n' %}
{% set ns.str = ns.str + ' {% set ns = namespace(result=[]) %}\n' %}
{% set ns.str = ns.str + ' {% for name, run_suite in all.items() %}\n' %}
{% set ns.str = ns.str + ' {% for pass, description in run_suite()|from_json %}\n' %}
{% set ns.str = ns.str + ' {% set ns.result = ns.result + [\n' %}
{% set ns.str = ns.str + ' [pass, name + \'.\' + description]\n' %}
{% set ns.str = ns.str + ' ] %}\n' %}
{% set ns.str = ns.str + ' {% endfor %}\n' %}
{% set ns.str = ns.str + ' {% endfor %}\n' %}
{% set ns.str = ns.str + ' {{ ns.result }}\n' %}
{% set ns.str = ns.str + '{% endmacro %}\n' %}
{{ ns.str }}
{% endmacro %}
{% macro assertion(result, expected, invocation, description) %}
{% set pass = result == expected %}
{{ [
pass,
invocation + (
'; pass' if pass else
'; fail\nexpected<' +
expected|string|trim + '> != actual<' +
result|string|trim + '>'
) + iif(description, '\n', '') + description,
]|to_json
}}
{% endmacro %}
Your mileage may vary with this, but I thought someone may find it useful or at least mildly interesting.