Lovelace based Scene Generator using Scene Gen

Despite all the awesomeness that is Home Assistant, I have always found that scene functionality is a little over complicated from a GUI perspective, with no clean way of being able to save the current state of lights into a scene. I realize that something like this is in the works for the upcoming ‘Simple Mode’ (Simple Mode in Home Assistant 1.0 | Home Assistant Developer Docs), but thought I’d have a bash (no pun intended) at it anyway.

After lots of googling I found the excellent external tool ‘Scene Gen’ (https://www.home-assistant.io/docs/ecosystem/scenegen/) which can capture the state of any given set of lights and switches (I’m focusing on lights only for now) and output a scene.yaml file with those states for use later as a scene.

Here’s what I’ve created in Lovelace:
sceneGen%20Lovelace

Under the hood:
Disclaimer: I’ve only been running HA for about 6 months, so am still learning. If anyone can make the below steps cleaner or more efficient, I’d much appreciate it!

  1. Get scenegen.py to work in hassio
    So the first step is to clone the scenegen respository into your config folder. As we will be using the shell_script component, this was the only way I could get HA to be able to access the scenegen.py script as the component is pretty locked down and doesn’t seem to allow access to the host file system (I’m running hassio in docker). Being a python script, scenegen requires the import of some dependencies for it to work (requests and pyyaml), which it would appear are not available in hassio shell, which I found out by trying to run scenegen.py directly as a shell command using HA’s configurator addon (in the top right of the addon, click the gear icon then ‘execute shell command’) and I got errors saying something like ‘module not found: requests’.

To get around these errors, I just installed each missing module by typing the following into the execute shell command box:
For requests, “pip install requests”
For yaml, “pip install pyyaml”

Please note that from what I understand, these dependencies do not persist on host restarts of hassio, but at this stage we just want to ensure scenegen is running correctly so that doesn’t really matter. I seem to get around this with a bash script (details of this in later steps).

Execute shell command
Command executed: ./scenegen.py: 1
Traceback (most recent call last):
  File "./scenegen.py", line 9, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
Command executed: pip install requests: 0
Collecting requests
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl (125kB)
Collecting idna<2.9,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl (154kB)
Installing collected packages: chardet, urllib3, idna, certifi, requests
Successfully installed certifi-2019.9.11 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.6
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Command executed: ./scenegen.py: 1
Traceback (most recent call last):
  File "./scenegen.py", line 10, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'
Command executed: pip install pyyaml: 0
Collecting pyyaml
  Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
Building wheels for collected packages: pyyaml
  Building wheel for pyyaml (setup.py): started
  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl size=44103 sha256=17e6ffb42a929ea1a55a9b10bacd4e6395a8a98f86710b1994816b99b4c9e75b
  Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
Successfully built pyyaml
Installing collected packages: pyyaml
Successfully installed pyyaml-5.1.2
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Command executed: ./scenegen.py: 2
usage: scenegen.py [-h] [-k KEY] [-s SCENENAME] [-m MAPFILE] [-f FILTER]
                   [-c {xy_color,rgb_color,color_name,color_temp}] [-t TYPES]
                   [--no-sslverify] [--cacerts CACERTS]
                   url
scenegen.py: error: the following arguments are required: url

As you can see from the final output above, scenegen.py is ready to be fed some arguments.

Scenegen has a lot of options, so I would advise reading up on how it works from its own docs. However for our purposes we just need to feed it the following arguments:

./scenegen.py https://someurl.ui.nabu.casa -k someLongLivedAccessToken -m somemapfile.cfg --scenename somescenename -f groupsToFilter >someSceneName.yaml

To break this down:
https://someurl.ui.nabu.casa
The first argument is simply the address to find your HA instance. I switched from Duckdns to NabuCasa for simplicity sake and of course to support the awesome HA project, however I don’t see why this wouldn’t work if you are using duckdns or even just your internal ip address (untested).

-k someLongLivedAccessToken
You need to create a token for scenegen to be allowed access to HA. Follow instructions here Authentication API | Home Assistant Developer Docs

-m somemapfile.cfg
You need to create a map.cfg file in the same folder where scenegen.py is located. This cfg will contain a list of your lights and their groups. See here for how to format the mapfile https://www.home-assistant.io/docs/ecosystem/scenegen/#maps-and-filters

-f grouptofilter
Following on from the mapfile, this allows you to select the specific light groups from the mapfile that you are interested in creating a scene out of.

somescenename.yaml
This is the final piece of the puzzle, and instructs scenegen to create a .yaml file with the states of all the entities that you are interested in.

So to fully test the functionality, using the ‘run shell command’ menu item, I ran the following:

./scenegen.py https://someurl.ui.nabu.casa -k someLongLivedAccessToken -m mapfile.cfg --scenename "bedroom warm" -f "Bedroom Lights" >"bedroom warm.yaml"

The result was the creation of ‘bedroom warm.yaml’ in my config directory:
bedroom_warm

  1. Create a bash script to run from lovelace provided arguments.
    As I mentioned above (but please correct me if I’m wrong), the pythin dependencies get wiped when rebooting the hassio host, so simply creating a shell command with scenegen.py won’t cut it. Through lots of trial an error however I found that if you run it with python3 in a bash script called from shell_command, it seems to work. I have no idea why, but it does!

So I created a file in my config directory called ‘exportSceneGen.sh’ with the following:

#!/bin/bash
SCENENAME="$1"
ROOM="$2"
echo $SCENENAME >>exportSceneGen.log
echo $ROOM >>exportSceneGen.log
python3 "scenegen.py" https://[myHomeAssistantAddress].ui.nabu.casa -k [myLongLivedAccessToken] --scenename "$SCENENAME" -m map.cfg -f "$ROOM" >scenes/"$SCENENAME".yaml

You can of course omit the log output once everything is working.

We can now pass SCENENAME and ROOM into the scenegen.py script by simply running the command:

./exportSceneGen.sh "bedroom warm" "Bedroom Lights"

and a new file ‘bedroom warm.yaml’ is created in your /config/scenes directory. (To make HA pick up scenes in this directory you must use the !include command in your configuration.yaml as described here https://www.home-assistant.io/docs/ecosystem/scenegen/#advanced-usage)

  1. Create your HA entities to allow lovelaced based scene creation.
    Create the following entities in your configuration.yaml (based on your own light groups of course):
input_select:
  scenegen_room_select:
    options:
      - Bedroom Lights
      - Bathroom Lights
      - Wardrobe Lights
      - Courtyard Lights
      - Balcony Lights
      - Living Room Lights
      - Dining Room Lights
      - Office Lights
      - Kitchen Lights
      - Garden Lights
    name: Scene Gen Room Selector
    initial: Office Lights

input_text:
  scenegen_scene_name:
    name: Scene Gen Scene Name
    initial: newscenename

And finally define your shell_command:

shell_command:
   scene_gen_save: 'bash "exportSceneGen.sh" "{{ scene_name }}" "{{ room }}"'

Last of all, we just need to create the script to tie this all together.

Sample of scripts.yaml
alias: Scenegen Save
  sequence:
  - data_template:
      room: '{{ states("input_select.scenegen_room_select") }}'
      scene_name: '{{ states("input_text.scenegen_scene_name") }}'
    service: shell_command.scene_gen_save
  - service: scene.reload
  - data_template:
      message: Scene name '{{ states("input_text.scenegen_scene_name") }}'
      title: New Scene for '{{ states("input_select.scenegen_room_select") }}'
    service: persistent_notification.create

All that’s left now is to add the above elements to lovelace as follows:

And that’s it. Bob’s your father’s uncle! Now you can set up your lights how you want, save them as a scene and they should become immediately available to home assistant.

If you’ve made it this far, congrats on following what I can only imagine is a very long winded approach to achieve GUI Scene simplicity. I only just finished creating this so I’m sure there’s more efficient ways to achieve the same thing. Would of course welcome any feedback, specifically with these ‘nice to have’ improvements

  • Automatically build map.cfg file based on HA defined light groups. No idea how to achieve that
  • Avoid having to hard code the long lived access token into the Shell Script. Ideally as this is designed to run on the same host as HA, it would be nice to remove the need for authentication entirely (but maintain security)
  • Auto populate the input_select.scenegen_room_select with all light groups.
  • Find a more efficient way to run scenegen.py without having to use any bash script.

Thanks for you time :slight_smile:

4 Likes

Awesome work, Im going to be trying this at the weekend. Shame there isnt an easy way to parse areas into the map file…
in the config/.storage folder are two files that are relevant I think, core.area_registry and core.devices_registry, I am bringing them into excel to manipulate to make a map file, its kinda ugly and a manual process, so if someone can write a plugin to do this it would be cool!

Ive come unstuck…

Error running command: bash "exportSceneGen.sh" "{{ scene_name }}" "{{ room }}", return code: 2

NoneType: None

In my scenes folder, I get two empty files, a .yaml file and a weirdly named LoungeHalloween?.yaml (not sure where the ? came from)

Weird. How did you run the command, and where did you see that error?

The error is in Developer tools/logs.
If I run the script manually, substituting the variables for real values, the script runs, just doesn’t like running from HA. i even put the full path to scenegen.py and map.cfg in, no dice :frowning:

#!/bin/bash
SCENENAME=“$1”
ROOM=“$2”
echo $SCENENAME >>exportSceneGen.log
echo $ROOM >>exportSceneGen.log
python3 “/root/scenegen/scenegen.py” https://ha.mydomain.com -k MYAPIKEY --scenename “$SCENENAME” -m /root/scenegen/map.cfg -f “$ROOM” >scenes/“$SCENENAME”.yaml

is how my script looks

Getting closer, I forgot to clone it into /config
It now runs, but the file created is weird and contains no entities
image

My map.cfg looks like this:

[Master Bedroom]
light.jimmy_bedroom:
[Office]
light.office_strip:
light.office_light:
light.012003156001946967c4:
light.0120031560019469633b:
light.extended_color_light_1:
light.extended_color_light_1_2:
light.251266732cf43204ac52:
light.office_main:
[Outside Front]
light.hue_outdoor_wall_1:

What does your scenegen save script look like? Exactly how I wrote it?

Might as well post each part of your setup. Could you paste your input_select.scenegen_room_select, input_text,scenegen_scene_name, shell command.scene_gen_save and script.scenegen_save ?

Also please paste the scenegen.log file that should have been created in your /config folder

As I said I’m far from an expert, but will have a go at making it work.

Also, how are you running HA?

Firstly, thanks for the help :slight_smile:
Running Hassio on Ubuntu

ExportSceneGen.sh:

#!/bin/bash -c
SCENENAME=“$1”
ROOM=“$2”
echo $SCENENAME >>exportSceneGen.log
echo $ROOM >>exportSceneGen.log
python3 “/config/scenegen/scenegen.py” https://ha.MYDOMAIN.com -k MYAPIKEY --scenename “$SCENENAME” -m /config/scenegen/map.cfg -f “$ROOM” >scenes/“$SCENENAME”.yaml

My scripts.yaml:

  zigbee2mqtt_rename:
    alias: Zigbee2mqtt Rename
    sequence:
      service: mqtt.publish
      data_template:
        topic: zigbee2mqtt/bridge/config/rename
        payload_template: >-
          {
            "old": "{{ states.input_text.zigbee2mqtt_old_name.state | string }}",
            "new": "{{ states.input_text.zigbee2mqtt_new_name.state | string }}"
          }
  zigbee2mqtt_remove:
    alias: Zigbee2mqtt Remove
    sequence:
      service: mqtt.publish
      data_template:
        topic: zigbee2mqtt/bridge/config/remove
        payload_template: "{{ states.input_text.zigbee2mqtt_remove.state | string }}"
  scenegen_save:
    alias: Scenegen Save
    sequence:
     - data_template:
         room: '{{ states("input_select.scenegen_room_select") }}'
         scene_name: '{{ states("input_text.scenegen_scene_name") }}'
       service: shell_command.scene_gen_save
     - service: scene.reload
     - data_template:
         message: Scene name '{{ states("input_text.scenegen_scene_name") }}'
         title: New Scene for '{{ states("input_select.scenegen_room_select") }}'
       service: persistent_notification.create

From configuration.yaml:

input_number:
  off_dimmer:
    name: Off Brightness
    initial: 5
    min: 1
    max: 254
    step: 1
# Input select for Zigbee2mqtt debug level
input_select:
  zigbee2mqtt_log_level:
    name: Zigbee2mqtt Log Level
    options:
      - debug
      - info
      - warn
      - error
    initial: info
    icon: mdi:format-list-bulleted
  scenegen_room_select:
    options:
      - Living Room
      - Kitchen
      - Bedroom
      - Office
      - Garage
      - Master Bedroom
      - Maisies Bedroom
      - Amys Bedroom
      - Back Lobby
      - Hall and Stairs
      - Outside Front
    name: Scene Gen Room Selector
    initial: Office

input_text:
  scenegen_scene_name:
    name: Scene Gen Scene Name
    initial: newscenename
  zigbee2mqtt_old_name:
    name: Zigbee2mqtt Old Name
  zigbee2mqtt_new_name:
    name: Zigbee2mqtt New Name
  zigbee2mqtt_remove:
    name: Zigbee2mqtt Remove

# Scripts for renaming & removing devices
shell_command:
   scene_gen_save: 'bash "exportSceneGen.sh" "{{ scene_name }}" "{{ room }}"'

Log files: (which also get a weird name)
image

And Lovelace:

Ok, fixed it, for completeness (if it helps anyone else) the issue was down to extra special characters in the shell script. Fixed by running
awk '{ sub("\r$",""); print}' exportSceneGen.sh > exportSceneGen.sh2
from the command line to remove the erroneous characters. then deleting the original and renaming the exportSceneGen.sh2 to exportSceneGen.sh

sorry for the late reply. I didn’t realise I don’t get an email notification for each new reply.

How did you write the bash script? If it was in sublime text, then you need to ensure the line endings are set to ‘unix’ (View --> line endings --> Unix). This would explain the wierd names ‘/r’ as that is the windows symbol for ‘carriage return’ when sublime text is set to windows format I think.

Just for good measure I would also try removing the paths for the mapfile, as I believe HASSIO considers your config folder to be root. Again I could be wrong on this so maybe try that after testing the UNIX line endings.

That may explain the issue. Let me know!

oops sorry again didn’t see that you identified the issue. So its all working now?

1 Like

Yes, working really well :slight_smile:

@theCheek many of the links you posted are dead. Can you please take a look?

I haven’t gotten this to work yet, but one addition that would be nice would be to generate a matching “is this scene on” binary sensor. It should just cycle through each device in the scene and see if it’s status matches what is defined in the yaml.

Hi mate. Yes you are right it seems that the scene gen python project link died. A quick google found the github here: https://github.com/home-assistant/scenegen

However I made this due to HA not having such functionality at the time. But now that it does, I would really advise using the official implementation found in Configuration --> Scenes using the documentation found here: https://www.home-assistant.io/docs/scene/editor/

I think I may be missing something in all that… Maybe you all need a lot more than I do, but I just use a card with templating and a HACS plugin that allows css styling:

type: markdown
style: >-
  ha-markdown {display:block !important;overflow:scroll;height:150px
  !important;}
content: |-
  \`\`\`
  - id: '<<uniqueID>>'
    name: <<scene name>>
    entities:
  {% for thisLight in states.light %}
  {%- if not thisLight.attributes["entity_id"] -%}
  {{"    "}}{{thisLight.entity_id}}:
        state: '{{thisLight.state}}'{% for attr in thisLight.attributes %}
        {{attr}}: {{thisLight.attributes[attr] | to_json()}}
  {%- endfor %}
  {% endif %}{%- endfor -%}
  \`\`\`
title: All lights YAML

(Note: the \`\`\` above should just be three backticks, without slashes between them, I don’t know how to show that in a code block without breaking the block formatting.)

No python, no bash required. This gives a lovelace card with a textarea containing YAML describing the current state and every current attribute and value of every individual light (but not lights containing several light entities) which can be copied and pasted directly into scenes.yaml without modification except to fill in a unique ID and a scene name.

image

If you don’t want to install the style plugin, you can skip it, just omit that style: >- css block, and you’ll just get the yaml displayed on a lovelace card. The textarea makes it easier to select for copying, though, by clicking into it and hitting the keystroke for select all (ctrl-A or cmd-A).

I find for some reason that when I edit scenes using the scene editor, the “save” button often doesn’t actually do anything, so this is how I do all my scene editing. Works perfectly fine.