Allow blueprint upgrades

Much of what you are asking for has a related working example to use as a start point. HACS.
I would perhaps try to keep the rev contained in the blueprint and try to not restrict to just GitHub, but it would be a nice to have for sure.
I’m sharing 6 blueprints in GISTS, and wouldn’t look forward to having to have 6 more repo’s to hold those, but it could be done. Then there are the ones like CarPresence and others that have multiple blueprints in the same repo.
UPDATE 2022-1-11: I moved all my stuff to a REPO now as it is easier to maintain.

2 Likes

I think the HACS model is the right direction. I don’t think one would need separate repos, simply setup the version.json (or yaml or whatever) such that there are entries for each blueprint in a given repo. For example:

{
  "some_blueprint": {
    "current_version": "1.2.3",
    "blueprint_url": "https://github.com/somebody/somerepo/some_blueprint.yaml"
  },
  "some_other_blueprint": {
    "current_version": "4.5.6",
    "blueprint_url": "https://github.com/somebody/somerepo/some_other_blueprint.yaml"
  }
}

From a performance perspective, having a single version file for a repo full of blueprints would save a lot of update checks.

10 Likes

Voting for this myself and just adding a note so I can keep updated.
Recently I started getting script errors in the log file and these were related to blueprints I am using for HA SwitchPlate so obviously I need to update these.
As there was no notification of updated blueprints and no easy way to update them, I didn’t know about any updates until I started getting errors and then manually looked.

Thanks :slight_smile:

5 Likes

Up, would be nice

1 Like

+1 definately a must have! Actually, when blueprints were introduced they were advertised as the first step with improvements like updating coming later. But they never actually came.

I made a shell script to update blueprints…
I don’t use a lot of blueprints, but figured this would be easier then checking them manually from time to time.
It should probably send a notification or something when a blueprint is updated, but didn’t bother to add that yet.

if you have a script which doesn’t have a source_url in the source file, add it to your local copy with “custom-” in front of the url source_url: custom-https://example.com/source.yaml the script will then use that url, and re-insert the source url after downloading.
edit: if the newly downloaded file has a source_url, the custom url isn’t inserted

Create the following script in the /config/scripts/ directory

/config/scripts/blueprints_update.sh

#!/bin/bash

DeBuG="false"
function _blueprint_update_debug
{
	if [ "${DeBuG}" == "true" ]
	then
		echo "$@"
	fi
}

cd /config/blueprints/
for file in ./*/*/*.yaml
do
	echo "> ${file}"

	# get source url from file
	blueprint_source_url=$(grep source_url $file | sed s/' *source_url: '//)
	_blueprint_update_debug "-> source_url: ${blueprint_source_url}"

	# check for a value in source_url
	if [ "${blueprint_source_url}" == "" ]
	then
		echo "-! no source_url in file"
		echo
		continue
	fi

	# check for custom source_url (the source_url doesn't exist in the source file)
	custom_source_url=""
	if [ "$(echo "${blueprint_source_url}" | grep '^custom-')" != "" ]
	then
		_blueprint_update_debug "-! custom source_url set"
		custom_source_url="${blueprint_source_url}"
		blueprint_source_url="$(echo "${blueprint_source_url}" | sed s/'^custom-'//)"
	fi

	# fix source if it's regular github
	if [ "$(echo "${blueprint_source_url}" | grep 'https://github.com/')" != "" ]
	then
		_blueprint_update_debug "-! fix github url to raw"
		blueprint_source_url=$(echo "${blueprint_source_url}" | sed -e s/'https:\/\/github.com\/'/'https:\/\/raw.githubusercontent.com\/'/ -e s/'\/blob\/'/'\/'/)
		_blueprint_update_debug "-> fixed source_url: ${blueprint_source_url}"
	fi

	# check filename is the same
	if [ "$(basename ${file})" != "$(basename ${blueprint_source_url})" ]
	then
		echo "-! non-matching filename"
		_blueprint_update_debug "-! [$(basename ${file})] != [$(basename ${blueprint_source_url})]"
		echo
		continue
	fi

	_blueprint_update_debug "-> download blueprint"
	wget -q -O /tmp/blueprint_update.yaml "${blueprint_source_url}"

	# insert the custom url, if it was in the original, and is not in the newly downloaded file
	if [ "${custom_source_url}" != "" ]
	then
		# check for source_url in the new source file
		new_blueprint_source_url=$(grep source_url /tmp/blueprint_update.yaml | sed s/' *source_url: '//)

		if [ "${new_blueprint_source_url}" == "" ]
		then
			_blueprint_update_debug "-! re-insert custom-url"
			sed -i "s;blueprint:;blueprint:\n  source_url: ${custom_source_url};" /tmp/blueprint_update.yaml
		fi
	fi

	_blueprint_update_debug "-> compare blueprints"
	blueprint_diff=$(diff ${file} /tmp/blueprint_update.yaml)
	if [ "${blueprint_diff}" == "" ]
	then
		echo "-> blueprint up-2-date"
	else
		echo "-! blueprint changed!"
		cp /tmp/blueprint_update.yaml ${file}
		need_reload="1"
	fi

	echo
done

if [ "${need_reload}" == "1" ]
then
	echo "! reload automations"
	hass-cli service call automation.reload
fi

And create an automation to be able to start the update.
you could let it run every couple of hours or days, or only on manual trigger…

alias: _Blueprints Update
description: ""
trigger:
  - platform: time_pattern
    hours: "3"
condition: []
action:
  - service: shell_command.blueprints_update
    data: {}
mode: single
2 Likes

Cool. Thanks for this. It is a good start.

Couple of issues.

  1. It breaks if there are spaces in the title of the blueprint. Can’t parse the title thru spaces.
  2. Need to make the shell command executable or it won’t run, so chmod 755 the script file in your instructions.
  3. Need to add this as well in configuration.yaml so the shell command will work inside HA.
shell_command:
  blueprints_update: /config/scripts/blueprints_update.sh
  1. it appears to update the Blueprint as well. Feature request be able to run this without updating. I don’t want blueprints updating every few hours by themselves. I want to be there… But I would like to know if the blueprint is out of date checked every couple of hours kind of thing.
  2. It didn’t seem to look in the scripts blueprint folder.
  3. It appears to me that even though they may possibly have a valid filename, blueprints that reside in the blueprints exchange and were imported from there originally do not seem to work. These seem to get -! non-matching filename

My output of the script in a terminal:

 /config git:(master) ✗ ./scripts/blueprints_update.sh
> ./automation/aderusha/MQTT_Tasmota_CarPresence.yaml
-! blueprint changed!

> ./automation/hanerd/zwave-js-schlage-lock-user-code-actions.yaml
-! non-matching filename

> ./automation/raz0rf0x/zwave-js-kwikset-910-manual-status-updater.yaml
-! non-matching filename

> ./automation/SirGoodenough/AutoFanControl HA_fan.yaml
grep: ./automation/SirGoodenough/AutoFanControl: No such file or directory
grep: HA_fan.yaml: No such file or directory
-! no source_url in file

> ./automation/SirGoodenough/AutoFanControl_MQTT.yaml
-! blueprint changed!

> ./automation/SirGoodenough/door_open_tts_cloud_say_announcer_nabu_casa_required.yaml
-! blueprint changed!

> ./automation/SirGoodenough/keypad_5_button_cipher_to_turn_on_something.yaml
-! blueprint changed!

> ./automation/SirGoodenough/Zigbee2MQTT - Xiaomi Cube Controller.yaml
grep: ./automation/SirGoodenough/Zigbee2MQTT: No such file or directory

i set this up some time ago, and already thought that the command needed to be made known in the configuration.yaml but didn’t look it up

i’m going on vacation in 2 days, so i don’t have a lot of time to make changes to this script, but maybe i’ll make it into a project after my vacation…

1 Like

I think what’s going on with the Blueprint Exchange ones is the way the Bluetooth import works. You point it at an exchange post, and it pulls out all the code from it. Therefore there is no actual filename like there is with ones from GitHub. So there’s no actual filename to match. You would have to grab the whole post that it points to and parse that for grave accents or something really messy I think…

that sounds really messy indeed…

in the meantime, i made a couple of updates to my script…
1- fixed, it works with spaces in both filename and source-url
4- i added --debug and --update, by default it doesn’t update anything anymore
5- not sure why that would happen, i changed the file-finding (because of point 1) so maybe it works now
6- if you can reply with a source_url, i can look at it, maybe there is an easy work-around

i also posted the script on gist, so you can get the new version from there…

So the spaces inthe filename was halting the script. I removed 2 blueprints and it got to the end and did the scripts.
Every blueprint exchange based blue print doesn’t find the file, like these:

Also it got to the end of the script now, the automation reload command isn’t functioning.


! reload automations
./scriptshell/blueprints_update.sh: line 88: hass-cli: command not found
Process Completed

(I added a Process Completed echo so that I know when the script was done running manually…)

Also I’m thinking the automation reload command has changed from when you wrote this.

I’ll load your latest and give it a try shortly.

i remembered that i have internet, and can look at the blueprints exchange myself :stuck_out_tongue:

the few i looked at use a github gist url, so i added a feature to recognise those and make a correct raw url out of them… since by default they don’t include the filename, or the raw url.

the 2 you link to indeed only have the code in the forum-post, i’ll look into that after my vacation, there is probably a way to get just the first post, which makes filtering easier, since home assistant itself also gets the content from the post, so if i search the home assistant code i should be able to find how they do that…

i removed the hass-cli call at the end of the script, since it wasn’t working anymore, and i couldn’t get it to install easily on my home-assistant server… so now it’s just a message that you should reload.
you also needed to make a long-lived access key to be able to do that, so that would only make it a lot more complex for now

next step is probably figuring out how to create some sort of notification that there are scripts that need tot be updated.
maybe that should be handled in the automation, based on a result from the script, i don’t know yet.

Just giving feedback. If you don’t want to work on it now that’s not a problem…
This still complaining.

> ./automation/SirGoodenough/Zigbee2MQTT - ZemiSmart ZM-RM02 Controller.yaml
-> source_url: https://github.com/SirGoodenough/HA_Blueprints/blob/cbcc9e3541c1239c3dda7d08252e875ffd9ab8e7/Automations/Zigbee2MQTT%20-%20ZemiSmart%20ZM-RM02%20Controller.yaml
-! fix github url to raw
-> fixed source_url: https://raw.githubusercontent.com/SirGoodenough/HA_Blueprints/master/Automations/Zigbee2MQTT%20-%20ZemiSmart%20ZM-RM02%20Controller.yaml
-! non-matching filename
-! [Zigbee2MQTT - ZemiSmart ZM-RM02 Controller.yaml] != [Zigbee2MQTT%20-%20ZemiSmart%20ZM-RM02%20Controller.yaml]

Also if the shell can return data in a meaningful way to the HA automation, we can service call to update the automations.

that’s why i just never use spaces in my filenames, it’s always such a hassle when scripting… and a underscore looks just as good for readability.

problem kinda is that i find it fun to work on this sort of stuff, but it can eat a lot of time…

i searched how HA does the import, and apparently you can get a forum-post in json if you add .json to the end, and then they just extract the code-block from the first post.
so i implemented the same, just with only jq, grep and sed commands, instead of a html parser…

i tested it with the kwikset blueprint, and that seems to work, i left a debug echo in there which just prints the downloaded/parsed blueprint on the command line, so if you run the script you should see at least 1 correct blueprint

1 Like

maybe the file-name check is also just silly, and is better off removed, or made optional

I can just change the names on the ones I own, but I’m not everybody…
The only one I have now that doesn’t work is the one that also doesn’t import from the exchange because the author has it messed up. More than one code block in the top message.

seems that adding custom- in front of the url gives an error, but i already figured i can fix that by just checking if there is a source_url in the downloaded file, and if there isn’t insert the one from the old file…
then there should be no more need for the custom- part

i just made a couple commits, removing the custom- part if it is in a file, and re-inserting it if it isn’t there in the downloaded file.
also made the difference in filename just a notification, and keep on going with the update-check instead of going to the next file

The custom- part was working for me, as Luma (CarPresence, HASPone) does not believe in adding the source url in the blueprint because he is upset that HA overwrites what he puts in there.
I’ll give the new version a whirl shortly.
We’ll see how the new version works.

was working for me as well, but after i updated HA it gave an error about being malformed…

and there is not really a need for it since i can just put it back in after downloading if it isn’t there