HASS Yellow migration - points to watch for (MQTT, Zigbee, restore)

Hardware
,
1

Hi,
After receiving a doom-laden email yesterday suggesting customs charges and UK VAT (Brexit - the gift they keeps giving :frowning: ), my Home Assistant Yellow kit arrived unexpectedly TODAY! :partying_face:

Despite lots of great features recently added to HASS making the migration from a RPi4 to Yellow fine, the process was far from seamless, so thought a few learning points might help others.

TL;DR

  • Crowd Supply are delivering HASS Yellow hardware right now. (Nice case and PCB design :+1:t2:)
  • The Zigbee migration tool is great, but is based on having access to both old and new radio hardware at the same time to directly transfer settings. A full HASS OS backup might also work.
  • Several integrations don’t automatically update IPv4 addresses, even if they were detected when first installed (e.g. MQTT)
  • Full backup, and restore seems to loose device location information.
  • Set aside a few hours for testing and reconfiguration - the migration works, but with more complex homes, you will need time.

A real migration

My migration process was as you’d expect:

  • Create a full backup, and save locally on a client machine.
  • (Try the new Zigbee hardware migration tool - more on that later)
  • Swap Z-Wave stick over to the new kit
  • First boot, restore backup
  • Lots of HASS OS and software upgrades
  • Go through all the missing entities, and test all devices

New hardware; New IPv4 address

mDNS hides a lot - typing “homeassistant.local” into a browser quickly resolves an IP address, and connects to the new hardware. Looks seamless to the browser, but as the hardware has changed, your router has allocated a new IP address (likely both IPv4 and IPv6) and you likely have broken configuration that needs intervention.

Here are the settings that needed manual change for me (MQTT):

  • MQTT Add-In Mosquitto broker - new IPv4 address
  • MQTT client devices (Tasmota/ Sonoff/ Shelly/ Octoprint) need manual updates to find the IPv4 of the broker as most config is done via IPv4 and not hostname + mDNS.
    • Note - the Tasmota Integration has a great “VISIT >” button to go straight to the device web interface to change the broker IPv4 address.
    • Note - Last I checked, HASS doesn’t work with IPv6 addresses, so be careful with hostnames as some routers will prefer IPv6 and break MQTT without clear error messages.
  • MQTT Integration - this surprised me as it doesn’t update, and needs a manual change to the IPv4 broker address (even through it was automagically detected when first installed)
  • Tasmota Integration - just worked once the MQTT Integration could connect to the broker again

Zigbee - migration doesn’t work the way I expected (but later recovered…)

The must useful feature in 2022.09 was the addition of a process to transfer Zigbee network keys and even the MAC address (IEEE address) from an old radio to a replacement, removing the need to reset and re-pair all devices to a new coordinator.
Whilst my existing Zigbee radio (a Sonoff 3.0 USB stick) would have worked connected to Yellow, migrating to the new SilLabs “SkyConnect” radio on the PCB seemed a great idea.

If you start the migration tool on the OLD hardware, it will:

  • backup the OLD stick settings (good!)
  • wipe the old stick back to factory settings (err…)
  • Ask for the NEW stick to be inserted (darn, blast, …)

Of course, the new stick is soldered to the PCB of the Yellow so can’'t be connected to the OLD hardware causing much swearing :face_with_symbols_over_mouth: (happy ending later).

The migration process looks to be designed based on this workflow:

  • Create a full backup and remove the old Zigbee stick
  • Restore the backup on new hardware, and connect the old Zigbee Stick
  • Run the Zigbee migration tool with both OLD and NEW radio sticks accessible.

Thankfully, Full HASS backup/ restore must save Zigbee radio settings, so the prematurely wiped Sonoff Zigbee radio settings were able to be restored WITHOUT having to re-pair all devices.

  • Create a full HASS OS backup
  • Restore the full backup on the new hardware
  • Upgrade all OS and HASS versions (took several steps on my Yellow)
  • Delete the restored Zigbee integration (didn’t work as my error erased the old radio back to factory settings)
  • Install a “new” Zigbee integration - this seemed to trigger the migration tool, and the on-board SilLabs Zigbee radio was configured with the MAC and keys from the old Sonoff device. Phew! :tada:
    N.B. The location of the USB radio is not automatically configured - I had to try a few variations of /dev/tty/AMA1 until it was detected (hint - if you are asked for radio type, the path is WRONG)

Although this worked for me, I can’t reproduce the once-only process to properly document it and tell you how to trigger the migration tool from a backup properly - sorry!

Location, location, location

The full HASS backup and restore worked, but was not seamless. The biggest omission found was whilst locations themselves transferred over, devices/ entities such as Zigbee, MQTT, Z-Wave, etc were created WITHOUT location details.
For most entities this was easily fixable (e.g. Z-Wave, MQTT have location in their device config), but perhaps due to my mis-understanding of the Zigbee migration process, I ended up with several identical IKEA 5-button remotes, and Sonoff TH01 temp sensors.
Strangely, the device IDs seemed to be retained meaning those entities with automations could be traced as they were still linked with hints available in properties.

Remote controls are easy to identify - open the Logbook and press a button to see the event.

Telling 4x identical Sonoff TH01 Zigbee temp sensors apart became a challenge:

  • All properties were identical (apart from current temp and battery)
  • Several devices were relatively inaccessible
  • The “Identifybutton” device property usefully flashes a small red LED on the device, BUT this can take over a minute for the device to wake up and poll.

Real patience is needed when using Zigbee “Identifybutton” with battery devices. Don’t be tempted to try another device, as you will end up with two flashing at the same time!

Conclusion

The HASS Yellow is a nice piece of hardware with a well designed case. The NabuCasa team has clearly been migration their own installations to Yellow, as features like the Zigbee migration tool are great (when you know how they are designed…) and very helpful.

I’m not seeing much difference from a RPi4 8Gb and Yellow with a RPi CM4 (2Gb of RAM, no Bluetooh, no WLAN) but:

  • Adding a M.2 SSD for storage and access speed will help in the future.
  • A cheap Bluetooth USB interface adds back what the kit CM4 lacks.
  • The new SilLabs “SkyConnect” radio will support both Zigbee and Thread will be very useful.
  • A 90-degree USB adaptor from Amazon works well with a Z-Wave USB stick, to have the radio stick upwards for maximum range.
  • The UK kit comes with a UK 12V 2A PSU, but as I misunderstood, also ordered a separate PSU which is a UK 12V 3A unit.

Thanks to all who have worked to bring this hardware and software to market - :clap:t2: :clap:t2: :clap:t2:

If this helps, :heart: this post!

18 Likes
2

I received my yellow today and was tearing my hair out at the lack of documentation on setting up the device. Thank you for your guide on getting the zigbee config moved across to the yellow.

I restored my full backup from my current NUC to the yellow, but it was not intuitive that you have to delete the ZHA integration to force it to load the integration from the backup. My yellow also used the /dev/tty/AMA1 location, so detected the SILABs device and loaded the devices / entities. Unfortunately the area associated with each device was missing.

At the time of writing all my entities are not connected, some are showing “blank” status, some are showing “disabled” and a small number “unavailable”. Also the naming convention for entities in my previous instance seem to have been lost so none of my automations are currently working.

I will wait awhile to see if the network recovers and update this reply with any further insights.

3

I’m glad my errors have helped you! :slight_smile:

It’s disappointing to hear that you also lost location tagging of devices, and suggests there is a bug/ missing feature in the restore or migration tool.

Strangely, some of my Zigbee automations kept working as they remained linked to the device which is odd as I tend to use entity_id (name) rather than device_id (GUID) in automations.yaml. This might help you reinstate previous config, but it’s still a PITA to loose bulk config especially with a structured naming convention.

Please do add what you find to this thread as there’s not a lot out there as Yellow is so new:
https://yellow.home-assistant.io/resources/

4

Thanks James, the positive aspects of this migration:

  1. The zigbee device is far more powerful than the conbee device I had, the yellow is currently on the bench in my workshop which is at the back of the house and most connected straight away. The only issue with connection has been a bunch of Aqara leak sensors which are unavailable.
  2. There some additional entities that were not there previously which need some investigation to understand their potential use.
  3. The whole system is far more efficient with about one third of the power requirement my NUC expected.

All my entities had the format device_type.room_description e.g. binary_sensor.utility_co with the device name retaining whatever the default was e.g. HEIMAN COSensor-EM.

Migration resulted in my CO sensor entities from the device name: binary_sensor.heiman_cosensor_em_iaszone

It’s a shame the migration tool did not copy the entities names from the zigbee database as well. So a bit of effort required to go through the 340+ entitites to rename them so the automations continue to function as designed.

5

NVMe Storage
I bought a NVMe disk to supplement the storage provided by the Yellow, on booting I noticed the data was migrated to the eMMC storage. This can be corrected by going into “Settings” > “System” > “Storage”, clicking on the “:” top right of the screen and clicking “Move Datadisk” a dropdown should list the NVMe installed, a reboot and the storage is now on the much larger disk space.

6

FYI, there is another thread discussing M.2 SSD and “Move Datadisk”:

It looks like boot and OS partitions remain on the eMMC - only the data partition is moved.

1 Like
7

As you experienced, only the data partition is moved, not the os. Much of these postings could be edited and added to the Community Guides where tutorials are hosted.

8

Update on progress, my Aqara devices (door and window sensors, leak sensors etc) are dropping off the network some hours after I have reconnected them. I saw this behaviour with my conbee in the early days, it takes multiple reconnections before the network stabilises. So probably not an issue with Yellow specifically.

9

Zigbee has a “network heal” process like Z-Wave doesn’t it?
The process isn’t mentioned in the ZHA docs, although the comment about pairing in the final destination (e.g. not next to the coordinator) was interesting - suggests the mesh is more fixed.

Z-Wave typically runs as a batch job at 02:00AM to avoid such disruption.

My engineer’s reading of the badge “SilLabs IoT Incubator” is still “beta firmware”, so as well as possible dual-stack Zigbee + Thread support (both IEEE 802.15.4), there’s no doubt some optimisations to land as well. I hope the apparent high demand for the Yellow and SkyConnect means Nabu Casa will have the resources they need to produce the best platform possible.

10

My experience with Conbee II was that I had better stability if I tried to onboard a new device through an existing router rather than through the co-ordinator. I don’t know about a “network heal” process. Zigbee I think reorganises itself over time to balance loads on various routers and the co-ordinator. With a max number of devices per router.

Looking into why my devices did not carry their entity names across, the json backup ZHA creates seems to only contain network attributes e.g. network key, node attributes and pan_id. I cannot see any device or entity information within the json. So maybe this is in the HA Backup but it does not seem to get to the new instance of HA. I wonder if you need to first load the ZHA backup to create the network then load the HA backup for entity and device data?

11

Update, I was away over the weekend and all the Xiaomi devices dropped off the network again. While I experienced this early in my HA career using Conbee II and phoscon, ZHA has been relatively solid since moving. Will start pairing them again and see what happens. Why is there no specific Yellow community section and somewhere to report issues?

1 Like
12

Now that sounds like a useful WTH I’d vote for!

2 Likes
14

If possible please help contribute to ZHA/Zigbee integration documentation to cover common scenarios:

https://www.home-assistant.io/integrations/zha

https://github.com/home-assistant/home-assistant.io/blob/0b40534ae1efcb7d55f06cfb50e6b95460ab76cb/source/_integrations/zha.markdown

Be sure to always follow best practices → https://github.com/zigpy/zigpy/wiki/Generic-best-practice-tips-on-improving-Zigbee-network-range-and-general-stability

Your Zigbee network mesh will repair itself automatically given time and I am not sure there is any way (e.g. zigbee or zigpy-cli commands) to manually trigger/initially or speed up the repair process, that would have to be a question for the zigpy/ZHA developers here → https://github.com/zigpy/zigpy/discussions/ and a related question was previously handled here but much have changed since then so might not be the same today → https://github.com/zigpy/zigpy/discussions/614

15

Hello guys,

So I still do not know what to make of the built-in ZigBee module. Supposedly this is the non plus ultra, but compared to the old Smartlight Coordinator, I have significantly more problems :confused:
Does anyone know how I can disable the Zha Discovery? I use the module with Z2M, but get after each restart the ZHA integration offered to install. However, it can not be clicked away or deactivated :confused:

16

Hello guys,

Any update on the issue on the issue that user defined entity id naming is lost after zigbee radio migration? I migrated to the yellow yesterday and migrated the radio from ConBee II to the internal chip and as others reported here: entity ids and names are set to default and area relation is gone. With ~60 devices and ~350 entities this is just not the way to go. So I switched back to ConBee for the moment which restored custom naming after restoring a HA backup.

17

As most of us only migrate once, you’re unlikely to get an update here - sorry!

18

It should work if have the latest firmware on the ConBee adapter and use the latest Home Assistant.

It must be a bug you and need to report issues to → https://github.com/home-assistant/core/issues

btw, noticed few Yellow related PRs → https://github.com/home-assistant/core/pulls?q=is%3Apr+zha

19

Make sure got enough mains-powered Zigbee Router devices (as those act as Zigbee repeaters / range extenders) and wait around 24-hours for all Zigbee Router devices in the Zigbee network mesh to update connections. Generally, battery-powered devices should be paired/joined last when building a Zigbee network from scratch as they usually do not move to Zigbee Router devices connected after them, alternatively re-pair/re-join those battery-powered devices again lastly. Also be sure to follow best practice guidelines → Generic best practice tips on improving Zigbee network range and general stability · zigpy/zigpy Wiki · GitHub

See related new core and frontend pull requests adding unique ID to Yellow hardware for ZHA discovery

Use a unique ID for the config flow when the Yellow is discovered. With home-assistant/frontend#14281, this allows for the ZHA discovery entry to be ignored.

This also correctly sets the name of the discovery card:

image

Add hardware as a config flow discovery source.

Without this change, discovered Yellow cannot be ignored:

image

20

Hi,

I received my yellow recently. I do not find an information how to import an existing mysql/mariadb.

My old system is a homeassistant.CORE installation which I still love. but is there a way currently how to connect to the mariadb-addon (docker) or an import mechanism?

otherwise I would love to install a standard raspian or something else and install homeassistant.CORE like the old way…
thanks
Marc

21

I didn’t mention a separate database server as I don’t use one, sorry. My suggestion would be to start a fresh thread.

As a recovering Oracle DBA, I’ve only ever needed to use SQLite Web to remove bad sensor data.