Migrating to Z-Wave JS broke my whole Z-Wave network and DB (Discussion and Suggestions)

qJake · January 7, 2022, 3:02am

First, let me start by saying that I love Home Assistant. My install has over 300 entities from over 40 integrations, I have multiple wall tablets, and even a HA t-shirt.

But today, I am disappointed.

Today, I tried to migrate to Z-Wave JS. It did not go well.

For some background, my Z-Wave network consists of primarily hardwired nodes (over 40) in the form of GE/Jasco/enBrighten in-wall switches. I have a handful of wireless sensors as well (less than 5).

Here are the steps I took to migrate from the legacy Z-Wave integration (outdated) to Z-Wave JS - note that these instructions can basically be summed up in the form of “I followed the migration wizard.”

Performed a partial backup (I excluded any addons that I didn’t need to back up, like VS Code, InfluxDB, etc). HA itself and all of the configuration folders were included.
Ran the migration tool.
Stopped my old Z-Wave network.
Copied the network key from /config/.storage (Issue 1)
Installed the addon, entering my old network key into the S0 option
The wizard informed me that ALL of my devices couldn’t be migrated. (Issue 2) I proceeded anyway, figuring that my network would still be operational and I’d just have Node1, Node2, …NodeX and have to rename them.
Completed the wizard. The Z-Wave network reported as “Operational” with a green check.

After the wizard, I found all of the entities for node status, with nodes in various states (alive, unknown, sleeping), but no actual control nodes. I couldn’t re-include/re-add the nodes or really do anything. (Issue 3) Effectively, all of my devices were non-functional and I couldn’t easily see the status of all of them or what was going on. I even tried “including” a device that was already on my network. No dice. So I restored my backup.

After backup restoration, I had to manually remove the new Z-Wave JS addon (otherwise both would be running), and then restart HA a second time so that the old integration would work again. After restarting (and rebooting the host, too, since the USB device seemed to be stuck), the legacy Z-Wave integration fails to start (but the entities show up in HA, they just don’t work if you try to toggle them on/off… so the integration is loaded, just not correctly). The error is “Z-Wave not ready after 300 seconds, continuing anyway”.

And to top it all off, my SQLite database was corrupted, so now I’ve lost 2 months of power and solar history (Issue 4).

Now, I did not come here just to complain. I have some constructive feedback that I hope can make this experience better in the future.

Issue 1: Network key migration does not work / is not present in the wizard. If migrating, please pull the old network key automatically and populate it into the S0 field, or at the very least provide instructions on where to find the value on-screen during the wizard. Don’t make the user go into a JSON configuration file to look for it and copy it out.

Issue 2: I have no idea why it told me that ALL of my Z-Wave entities could not be migrated. All of the products I own are standard and from well-known manufacturers (GE/Jasco/enBrighten, and a few Aeotec and Zooz devices) and are not complex. Most aren’t even “Z-Wave Secure” because they’re hard-wired. So there should have been no issue whatsoever in migrating.

Issue 3: I’m now left with a completely broken Z-Wave network, and over 40 devices to try and figure out how to manually exclude/reset and then re-include AND re-setup with the correct friendly names and more importantly, the correct entity IDs so that my automations continue to work correctly. If the wizard told me to take a backup first, and the backup doesn’t work (the Z-Wave integration didn’t restore correctly or isn’t functioning correctly), then why even take the backup in the first place? In this case, the wizard should warn you that this is irreversible and restoring from a backup will not work.

Issue 4: I understand databases get corrupt, and I know that improvements have been made so that Home Assistant doesn’t just completely stop functioning when this happens. But now that we have statistics and long-term measurement storage in the database, it really sucks having to lose all of that data with no (easy) way of getting it back, especially with the new energy dashboard being able to show months or even a year’s worth of data at a time. It would be great if we could configure the statistics data to write to a different database than the standard entity recorder/history integration, so that if the main history recorder gets corrupted, we don’t lose all of our historical statistics.

Again, I Home Assistant, and this is certainly not the end for me - but this whole experience has left a sour taste in my mouth. Having to spend hours and hours to fix this stuff, effectively being “punished” for simply trying to upgrade from a deprecated integration/addon, is frustrating and tiring.

AllHailJ · January 7, 2022, 3:18am

When I converted, I lost everything as well - 89 z-wave devices. Loss was Early on when I knew less about HA than I do now so I can’t blame HA for the trouble - it well could have been me. There is generally a factory reset sequence like turn off power, turn on power and within 1 minute toggle switch 5 times. I don’t remember what it is for enbrighten. I used HA to remove the devices and it removed fine.

I still don’t regret the hassle. z-wave js is faster and better supports the devices. If you load the HA app on your phone at least you can walk around and remove and include right at the switch.

Good luck and I feel your frustration.

qJake · January 7, 2022, 3:20am

Thanks for the info! So you can actually upgrade to ZWave JS with your network in a broken state, and still use the “Remove Node” command successfully?

I didn’t know that, I thought the network had to be functional to remove a node (for security reasons), but maybe I’ll give that a try. Thanks again!

AllHailJ · January 7, 2022, 4:32am

I think secure devices like locks you have to reset.

finity · January 7, 2022, 1:29pm

I really wouldn’t start yet with a factory reset of anything. I’m not sure why restoring from backup didn’t restore the old zwave integration.

I don’t have the time right now to add anything additional but I can in a few hours possibly so if it were me I would hold tight for some further assistance from the zwave gurus.

ski522 · February 6, 2022, 12:00pm

Just gave this a shot myself and found the migration isn’t as smooth as I was hoping. They majority of my nodes were not migrated over. I run HA has a VM, so going to restore the snapshot and maybe try this migration at a later date.

MarcellusWallace · February 17, 2022, 11:54am

I was planning to do the migration also, but reading all of this doesn’t comfort me.
The snapshot restore went fine I hope.

ski522 · February 17, 2022, 2:11pm

Yes, restoring my snapshot brought every thing back it’s original state before the upgrade. One of the main reasons I run HA in a VM. This gives me the chance to tackle the upgrade at a later date.

The migration did discover my devices, but the problem I had is it changed the name for a number of devices in HA which was messing up my automation. Also the migration didn’t discover my door locks which I suspect is security related…hopefully I don’t have to un-pair and re-pair those.

Basically I’m going to need a few hours to clean it up when I do migrate to the newer Z-Wave JS integration which is I guess a question of when and not if.

Also I had followed this guide which was helpful especially about copying your old z-wave info before migrating: Migrate from Home Assistant legacy Z-Wave to Z-Wave JS | David’s Blog

jimh12345 · February 18, 2022, 7:39pm

I’ve been on ZWaveJS since my first Z-wave device and while I currently have everything working, it hasn’t been simple or easy. After a while I realized I’d have been better off with ZWaveJSMQTT because of its enhance UI and its ability to do firmware updates. But after reading posts from people who attempted THAT migration, I’ve decided it’s not for me. It seems possible, but if you make a mistake, you’re in deep weeds and it’s a long road back.

So all I’m offering here is empathy. I also wish I’d never heard of Z-wave and just stuck with wi-fi - but I’d never say that out loud.

My hope now is that ZWaveJS is moving forward and will eventually include diagnostic and maintenance capabilities comparable to ZWaveJSMQTT. Does anyone know if that’s likely?

AllHailJ · February 18, 2022, 7:46pm

Considering that Z-wave js is only about 18 months old and they wrote the first version in less than a month, then I would say they are just getting started. They have been adding features consistently at each new release. This is just speculation on my part but I feel confident.

MarcellusWallace · February 19, 2022, 3:15pm

Hereby is my journey to Z-Wave JS. (Make note of my latest additional information at the end of this post)

Last Thursday I made the change. It was all but a smooth transition. Everybody is talking about the better speed and responsiveness of the Z-Wave JS integration in relation to the legacy Z-Wave integration. But I can’t agree with that.

After starting the migration wizard it told me that it wasn’t able to migrate almost all of my devices. I really don’t see a lot of value in this migration wizard in that part. It helps you install Z-Wave JS and in the end remove the legacy Z-Wave integration but that’s about it.

My Z-wave network was working fine with the legacy Z-Wave integration. I’ve moved over from my Fibaro HC2 controller to HC2 somewhere half 2021 and I found out that the Fibaro was much quicker and more reliable than the Z-Wave integration of HA. But I wanted to consolidate my solutions so I worked around the ‘minor’ issues hoping that Z-Wave JS would solve that in the future. But actually, it didn’t.

Z-Wave JS still is missing some basic features like ‘associations’ (yes, I know… it’s on the roadmap) or just a setting for a global polling cycle (My Fibaro controller had this feature and did calculate the value based on the number of devices you had. In my case it was a global polling cycle of 300sec, this global value could be used at the device level, or you could just add a special interval for a specific device).

Another thing is that I now have some devices that don’t report the status back when operated at the device level, those devices worked fine with the legacy integration. So, I’m not sure what causes that problem, have to use a script now to update the status with “zwave_js.refresh_value”. Those devices are from Fibaro (most of my devices are). It is an older device, a dimmer (FGD211 with firmware version 1.7). I have another one that’s working fine (newer firmware, version 2.2)

I know that Greenwave devices need to be polled to get the power (and other) readings. That was needed also within my Fibaro HC2 controller, so I’ve also used the “zwave_js.refresh_value” service call in an automation rule for that.

I really hope that the (IMHO) basic functionality will be fixed soon.

I’m running HA as VM in Proxmox with the following versions (all latest versions usable at the time of this migration):

Host Operating System: Home Assistant OS 7.4
Supervisor Version: supervisor-2022.01.1
Home Assistant Core version: core-2022.2.8
Z-Wave JS integration: 0.1.54
- Z-Wave JS to 8.11.6
- Z-Wave JS Server to 1.15.0
  - With a Z‐Stick Gen5 USB Controller (AEON Labs ZW090, Firmware: 1.2)

Edit: March 23, 2022.
The problem about devices not reporting back was due to a faulty unit that flooded the network.
That was also the reason why my network responded so very slow.
All is fixed now, and everything responses lightning fast again. Happy user again.

barroomhero · March 2, 2022, 9:58pm

Guess it’s time, whether we’re ready or not.

This release of Home Assistant Core 2022.3 will be the final release that provides these integrations. Both the old zwave and ozw integrations are pending removal for Home Assistant Core 2022.4.

ski522 · March 5, 2022, 10:57am

Yup, decided to make the jump because this is the last release with the old integration.

All my devices were moved over, but the biggest pain was that the all the device ID’s were changed, so I’m currently spending the time to setup my automatons & Lovelace cards again with the right names and right device name.

I would be lying if I didn’t say this was high on the PITA list.

PeteRage · March 10, 2022, 1:11pm

Yeah, I’ve been following this topic for a while, I have two houses both running the deprecated zwave, with the only issue being newer devices come up as unknown. This isn’t a big deal since I’ve gone to scripts anyways to configure the zwave parameters in the devices - to make it repeatable. Anyways, it’s been very reliable when running and had zero problems for a year plus. So,

“if it ain’t broke don’t fix it”

While I know I can run the old code as a custom component, I would like to experiment with migrating, but I am concerned about not be able to roll-back. I’m highly concerned with the opening statement here that says the zwave network would not start and roll back failed. Is there any insight into why that happened? Does zwave-js change the config on the stick or network Config on the nodes?

Thoughts on this being a backup strategy. Using HA container, MariaDB and Aeon Labs Gen 5 stick. This assumes I’ve already updated to latest HA.

Backup the gen5 stick.
Shutdown HA
Duplicate the HA config folder
Create a new container
In new Config disable MariaDB (usually do this by editing connection string to be invalid)
Startup JWAVEJS-MQTT container, get it setup with the stick and network key
Wait for the interview process to complete / go around and wake up nodes
Start up new HA and try migration

If it fails, I can go back to the old container and if zwave fails to start I can restore stick from backup,

Any other advice?

bs87 · April 11, 2022, 4:01am

Attempted the migration tonight to get to 2022.4 and just wanted to add in another tally for the migration breaking a rather large implementation. 99 Z-wave nodes, 300+ Z-wave entities and 0 devices or entities came across. Additionally, every node has only a Ping and Health entity. Re-interview does nothing.

On top of this, restoring from a full backup done about an hour before the migration attempt does not restore to prior state. Z-wave JS integration and add-on remained preventing Z-wave from starting and Z-wave contained a note that the migration was already completed. This makes me completely distrust the built-in backup\restore function.

Luckily, I do hypervisor level backups and was able to do a full VM restore until I can dedicate probably a full day to try again and exclude\include + rename everything back.

Very frustrating indeed. IMO there should be very obvious disclaimer in the migration documention to not rely on Home Assistant backups and that you may be rebuilding Z-wave from scratch.

dlthompson06 · April 13, 2022, 12:27am

I thought I had it bad with my 53 Z-wave devices. 99…yikes! I finally got around to trying the migration today (via ZwaveJS2MQTT) and with a lot of trial and error and a bit of blind luck, I was able to get the devices I wanted switched over (I had several dead nodes that I was happy to leave behind). However, I lost all of my entities and am left with only the Ping and Health entities for each node, so I have no idea where to go from here except maybe to remove each node and manually add it back again. What a pain. I love Home Assistant (came over from Smart Things a few years ago) and don’t see myself ever going back, but I’m disappointed in this implementation right now.

Mattie · April 13, 2022, 8:16am

Keeping an eye in this thread, I dont dare to migrate after reading this. Has anyone done the migration smoothly?

As I feel now I have to plan a weekend for doing this with my danalock and all other deviced. It doesnt feel like the will be migrated smoothly atm

bs87 · April 17, 2022, 11:29pm

Ok, after 4 migration attempts (and full VM restores) I was sort of successful on my fifth attempt. This appears to be mostly just poor documention online and in the wizard. Piecing together snippets of information from various posts I had several failure points.

First attempt: I used an old S0 key that was incorrect, my fault. Found my correct key in /.storage/core.config_entries.
Second attempt: The old Z-wave key was in the 0x12 0x34 etc format and the new integration and migration wizard wants that key in the 1234 format. I didn’t see the required format listed in the documentation or wizard and found it in some thread on here.
Third attempt: Something went wrong here and I’m not sure what. When I clicked stop network, HA rebooted. Tried again and many nodes were unavailable. I believe this attempt failed because I attempted the migration before the old Z-wave had finished starting and interviewing devices.
Fourth attempt: This succeeded but I ended up with no entities and devices were named by Node #. At this point I fiddled around with the Z-waveJS2MQTT control panel and realized after a while entities were starting pop back up (not named correctly) and devices were finally being detected correctly (eg GE In-wall Dimmer, etc).

In my last attempt before throwing in the towel and just redoing everything I tried just being patient.

Start the migration
Wait for any line powered devices to show as ready via the old integration
a. Pretty much all of my battered powered devices said not ready and would never become ready despite waking them up (and successfully performing actions with them)
Stop network
Install the new + supervised add-on via migration
Wait at the screen that says 5000 devices wont be migrated. DON’T CLICK MIGRATE. I just let it sit at this screen for about an hour. Nothing changed on this menu but when I finally clicked migrate, I suddenly had almost all of my nodes correct and entities were correct.

I had to exclude and re-include all of my battery powered stuff but luckily that is only 4 minimotes, a wallmote, and a few motion sensors. I had 1 of 30+ GE switches come across but lost naming, easy enough to fix. All other GE switches came through fine. Also, all of my Inovelli Red switches came across with lost names, easy enough to fix.

So, I am officially migrated now without having to tediously redo everything but documention both in the migration guide and migration wizard should be greatly improved to reduce frustration.

Side note: For renaming things that fail, a good resource is the migration JSON that gets generated at /.storage/zwave.legacy_zwave_migration. It lists entity to node associations to get them back to what automations expect.

AllHailJ · April 18, 2022, 3:54pm

I would also recommend that the HACS integration WATCHMAN be installed. This will check all automations to make sure you don’t have misnamed entities and devices. It really is handy especially as your system grows.

Mattie · April 19, 2022, 8:51am

Great info! How did you convert the S0 key in the end? My key is written like: 0xXX, 0xXX, and so on 16 times. I guess you remove the commas?

Another thought, is it possible to use the deprecated zwave and the new Zwave JS integration at the same time? In that case I could just screw the migration tool and move my devices one by one as I want myself without having to worry about wrecking the whole thing.