Hahahaha… thanks @Ciwyn … I consider imitation the highest form of flattery. 
BTW, you better buckle in then, because I’ve done something rather interesting since my last post.
As previously mentioned above in the past, before I began the hardware rebuild - I knew I needed to get my primary HA controlling Pi moved up to the latest current releases and off of version 0.53. But, I’d hit the Python2.7 --> Python3 wall in terms of upgrading directly on the Pi itself, but couldn’t upgrade HA until I upgraded Python which was sure to break everything (high potential of risk.)
Additionally, there was another operational detail that was becoming a real PITA. Restarts of HA were taking anywhere from 10-15 minutes, thanks largely due to a massively large config file & lots of checks and scans being performed at startup. And this was on a very recent RaspberryPi 3+. And having to step through each major version of HA on path to 0.7x goalposts to figure out the various “breaking changes” quickly along the way … meant this wasn’t going quickly at all. I was using a secondary RaspberryPi as my ‘evolution server’ in this process…but as I said, it was a PITA.
Then, something fortunate happened - I had another computer failure in the house for another service I have running, and it’s replacement got me to thinking about VM’s and the possibility and benefits to run HA on a VM, instead. And on top of that…I learned about the new OpenSource project to reclaim control of XenServer, called XCP-ng, from the same group that develops Xen Orchestra. We’d used XenServer at my former employers, but I wasn’t responsible for running it & never sat one up myself… so I thought I’d give that a try over VMware’s ESXI, esp since it’s OpenSource.
Dug out a couple of old Laptops (circa 2010) that had VTd / Virtualisation support, slapped in 8GB and a 120GB SSD and started installing XCP-ng and building Xen Orchestra Community Edition and then Home Assistant on a bare min Debian 9 install, and a companion Debian 9 VM to host the PostgreSQL server, sporting a Virtual Network linking the HA VM and the PostgreSQL VM.
BTW, important note: Everything get’s logged. Everything. I know this isn’t ‘recommended’ or ‘desirable’ in 99% of the cases, but I have my own reasons for doing so & migrated to a dedicated RaspberryPi for DB functions. It was a big improvement for awhile, but over time, as my HA install and feature set grew, even my dedicated Pi for this function was getting bottle-necked with the 100mb Ethernet interface.
I did consider going to a AlternatePi hardware platform, and there’s some good candidates, as well as installing USB based SSD’s for storage over the SD cards I was using now, but there was another issue in the back of my head: Backups. I have backups now, but going through a restore / recovery process is still a pain. I wanted something better.
After about almost 2 months with it, I found XCP-ng and Xen Orchestra give me what I’m looking for and am really happy with the performance. Actually, the performance is amazing. Boot times of Debian AND Home Assistant (both) are under 30 seconds. If for some dreaded reason, I have to reboot the whole computer, it takes just under 3 minutes for XCP-ng’s Hypervisor & Four separate guest VM’s to spin up: HA, Database Server, Xen Orchestra VM (the tool to manage your VM’s through. web interface), and a Windows10 running Seneye Connect App.
(IP and MAC addresses removed to protect the innocent)
Furthermore, I get the added ability to do Snapshots of my entire HA environment (including Debian OS). This ability to back out of a botched change to my Config file or various other add ons I’ve got in my install is just a dream, as the ability to make a snapshot of everything - just single digit seconds, whereas my old backup script would need up to 2 minutes to run, if HA was running at the same time. Maybe 30 seconds, if it wasn’t.
Further, since HA controls a major portion of my house & my aquariums, I have always been keen on Disaster Recovery. And that’s what the Second laptop is for - to have a place to stream Disaster Recovery images too, ready to fire up as ‘warm stand by’ backup copies, in the event the primary Hypervisor / Laptop fails.
What’s really cool about this - is when you have similar enough hardware specifications - you can not just copy VM’s between machines for backup, but you can also migrate said VMs to another Hypervisor hardware host AS THEY RUN…which is just amazing and reminiscent of the time I first saw this on Sun Solaris on E10000 Hardware that cost hundreds of thousands of Euros.
In the screen cap above, the VM’s running across two Laptops (Thinkpad and Daisy) are shown.
On my old RaspberryPi 3B +, system load used to run around 0.9 on the load side, and could easily peg 2.0 and just stay there for up to 30 minutes. Loading times on iPads and in browsers would still work, but you could feel it was “slow” sometimes on the Pi’s (disclaimer: I have lots of Tabs and dynamically changing Panels on my HA interface). And if you attempted to view the History Graph page or Logbook page… it would either bring the Pi-based HA instance to a crawl for up to 3-4 minutes before recovering… or crashing and freezing up because of RAM issues.
The log / history viewing problems are all gone now. So is the high load. It runs along nicely at between 0.05-0.2 load and with 3GB+ of RAM available, even the long standing Synology Cam memory leak issue isn’t a problem so much anymore - more of a nuisance than a stability issue.
All of this is just peanuts to my real goal with all of this tho: I finally just completed the migration, testing, debugging of my HA version to version .77. Finally, I am “current” on all my HA instances now (both Pi and VM based), and can move onward toward rebuilding CoralPi and documenting all of that.
BTW, those who might be asking “How do I get GPIO’s physically out from my Home Assistant VM?” The answer is, I don’t. In fact, on the “core controller” (aka Joshua in the screen caps above) - that RaspberryPi was always a GPIO “free” box & it was based upstairs away from all my aquarium stuff. Coral Pi and NemoPi are the RaspberryPi’s that have all the GPIO connections and are located by my aquariums. TimelordPi also has GPIO aquarium related sensors for my Phytoplankton farm and provides GPS synced Time to the network, but all of that is on the roof. Splashes, Spills and “knocking things over” happens around the aquariums. I didn’t want my primary logic controller for everything to be around the aquariums - so HA Pi and Database Pi were always located far away from that - everything happens over HTTPS, MQTT and other protocols. This makes “virtualisation” of my primary logic controller rather easy.
Oh yeah, I also got rid of the dedicated PC running Win7 to host the Seneye Connect App - or more correctly stated - recycled it and moved it upstairs where it now runs Win10 on a XCP-ng VM. I used the brilliant VirtualHere USB client app on Windows to forward USB over Ethernet and plugged the actual Seneye Device into one of my RaspberryPi (NemoPi), which also now runs a VirtualHere USB Server along with an instance of HA. This should hold me over until I can get this working natively in Python on a RaspberryPi & means I don’t have to dedicate an entire laptop to this tiny function, and can now use it for other things as well… like automated redundant backups of my entire HA environment.
If you’re interested in looking at the OpenSource alternative to XenServer (or ESXI or other) for hosting your HA environment, have a look here to getting started with the core Hypervisor:
https://xcp-ng.org
To manage it from a Windows desktop, install this windows client: https://github.com/xcp-ng/xenadmin/releases
To manage it from a Web interface (i.e. - Mac/Linux users) have a look Xen Orchestra Community Edition:
https://xen-orchestra.com/docs/from_the_sources.html
Tip: If you decided to build the XO Community Edition, you can build it piece by piece following the instructions above - OR - as I have done more recently… follow this instructions which automates the whole process, kindly produced by Lawrence Systems (see Youtube channel of same name). It’s basically downloading their install script, running it and go for coffee and 15 mins later, it’s done.
Furthermore, I’m now seriously considering… AFTER I do the CoralPi / NemoPi Rebuild and co-documentation for the DIY guide… … I’m seriously thinking about doing a Publicly shared prebuild of Home Assistant + Debian and essentials on a VM for public distribution & use on XCP-ng (should be backwards compatible with XenServer, me thinks)… not unlike Hassbian (my fav of the Home Assistant distros) but for XCP-ng, and might even explore the idea of creating a custom HA component to manage XCP-ng right from the HA interface, thereby eliminating the need for Xen Orchestra. Hmmmm…
But again, and for now, I’ve got all my HA instances upgraded and current and I freed up two RaspberryPi 3+'s for re-use in my Coral Pi rebuild (NemoPi is already a 3+).
I’ve also revised my rebuild plans a little bit. I’m going to do two rebuilds and focus a bit different on each rebuild. I’m going to do what I call a “beginners” rebuild of Coral Pi as planned, but with the emphasis on “beginners and novices” who are new to the whole Hardware building topic / skillset. The idea will be to really try to pull you ladies and guys in further into your own comfort zone about getting your hands dirty with electronics and soldering. Then we’ll dive into installing HA and linking all the hardware to the software sensors we create & show how that is best done and maybe look at an alternative approach or two, and the reasons you might want to consider that. This should be interesting because until now, Coral Pi is the only Pi that doesn’t run HA actually. It’s a huge collection of scripts that communicates with my remote HA Pi upstairs (Joshua) but yet, has the single largest collection of GPIO sensors of all the RaspberryPi’s. It’s an original model RaspberryPi and it’s GPIOs are nearly all consumed - so the upgrade to a 40pin GPIO bus will be much welcome.
The second rebuild will focus on NemoPi and will come a bit later after CoralPi. It’s focus is going too be how do you take something from a “Breadboard (Proof of) Concept” design (which is what CoralPi is now, and will still kinda be after the rebuild) and turn that into the kind of thing that uses your own Printed Circuit Board designs and how you tackle doing all of that. It’s more Intermediate / Advanced level but the idea is to pick up where the Coral Pi rebuild leaves off. It’ll also serve as a nice topical bridge to other projects which I’m looking at doing & even maybe getting my wife to help with Video production of for a YouTube channel or something. Not that I really like being in front of a camera (just the opposite) but I am convinced that many people find instructional videos the more attractive way to learn things themselves.
I decided to take these extra steps because of the feedback I quite literally had - to keep in mind some of the intended readers / viewers of DIY Build guides aren’t so skilled with hardware (yet) - and also would like to say again Thanks to those who did say something to me about it.
Hopefully, now that I’ve sorted some operational / logistical pre-requisites, I can get started on the actual rebuild of CoralPi now in just a couple of weeks or less. It sucked to have to put everything on hold a couple of months ago, but now that I’ve managed to use that as an opportunity to solve / address other issues, I can pick up speed and get this out the door now,… finally.
