Going to next level of Aquarium Automation...who's with me?

cowboy · October 3, 2019, 5:18pm

Good questions. And yes, I thought early on about UPS Systems, but decided against beginning there for the following reasons:

1.) UPSes don’t address / solve all possible problems in terms of High Availability.
2.) I’ve seen my share of “silent failures” with UPSes; they work great, until they don’t.
3.) The majority (read: all) of my mains power failures have been local (circuit breakers, GFCIs, etc.)
4.) Because of my home location’s on the electrical grid (very Centre of The Hague, Netherlands) and insider information (a friend who works in cybersecurity at the electric energy producer) I know we are at very, very low risk of this particular power company grid failure.

Risk wise, I am at a measurable higher risk of a Network Switch failure than Power Grid failure. BTW, I use enterprise Cisco switches but have seen it already more times that one of the power supplies in a network switch will fail at 5am long before our power grid actually goes out. A UPS wouldn’t address that, but a HA construction of Home Assistant does.

Further, when I started this, I realised, I could either BUY an aquarium controller (at a few thousand total price, at the very least) and a small UPS, or try to build my own controller and re-allocate those funds from controller to a much bigger UPS, or multiple UPSes. That is still the plan. Maybe we’ll even get a generator instead now.

We do have dual Internet uplinks (DSL and Cable) so if one link goes down, it’s not a problem. But to answer your question, yes, I’ve noticed UI issues in HA when the Internet isn’t available. We had this when our Network Firewall’s hardware died last summer (blown caps). This is because certain elements of the HA interface still use the Internet Cloud services - if you use those features, like the historical graphs component. I’d much prefer to have some of this local rather than Cloud based, as I think that would better embody Paulus Schoutsen’s original idea. The solution to this, is use as little of those features as possible, if that’s a risk. If you do use it, have it setup in a way you can disable it in a pinch should that type of risk be realised.

Edited to add: UPSes have their place in a High Availability solution / architecture, but they should only be a component of it, not a single point of failure themselves.

mcclown · October 3, 2019, 11:08pm

Great to see the progress @cowboy

I wrote an article a few weeks ago saying that every software development team should have an aquarium. I wrote that aquarium automation is a good surrogate for the complex systems that companies deal with but with a much shorter feedback cycle. You have further confirmed this idea with this post. Here’s the article: https://medium.com/@mcclown_38611/motivating-your-dev-team-with-real-time-feedback-and-a-fish-tank-5a3a891cc0c1

cowboy · October 4, 2019, 9:51pm

Actually, you got me thinking again about the UI issues if / when there is a complete Internet link failure and the GUI depends / expects Internet connectivity to render History Graphs and other eye candy, resulting in sluggishness of the UI. And to be honest, I don’t know why I didn’t implement it before - I have it already for another reason.

Fortunately, the History Graphs component doesn’t render anything until it’s viewed. If you try to view it without Internet connectivity, it can’t reach the Google API it needs and goes into Time-out-Land.

The solution may be just to make it “not viewable”, automatically.

I previously built a “Control Panel” pane that let me hide / unhide other panels on the Aquarium Interface - partly for safety, partly to clean up the Panel overload, as demonstrated here:

Here all the extra panels are turned off and not viewable. Things like heater control, pump control, etc. But in the next image, the Heater Control Panel is toggled on, and it becomes viewable.

I just have to set this up for all my history graphs, but instead of using an input_boolean for the toggle, I just have to automate it, based on the Internet connectivity status - which is already monitored by Home Assistant.

- alias: 'Heater Control Hide Panel'
  initial_state: true
  trigger:
    - platform: state
      entity_id: input_boolean.heater_control_panel
      to: 'off'
    - platform: homeassistant
      event: start
  action:
    service: group.set_visibility
    entity_id: group.heater_mgmt
    data:
      visible: False

- alias: 'Heater Control Show Panel'
  initial_state: true
  trigger:
    platform: state
    entity_id: input_boolean.heater_control_panel
    to: 'on'
  action:
    service: group.set_visibility
    entity_id: group.heater_mgmt
    data:
      visible: True

In my case, if it see’s both Internet Uplinks down - hide the panel, and that should prevent sluggishness, I think.

My previous fix was to restart the server with those groups manually commented out, but I somehow suspect this might be a more graceful and automatic fix to that problem. Will give it a try soon and let you know.

mrand · October 11, 2019, 3:40am

Interesting solution. I just reloaded my main page when the internet was down, and my iframes were hanging. I’m going to play with it more, but I might have to adopt something similar.

JakeK · October 28, 2019, 5:25am

How did some of you get sensor readings for nitrate, nitrites, and ammonia? What kind of sensors are you using?

cowboy · November 1, 2019, 11:13pm

I’m using the Seneye Marine Sensor together with the Seneye component for Home Assistant that @mcclown was so kind to create and share with us.

cowboy · January 10, 2020, 2:35pm

So I’m a bit late with this, but I’d like to wish everyone a happy new year and best wishes for 2020!

This coming May of this year, will mark 3 years I’ve been using HA to manage all my aquariums and home, and looking back at the number of times over those years and thinking about all the disasters alone that HA has saved my aquariums from, it is truly amazing. And then throw in all the extra time I have now reclaimed (think Automated Water Changes, remotely topping off the Sump with Salt Water when I’m in the next room vacuuming the sand in the display tanks, programming auto-dosers, etc.) - I become literally almost overwhelmed (in a good way) at what an excellent product Home Assistant really is.

And on that note, I’d like to especially thank all the fellow contributors who’ve joined the effort and shared their code, configuration files and build plans back into the project. I remain very appreciative of you and I can’t wait to see what we all can come up with in 2020.

And on that note - for those that aren’t following the thread about using Home Assistant as an aquarium doser (or hydroponics doser), check out @Twinsen68 brilliant contribution. He’s taken my original idea and created his own 4 channel reef dosing solution from scratch & is working on a calibration function. w00t!

Cheers!

Ciwyn · January 12, 2020, 3:43pm

Glad this is still going!

cowboy · January 15, 2020, 6:15pm

Sorry, my initial reply to you had you confused with mcclown. Since corrected once I saw my mistake.

Yes, indeed, very much.

How’s your setup going?

Twinsen68 · January 17, 2020, 12:03pm

Have you see it ? https://sites.google.com/view/leviathan-aquarium/home

cowboy · January 17, 2020, 1:44pm

I was familiar with Reef-Pi when I started this project, but at that time, it appeared that development had kind of stalled as a project. It’s since seemed to pick up steam again, but I don’t see anything gained from switching from HA to ReefPi now, as I now have HA doing everything that Reef Pi can do, and more.

Further, in my humble opinion, Reef-Pi, like the Arduino based RoboTank, are not exactly scalable to the degree HA is. For the person with just one or two aquariums, ReefPi and Robotank are probably fine. But I have a higher bar than most reefers with coral farming & fish breeding operations going on here since 2007.

At any given time, I may have up to 12 aquariums running or more. And they are not all located in the same room as each other. So I needed a more economically “enterprise-like” scalable solution. For instance - how many switched sockets does the ReefPi control?

Also, as stated in my original post discussing build requirements, I wanted an extreme level of high availability / redundancy. Microcontrollers (Arduino/ESP in our case) are typically really robust but there’s only so much one can do in terms of high-availability and redundancy with them. Sure, you can link ReefPi or RoboTank to HA and raise alarms via HomeAssistant if your aquarium controller stops responding, but that doesn’t do me a lot of good if I’m on a business trip to Romania or India.

I wanted a level of protection against even the controller crashing or failing. Which is why I went the route of multiple RPI’s running HA instead.

For reasons of high availability, redundancy and scalability, I wanted to abstract out everything from Temp probes to Switched PowerSockets. This way, if my primary instance of HA were to crash while I was away, another RPI could jump in and take over control of the switched power sockets of the ReefDosers, or the Main System Circulation pump, Lights, etc. Or if the RPI in the fish room which has the temp probe in the sump for heater control were to crash or fail, the main HA instance upstairs will switch over to using the Temp probes from the RPI in the living room. In fact, this level of redundancy using HA is pretty brain dead easy to setup.

I have 84+ switched sockets controlled already by Home Assistant. If those related to the aquariums (28, I think) were hard soldered to a ReefPi or Robotank… I’d lose the redundancy / high availability control of those sockets if the ReefPi / Robotank glitches out and locked up. Excuse the pun, but they’d be “dead in the water.”

This is why for awhile, I used to call my setup System/4 Pi. Not because it was based on RaspberryPi’s but because System/4 Pi was a special IBM computer used for Flight Avionics and even the Space Shuttle (my uncle worked on the integration of System4/Pi on the Space Shuttle) designed for robust fail-over. It was based on 5 computers that were all checking and monitoring each other as well as responsible for the flight and navigation operations.

I’ve since changed the name to a tribute to the movie WarGames (my wife’s request) and it’s now called WOPR - Water Operations Planned Response - aka Joshua. But my design goals of System/4 Pi-like redundancy lives on.

Additional benefits of my approach - is for my wife. She doesn’t have to learn yet another GUI interface. She can just go to the “Aquarium” tab of Home Assistant and take total control of the aquarium because she’s already using Home Assistant for so many other things in the house. Sure, once again, I could tie the Home Assistant interface to a Reef Pi or Robotank - but that’s just another level of unneeded complexity in the architecture.

However, the idea of a Hat for a RPI for WOPR (like the Leviathan, but geared more for HA-Reef) is something I’m currently working on. But it’s got a lower priority than some other features we realised no-one had taken up yet, which we decided to focus on first.

cowboy · January 19, 2020, 2:57pm

And now for something completely different… a recycling project to make aquarium or home lighting out of old aquarium heaters. I’ve already recovered about 5-6 glass tubes from old and dead aquarium heaters, and was thinking about using them to make some ESP32 based Refuge lights by inserting LED grow strips into them. BigClive seems to have had a similar idea using other LEDs for home lighting.

Thought I’d share it for the inspiration value to your own home …and aquarium projects.

cowboy · January 19, 2020, 4:49pm

So another “Lesson learned” I thought I’d share…

This is the first time I’ve used the Seneye Sensor with Home Assistant integration (Thanks @mcclown) on a 10 litre quarantine tank (not previously cycled - was dry before setup) with a fully grown clownfish in it. It’s only been in there a week and I started to notice something right away. The Seneye Ammonia sensor is so sensitive, it starts to register ammonia levels increasing almost immediately after the fish takes a dump. LOL. And what’s more, NH3 Ammonia levels also immediately take a downwards turn within just an hour or two of vacuuming up the fish pooh at the bottom of the 10 litre quarantine tank.

I figured this out by simply noticing the daily NH3 values rise in the graph displayed on my dashboard, and would go downstairs to check the fish & would find the pooh. After vacuuming the quarantine tank, I then began to notice that the NH3 values would almost immediately respond and start to drop again.

That means - I can totally create a Home Assistant Automation to play a TTS Voice notification through the house, informing me my fish has taken a pooh! LOL. This is hilarious, but practical as all heck for Quarantine / Hospital Tank management and helping our fish recover. Particularly if you had to setup your quarantine using a dry tank in a pinch and it wasn’t already established with nitrifying bacteria. In fact, that’s almost a serious game changer for running quarantine tanks.

Just thought I’d share that for what it’s worth.

Ciwyn · January 20, 2020, 5:14pm

See I’m still too concerned with HA crashing for various reasons to fully migrate critical tank functions like heat and ATO to HA. If I lose power to the house and my wifi does something wonky. At least my reef angel will keep on trucking with those functions even if I can’t communicate with it. I seem to have some kind of goofy problem with HA going down due to my own fault or something related to a power failure a few times year. I will admit things are getting much better in regards to stability of HA as new iterations come out though.

That being said I also feel like I want to move on from reef angel. It’s a bit outdated. The new reef angel star looks interesting but at the same point I feel like any interest in the controller has really diminished where I’m not sure how much longer it will be around.

The new coralvue controller looks very interesting but they may not release their api to allow for HA integration.

On a final note I ditched that pi zero for temp sensors and am using esphome with a nodemcu. My God is that a much simpler way to go!

cowboy · January 20, 2020, 6:39pm

Well, I don’t want to push you into something that would cause a catastrophe in your tank, but that’s not been my experience at all. In fact, my Pi’s / HA instances have uptimes of significant length.

Just before Xmas, I rebooted the Pi in the attic running HA and it’s uptime had been 180+ days. But at this minute, this is the longest running HA / RPI instance:

A couple of things I did to relieve myself of that worry tho:

1.) Don’t run cutting edge releases. Unless there’s an applicable security vulnerability, I tend to stay behind the pack a few versions. This gives others a chance to flesh out identified bugs etc.

2.) Designed the high availability setup. It’s more important that the service of HA be available than any one instance of HA in my setup.

3.) Abstracted out things like Heaters, Dosers and ATO so that any instance of HA could take over, if the controlling HA instance fails. If a non-controlling instance of HA fails, the controlling instance of HA will attempt to recover and even hard reboot the non-controlling instance, while raising alarms.

And it works the other way too. If say the Fishroom located “CoralPi” (where sump, heaters, temp probes, etc are) were to crash, Joshua - the primary HA controller, will switch to using NemoPi (in the living room, where the 3 display tanks are) as temp inputs. Since heater switches are networked, there’s no dependancy on CoralPi.

Or, if Joshua were to die in a puff of purple electrical smoke, all other remaining instances of HA will raise alarms and notifications. NemoPi would take over Dosing and CoralPi would take over ATO and Heater functions, until Joshua reappeared.

I’ve tried to make everything “fail safely by design”. And so far, it’s worked exceptionally well. I’ve had 2 SD card failures which weren’t big issues, because of the above, and because everything gets backed up and there’s “cold standby” SD cards taped to the RPI’s for even if my wife is alone in the home. But those are all about to be upgraded to SSDs soon, and that risk will be gone.

We have had a 24 port Cisco die at 5am. Joshua woke us up for that. Cold standby switch got dropped in and I was back in bed within 25 minutes. We had a firewall with blown Caps die on us, and everything kept working just fine through the night. (But again, HA told us about that.)

Three times tripped circuit / GFCI breakers downstairs, and HA told us about that immediately (because each HA instance is on it’s own breaker).

I think with the right architecture planning, you can easily mitigate a lot, if not all, of your concerns with HA. It’s what I sat out to do when I began this - basically replicate the same way I’d build GSM networks with 99.97+% service uptimes, for 15 years.

Oh, last thing… sometimes people ask me about UPSes for all my RPI’s, but I often ask “why? I’ve made the solution resilient to suffer with // work around more than a single RPI failure, regardless of Mains power failure, or a crashed OS / HA instance.” And UPSes only cover mains failures. They won’t cover for that case where one of my 5 cats decides to pee on the RPI.

As a risk management professional, I try to plan for the failures we cannot yet imagine. And that’s the philosophy I took in my design. I encourage others to do likewise.

cowboy · January 20, 2020, 6:58pm

Funny…I just looked at the Hydros controllers that are coming out in 2020… and it appears almost as if they lifted my architecture design philosophy from my original post. Not sure about their decision to utilise the CANbus, I wonder if it’ll have security vulnerabilities available over that interface like Cars and Industrial controllers do? Hilarious!

Someone told me just over Xmas break, that their Alkatronic has a RaspberryPi inside. I’m curious if that’s true.

diverjoe · February 3, 2020, 1:10am

Too much to read in one sitting, but looks like we are on similar tracks. Here are a few words just to kick things off

Docker running on Pi on a secured os - 5 VMs running inside its own world MQTT, Influx, Node-Red, Samba, and a few others supporting actors

Tasmota - break out all sensors to MQTT enabled $5 ESP8266 chips

Node-Red for Logic and routing

cowboy · February 3, 2020, 9:27am

Nice and welcome.

Tell us more when you get a chance. Do you already have an aquarium hooked up to home assistant? How long have you been using HA for that? How far down “the tracks” are you with it?

diverjoe · February 3, 2020, 2:10pm

Been going several years. The thing I wont be able to bring to the table is HA itself. I learned of Node-Red when reviewing HA and OpenHab as a platform for my automation. There are some things i really like about it and am actually just starting to dabble in home automation so I will be standing up HA shortly. Here is my “features”:

Freshwater
RO ATO
2x/day water change with re-mineralization and PPS fertilizing for planted tank
rampup/down of lights following a sinusoidal wave pattern
the usual temp/ph/tds
waterlevel using air pressure
heater control
hack jabeo doser for wifi w/pump calibration
working on DMX light control to use RGB wall wash lights

re-factoring everything using 20/20 hindsight in prep for a 180g standup

cowboy · February 3, 2020, 5:43pm

Yeah, I started originally in March/April of 2017 with OpenHab but once I discovered HA (in May of that same year), I threw all that OpenHAB stuff away and started over again. HA’s internal architecture seemed very similar to the same type systems that would handle CDRs (Call Detail Records) on GSM networks (read: very fast & extremely scalable) between the HLRs / VLRs and Billing systems, which I have about 15 years of experience with. Also, YAML / Python only needs any old text editor (VIM old-hand here) as an editor, where OpenHAB seemed to require (at the time) Eclipse.

Unfortunately, I won’t be able to help with Node-Red - I don’t use it, but there’s other guys who do on here, so maybe they can if you run into any issues. For me, it was too easy to write the automations native in HA & easy to replicate across multiple instances of HA to create a high availability redundant setup.

How does your water level using air pressure work? That’s peaked my curiosity.