Going to next level of Aquarium Automation...who's with me?

This is such a great idea for logging. I have immediately copied this. I also like the graph of the redfield ratio.

I was sort of familiar with it but never looked at it closely and it could explain a lot.
My tank has had high nitrates basically since I started it two years ago. I really have no idea why. Initially some phosphate was registering on my hanna checker, then it fell to zero after a few months. So I’ve had this tank with high nitrates and zero phosphate for sometime. Soft coral does OK, zoas do quite well, but sps was never happy or would just die quickly. I tried carbon dosing but it didn’t make a difference. Recently I have been dosing phosphate to actually have some detectable in the tank (corals hate when you start that as well!). Part of my idea for dosing phosphate was perhaps if I had detectable phosphate in the tank the nitrate might naturally start to drop to fall in line with the redfield ratio. This doesn’t appear to have changed my high nitrates though unfortunately (my current ratio is about 900:1). So now my plan is to build and install a denitrator like this one: Poor Man's Nutrients Control - Donovan's Nitrate Destroyer | REEF2REEF Saltwater and Reef Aquarium Forum

Hopefully this finally gets the nitrate in check because I have no idea how to do it otherwise.

Anyways all this rambling was just me complimenting you on another great aquarium automation idea!

1 Like

Excellent to hear that you also have a use for logging of the manual water tests. Since that post describing this, I’ve been very happy with making the transition from using AquaCalculator. AquaCalc was a great app that served me well for years, but I do enjoy using a unified interface in Home Assistant for tracking all values, rather than having my data split between two software solutions.

Redfield Ratio helped me understand something I thought unusual back when we adopted our initial 200 litre tank from it’s former owners. When we got it, it was in really bad shape. Overstocked with fish, canister based filtration, no PS, and crushed coral (not sand - so it was packed with detritus) on the bottom, and PO4 was registering 5-10 and NO3 was at around 120. Corals looked sick, fish were malnourished & both HLLE and Cryptocaryon real bad.

We couldn’t get larger tank, because of the small rental unit we were in, and we were already shopping for a new house, so we had to overhaul what we had. Within a year, the tank looked amazingly different. I eliminated a lot of problems (canister filtration replaced by PS and refuge and sump), upgraded lights, replaced crush coral with proper DSB sand, and was able to get HLLE and Crypto under some managed control - until we could build proper facilities in new house for proper treatments. But at this time, the corals looked amazing and were growing like gangbusters - despite what I thought was still kinda high PO4 / NO3 values (approx 2-5 / 20-40). This was the first time I’d ever had such high values & had really excellent coral growth - almost too fast - and I couldn’t understand it. I went searching on various forums more as a matter of curiosity and read the first time about Redfield ratio & realised I had basically stumbled across those balanced values & was seeing the result. It didn’t explain everything…the forum posts talked about it in the context of low / ultra low values…and I was running at high values. I continued to try to push my values down, but at that point, I wasn’t so obsessive about it. The tank already looked amazing, let’s not look the gift fish in the mouth, too much, I thought.

These days, I let a combination of Redfield and low PO4 / NO3 be my goals. When I am breeding and raising out baby clownfish, my bio load and PO4 / NO3 values go way up (now I call it ‘Running my tank hot’) - and I tend to focus more on Redfield Ratio when my tank is running ‘hot’ like this. But since suspending growing out the hatchlings at end of last year, my tank is running ‘cool’ now with ULM values again - but my Redfield ratio is about 0.8 : 1 at the moment. Corals are great and growing really well, so I’m not too stressed about Redfield. But I am paying attention to it in my battle against Dino’s which I’ve been afflicted with since the start of the new year. I recently installed some new more efficient fluidised bed filters for GFO, and am driving my PO4 values down now, so I hope to have Redfield corrected better in coming months & am very curious how that affects my Dinos.

BTW, unlike some reefers, I’m not panicking or stressed by the Dinos. They mostly make the and look bad and brown, and kinda get on the rocks, but don’t seem to affect much else. So I’m playing the long game solution with them… almost like a scientific curiosity. Considered upping my NO3 / PO4 values to combat Dinos - but am holding off on that to see if I can find another way for the moment. We shall see. :slight_smile:

BTW, here’s the new Lovelace version of my dashboard. Finally made the jump back in the Spring of this year. I also separated out “Monitors” and “Controls” to two separate tabs. Because I like to leave a window on my computer up at all times for the Monitors, but learned my lesson about having “controls” on the same page as that is a bad idea with 5 cats in the house. LOL

1 Like

You’ve nearly got me convinced to move all my control over to homeassistant. It really has become extremely reliable. I saw you mentioned having the proper architecture a while back which makes sense. My biggest hurdle with all this right now would be the time to do it.
Since ESPhome has become available in homeassistant adding temp probes, scales, or ultrasonic sensors is very easy.

I still have slight concerns with if homeassistant goes down will my heater get stuck on or will a pump get stuck on. I could still plug the heaters into a simple backup heater control for redundancy. I’m thinking ultrasonic sensors for top off control and other things would be the slickest way to use them; I just have to figure out a redundancy for that.

Finally I am probably going to steal your dosing pump idea as well. I have an old jecod one that still works, but it would be nice to track the dosing history via homeassistant and know if something isn’t working correctly.

So this is the advantage of using Network controllable PowerBars / Sonoff 4CH solutions, rather than having direct control via a serial link. With the later solution, which is how APEX and other controllers work, if your controller freezes up / fails, you lose the power bar.

Using network controlled powerbars gives an advantage to creating more robust or even high availability Home Assistant solutions.

One super easy and cheap way to ensuring the heater stays on is to use a dual - backup strategy:

1.) Set the thermostat on your actual heater (if it has one) to just 2C above your target temp. I run my tank at 25C… but if it were to stick at 27C because everything else failed, that’d be okay - nothing would stress too much or die - even if it ran for days at 27C.

2.) If using a Sonoff with TASMOTA installed as your slave control of your heaters, you can implement a sanity check in in TASMOTA… always be pinging HA to check to see if it’s up - and if it goes offline, default the relays to OFF and keep them off.

That way, you’ve got a single node HA implementation, that has two layers of Fail Safely by Design functionality built into it.

Alternatively, you could take it another step and add a second HA Node.

You can almost literally Copy and Paste all your automations from your primary node to this new secondary node, and generally speaking add one conditions (and device tracker / sensors) to all your automations - so that none of the automations actually run until the condition that your Primary HA node has gone offline for 5+ minutes.

That 5 minute value should give you enough time to do a reboot or restart of HA on the primary system without triggering your failover backup instance of HA. Adjust to 10 minutes, if you need it. :slight_smile:

Now, running a second HA node this way, works pretty well, but there can be one dependancy you need to give some thought too - and there’s a few ways to solve this dependancy - that of MQTT server.

The easiest way to solve this is to setup a 3rd host (RaspberryPi, NUC, or VM) dedicated to MQTT (mosquito is what I use). But this doesn’t actually buy any redundancy for MQTT itself. But it does isolate and separate MQTT away from HA installs, so if you take one HA instance offline (maybe backing up the SDcard, or something) the MQTT service is unaffected. Also, MQTT is pretty damn stable - in 3 years, I’ve never had MQTT crash on me. So this is the solution I use - MQTT is on it’s own dedicated VM, completely separated logically and physically from each HA install.

Alternatively, if you want a failover MQTT host - that’s also do-able, and this is what I would do, if I wanted to go this way, if say using RPI’s:

Setup Mosquito on each RaspberryPi - but with a 2nd IP address on each eth0:1 interface. This 2nd IP Interface on every RaspberryPi would be configured to share the same IP address. And each RaspberryPi would be configured NOT to bring up either mosquito server or that secondary IP Interface, except for an assigned Primary MQTT RPI. The Secondary RPI’s also with MQTT and 2nd interfaces would only activate if they saw the Primary MQTT server go offline for greater than 3-5 minutes, then they would first bring up the secondary IP interface and then launch MQTT services on it.

If the Primary comes back online, it’ll be configured with some customised scripting to first check to see if the Secondary instance of MQTT is already running & IP address is in use - in which case, it will wait until the Secondary disappears before resuming it’s role as the primary MQTT.

This way, from HA’s perspective, the MQTT’s IP address never changes - regardless which RPI is running MQTT services - it’s just always there, and at worst, it’s not available for up to 3-5 minutes.

(you could also do this using VMs or NUCs, if you wanted)

The bash / pythons scripts to do all of this would have to be written, but that’s pretty easy if you are semi-proficient in either of those.

Basically, I think it would do the following:

1.) Test “did I just boot up?”
2.) Check for expected MQTT IP address, if not found, bring our IP up and launch MQTT
2b) If found IP, check to see that MQTT service is running, if so, go to sleep, checking only for existing service to go offline every 5 minutes
2c) If IP found, but MQTT not found, host may be hung, force it offline (remote login shutdown scripts and/or toggle power off on that node)
2d) bring up local 2nd IP interface and MQTT service

Maybe that logic needs some finessing, but at a high level, that’s it.

Let me know what you end up deciding to go for. I’m curious. :slight_smile:

Well since my reef angel water level sensor seems to be acting up and the joystick on it has stopped working I think its time to move it all over to homeassistant.

I have a spare pi2 that will probably suffice for the separate aquarium functions. I guess where I plan on starting is switching the auto top off to homeassistant. I wanted to use the sonic distance sensors set up through ESPhome but I’m wondering if that is the best method? With my reef angel it used a water level sensor that would measure pressure in a pvc tube (there’s probably a way to migrate these over to homeassistant but I haven’t looked into that.)

What’s everyone think the best idea is for auto top off?

1 Like

The pressure based PVC tube sounds interesting. If it uses 1-wire or I2C or something like that, it should be easy to migrate over, I think.

I simply use multiple float sensors, both vertical and horizontal versions, and haven’t had any problems with them in 3 years. I do have places to mount those sensors which are mostly salt-creep free zones. That said, I used to have two different Tunze Osmolators in our aquarium setup in our old pre-2005 house, and in that setup, there was no salt-creep free zones as it was a very small setup. And still, the Osmolators optical sensor still ended up with gunk and overflowed a few times.

I can explain what I’ve done in my current setup and you can decide from there.

This is the detailed (normally hidden) overview panel of all my water level sensors:

Normal Ops Waterlevel: This is a horizontal (thru-hole design) swing arm float valve design, which I mounted on a recycled Mag-Glass Cleaner, that way it’s very secure, but easily adjustable and/or relocatable. This sensor is set to the height I want to maintain the water level at when all pumps are ON. The magnet had become too weak for the thick glass on my 1000 litre tank, but on my plastic tub sump, which is very thin relative to the glass, the mag has a lot of holding power. When the sensor measures low, then the Reverse Osmosis Top Up dosing pump is activated. (Bonus Feature with this sensor: Not only does this control RO Water Top off, but if the Saltwater Transfer Pump is activated while the Main Circulation Pump is running, it will cut off the Saltwater Transfer Pump once the sensor triggers. This is handy when I scoop out either large corals, rocks or water from the display aquariums in the next room - particularly if I’m vacuuming the sand-bed. Then without having to leave what I’m doing in the Living room, I can toggle the Transfer Pump to replace the lost water/mass I took out of one of the display aquariums and HA will take over turning the pump off. (Before HA, I used to have to run back and forth between rooms to do this manually.)

Sump Overflow Status: Again, a thro-hole swing arm sensor which is permanently mounted into the side of the plastic tub. This is set to the “Oh F***!!!” Water-level, which is what you might say when water is just 2cm from the top of the sump and still rising. When this activates, Home Assistant heroically and valiantly turns on the Purge Sump Pump, only de-activating the Purge Sump Pump when the sensor indicates it’s returned to returned to sane levels for more than 15 seconds. This sensor also activates a WAV file of a specific “fish ops alarm” to be played throughout the house & TTS spoken description of what alarm event has occurred and what to do (for like house / tank sitters, etc.). And push notifications to my phone.

Sump Full Cutoff Status: Same sensor and mounting as above, but this is mounted just a little below the Overflow Status sensor, and is used mainly for Automated Batch Water Changes. (FWIW, I also have Automated Continuous Water Changes using dosers also setup, but now I digress). This is used to cut off the Saltwater Transfer Pump from my 180L mixing barrel during batch water change operations. This also triggers informational TTS and sound effect alarms & push notifications.

What is not listed here, but is an important part of my Redundancy; I already had two Aqua-Medic Auto-Top-Up dosing pumps with vertical float sensors. I installed both of theses, dedicating one to use for Reverse Osmosis Top Up, and the other for Kalkwasser Top Up. Each pump is hardwired to it’s own vertical float sensor and I mounted both of these about 1-2 inches above the level the HA controlled swing arm sensor is mounted. This way, if that HA sensor fails (or let’s assume that HA RPI fails), the vertical sensor of the AquaMedic Dosers will act as a backup. The power lines from the Aqua Medic Dosers are plugged into one of my network controlled powerbars which HA integrates with and controls.

Salt Water Mix Status: same sensor and perm mounting on my 180ltr mixing barrel, because even the filling of the barrel is automated by Home Assistant. (Literally, I pour in 5 litre size cups of Salt, close the top, hit the Mix Salt & WaterChange function on Home Assistant and walk away. Home Assistant fills the barrel, mixes the salt, brings it to temp, and then executes the whole batch water change process.

Skimmate Bucket Status: Again, I choose for the swing arm designs here because I find them the most resilient / reliable design. And the stuff collected in this 20 litre bucket can get quite gunky. When water level is achieved (approx half full), it simply pumps the bucket down the drain. The size of this bucket is a hold-over from the days I used to empty this manually approx every other week. Notifications occur so I can easily tell if it’s not pumping / purging on a bi-weekly basis, something has gone wrong / needs cleaning. :slight_smile:

Salt creep isn’t a problem on my aquariums, and here I use standard vertical sensors, which is the rest of the sensors on this list. However, these do get a fair amount of use. I have a healthy self-sustaining population of snails. And I have 4 x 40cm pipes under the floor, making runs as long as 15 meters (x4) of pipes between 3 Aquariums in the Living & Dining rooms, and the sump / refuge / working tanks located in the dedicated fishland infrastructure room. And big snails can create quite a drag on the currents in those returns back to the sump. Once in a blue moon, an air bubble can get down there and slow things down. So if any of these trigger for longer than 30 seconds, Alarms start playing announcing in 30 seconds the main circulation pump between the tanks will be cut off, and then it does it. If the water level returns to normal, it’ll wait 60 seconds and alarm that it’s going to fire the pump back up again, and repeat this process up to 5 times in an attempt to clear the line. If it cannot succeed it stops trying and waits for manual intervention.

Next stop - Kalkwasser as a component of AutoTopUp
This is my detailed control panel for all Top Up functions, including Kalkwasser

Originally, I had Kalkwasser in serial with RO water. Meaning, it would dose Kalkwasser up to the setup point and switch over to RO once it hit the dialled in amount. But what would happen, particularly in spring and fall, it would end up dosing Kalkwasser still throughout the Light On period, since not as much water would evaporate during the evenings / night. So now it works in parallel during the night from 12am until 12PM (my tank lights come on at 2pm and go off at 11pm). This balances my pH Day/Night shifts a lot better.

And all the important stuff described above is minimised and displayed more succinctly on my Dashboard overview like this (I have another page tab for detailed controls view separate from the Dashboard monitoring view. Because 5 x cats & keyboards are bad mix.):

Oh, and of course, I have HA to raise alarms (and attempt a shutoff) if for some-reason the AutoTopUp runs for more than 2 hours. The advantage of using dosers to do auto-top up is when they go wrong, they go wrong in slow-motion; giving on the benefit of time to detect and react to those undesirable events, before they become incidents and impacts. :smiley: Additionally, as you’ve probably noticed, you can also calculate / track how much water you’re consuming as a factor of evaporation that way.

A thought for your ideas…why not use both Sonic sensor and float sensor together? You could have your float sensor as the actual control, and create a sanity check in Home Assistant using your Sonic sensor to raise an alarm if they disagree. If they agree, run normal TopUp functions. And you get precise information what your water levels are to boot.

1 Like

Curious on how you got Free Ammonia and Ammonium levels in your tank? I have a freshwater tank I am attempting to do the same. So far my system has 2 temp readings, IR controller for lights, TD Sensor, display for showing tank properties, water leak sensor, and warning light. All this controlled with a ESP32. I was killing myself trying to find the ammonia levels in tanks. Any help on that?

Hey @agengr1293,

Apologies for the delayed response; I just now saw your inquiry. I think something went wrong with notifications or something.

Yeah, the NH3 monitoring I’m doing is from the Seneye Sensor.

Do you have one of these? Did you setup the integration component? Details about that integration can be found here.

https://community.home-assistant.io/t/adding-support-for-seneye-aquarium-pond-sensors-removing-the-need-for-a-seneye-web-server/

This is my first code on Python :)))

Hey @Xinil I’m going to rope you back over onto the main thread since the High Availability topic is a bit off-topic for the Autodoser thread, if that’s okay with you. I’ll quote your latest comment on the autodoser thread here, just to make it easier to follow.

Thanks so much for the detailed walkthrough of your home system, cowboy! You’ve got an amazing setup and a seriously impressive workflow of failure-triggers, backups and redundancies.

You’re welcome. :slight_smile:

FWIW, this setup began itself life in 2003 as a single 250litre tank which I adopted from a previous owner who was moving back to Germany. In a way, it was more of a rescue case as they had let it fall into neglect and it had Old Tank Syndrome after 5 years, and just wanted to get rid of it all. I nursed it back to health, and started growing it out slowly by just adding more and more tanks to the overall system through the years. At some point, I realised I had so much time and labor invested in it, I really wanted to have the same kind of reliability in the aquarium automation controller that I was familiar with building in my day job working at mobile phone network operators, where network & system uptime HAS to be better than 99.97% available. And I’d already been scratching my head wondering why wasn’t there any commercial or open source aquarium controller that offered this kind of guarantee. Since there wasn’t any, that’s why I resorted to building it myself.

First things first, I need to set up a secondary/backup HA! I don’t know why I haven’t done this yet but I guess I needed to see other people doing it first to know it was possible. I run HA in a VM as well using the qcow2 image. So my plan would be to install HA on a separate machine (I have two Unraid servers each running VMs/dockers) and set up a similar set of ‘secondary node conditions’ so that nothing runs without first confirming the primary is down. Do you have any more details on how you set up those conditions? I use Node-Red/MQTT for 90% of my automations.

Well, if you have two Unraid servers running their own VM/docker containers, you’ve already got a good base to work from. I can’t advise on Unraid as I use XCP-NG (XenCloudPlatform-NextGen) as my VM hosts, largely due to 1.) I was already familiar with Citrix XenServer at the GSM Mobile phone companies I worked at, and 2.) XCP-NG had just launched as an open source project with intentions to reclaim Xen back from Citrix who was moving at the time to make Xen Server more proprietary and closed source, and 3.) If one builds the XenOrchestra Community Edition management tool from the sources (which is pretty easy), then one get’s ALL the enterprise management tools like Live Migrations between hosts & full backup suite and other functionalities.

One thing I forgot to mention in my previous reply, is for my VM install of HA, I actually broke out the Home Assistant DB SQL server & MQTT server into their own dedicated VMs, for a few reasons, and recommend others do the same. My primary HA node which runs on my VM, is the one node that runs all critical (i.e.- aquarium automations) and non-critical (i.e.-streaming beach & sunset cams, etc.) functions when everything is working properly. I want to keep a very long DB history (several months-to-a-year) of water parameters, and automation history, which I don’t want to lose if I use a snapshot / backup restore function of the Home Assistant VM itself.

Further, the reason for doing this for the MQTT server is that I want the other RaspberryPi nodes to be able to talk to each other, even if the VM HA host goes down or is being rebooted or upgraded. I’ve found that Mosquito MQTT server is very fast and very reliable in practice, but in that extremely rare case that it were to crash, my XCP-NG box is setup to reboot it (which takes just like 15 seconds) or spin it up on my secondary XCP-NG host. (I won’t go into detail here, but there is a way to have a redundant installation of MQTT server on a RaspberryPi in a cold-standby mode that only fires up if the primary MQTT server were to fail, but I do not use that at this time.)

I also do not use Node-Red at all. By the time I saw more people were starting to do this, I had become quite comfortable doing all my automations natively in YAML on HA and felt that NodeRed would just add more complexity to my solution, which I wanted to avoid. It might be possible to do this using NodeRed, but being I’ve never used it, I can’t advise on that myself.

I also thought Zwave was limiting / blocking in its capabilities and reliability. Others have had better experience with Zwave, but in my limited trials with an Aeotec Zwave dongle and one single Aeotec SmartSocket6 hooked up to our coffee maker, I had to many Zwave network issues despite the dongle & smart socket just being a few meters apart & clear line of sight. Also, since the dongle has to be plugged into a host, if said host were to die in the middle of the night, this could present problems. I found HTTP, MQTT and SNMP much more flexible, reliable and scalable in this regards compared to Zwave.

For the High Availability functions, I broke out each function as much as possible / practical to do so, for more granularity of control, specifically disabling specific HA-HA automations should the need arise, without disabling the full suite of HA-HA functions. For instance, maybe I want to turn off the Sound Alert automations for a failure, without disabling the shutdown of of the autodosers themselves.

For example, this is the function to safely shutdown any/all dosing operations (with extreme prejudice - i.e.- I just assume they may be running, rather than testing to see if they are running) if one of my RaspberryPi HA-HA nodes detects either the VM Host has gone offline, or if the HA VM has gone offline or if the HA application itself has stopped sending MQTT heartbeat signals.

  • alias: “NETMON: VM Node OFFLINE Doser Shutdown”
    initial_state: true
    trigger:

    • platform: state
      entity_id: sensor.vmhost_tracker
      to: ‘OFFLINE’
      for:
      seconds: 30
    • platform: state
      entity_id: sensor.ha_vm_tracker
      to: ‘OFFLINE’
      for:
      seconds: 30
    • platform: state
      entity_id: sensor.ha_vm_mqtt_heartbeat
      to: ‘OFFLINE’
      for:
      seconds: 30
    • platform: homeassistant
      event: start
      condition:
      condition: or
      conditions:
      • condition: state
        entity_id: sensor.vmhost_tracker
        state: ‘OFFLINE’
      • condition: state
        entity_id: sensor.ha_vm_tracker
        state: ‘OFFLINE’
      • condition: state
        entity_id: sensor.ha_vm_mqtt_heartbeat
        state: ‘OFFLINE’
        action:
    • service: switch.turn_off
      entity_id: switch.reefdoser1_pump1
    • service: switch.turn_off
      entity_id: switch.reefdoser1_pump2
    • service: switch.turn_off
      entity_id: switch.reefdoser1_pump3
    • service: switch.turn_off
      entity_id: switch.reefdoser1_pump4
    • service: switch.turn_off
  • alias: “NETMON: VM Node OFFLINE Sound Klaxons Voice Prompts”
    initial_state: true
    trigger:

    • platform: state
      entity_id: sensor.vmhost_tracker
      to: ‘OFFLINE’
      for:
      seconds: 30
    • platform: state
      entity_id: sensor.ha_vm_tracker
      to: ‘OFFLINE’
      for:
      seconds: 30
    • platform: state
      entity_id: sensor.ha_vm_mqtt_heartbeat
      to: ‘OFFLINE’
      for:
      seconds: 30
    • platform: homeassistant
      event: start
      condition:
      condition: or
      conditions:
      • condition: state
        entity_id: sensor.vmhost_tracker
        state: ‘OFFLINE’
      • condition: state
        entity_id: sensor.ha_vm_tracker
        state: ‘OFFLINE’
      • condition: state
        entity_id: sensor.ha_vm_mqtt_heartbeat
        state: ‘OFFLINE’
        action:
    • service: media_player.volume_set
      data_template:
      entity_id: media_player.node2_speaker
      volume_level: “1”
    • service: shell_command.vmhost_offline_siren
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’
    • service: shell_command.vmhost_offline_announcement
    • delay: ‘00:00:06’

So as you can see, while the triggers are the same, the automation actions (and automation labels) are different for that greater granularity of control - I can turn off a specific High Availability or Alarm function while leaving the rest remaining. This is useful for debugging, testing and even at times, during upgrades (because I tend to upgrade one host at a time).

BTW, in my professional experience, I’ve seen VM Host network stacks do a “half-crash” - where the VM’s keep running & can keep talking to each other & keep existing open connections to external boxes open, but any new connections made from an external box to a VM host get denied / dropped. It’s an extremely rare condition, but having seen it happen in enterprise environments, that is why I reply on pinging the VM host, the VM container, and an MQTT heartbeat from the VM container - this can reveal the nature of the failure that occurred & how to react.

Like the above examples, I have another automation with the same triggers but a different automation action that toggles an input boolean for the locally implemented (on that RaspberryPi) autodoser automations. These local automations are the very same ones which are already shared on the AutoDoser thread with an extra condition added for that input boolean so they do not run unless the boolean is activated. The trigger also has a longer timeout (10 minutes) so that the dosers are not run from a secondary node in the event I’m just rebooting the VM (30-40 seconds) or VM host (5-6 minutes).

As for dosing…there’s a lot of consider. I’ve already installed a Reef Pi (running on a Pi Zero) and I have a pH probe reporting to my HA system. it works wonderfully.

FWIW, I investigated the ReefPi project, but at the time (end of 2016/early 2017) it had appeared the project development had stalled & documentation was kinda spotty, so I didn’t pursue that. However, I am glad to see it got picked up again and is moving forward once again. I’m familiar with several people on this thread here, whom have integrated the ReefPi with HA so I know it’s totally doable, but I never got into it myself.

Another contributor ended up writing the code to natively support Seneye devices (which I have 3 of) and I’ve been happy with that path the last few years for pH and NH3 Ammonia monitoring.

Given I can do more with this Reef Pi, my two options are:

A) Add a Reef Pi doser and make it part of that system - independent of HA and open to failures on its own (adding redundancy here seems very difficult and I’m honestly very if the risk is worth the ‘simplicity’ reward.)

or B) get an AC powered doser and trigger it via HA and Z-wave. I need to set up my secondary HA before I’d be comfortable going down this route.

I can’t comment much for option A, but could speculate if you go this route, you have two sub options with this path - run the doser logic from HA through ReefPi or have the doser logic run natively in ReefPi itself.

For option B, there’s two options actually - you can buy a Sonoff 4CH that supports just AC switching (my existing doser was AC…so the choice was that) or if you want to use DC pumps, Sonoff makes a variant of the 4CH that is AC/DC switching compatible.

What I can say is while MQTT is extremely fast, it’s not atomic fast. There is a tiny, tiny bit of network latency. In practice, and because my dosing pumps pump rather slowly (just 0.41667ml every 1 second for my MG/CA/Alk/Iron dosers & double that for my Reverse Osmosis and Kalkwasser dosers) this tiny bit of latency doesn’t translate into much in the way of variation. For instance, my Kalkwasser is set to dose 6 Liters over the night period, split up 12 times every hour. At the end of the day in practice, this comes out to maybe 6002ml dosed one day, the next day might be 5997ml, the next day it might be 5999ml, etc. So the deviation in practice is quite small. The smallest amount of daily dosing is the Iron doser, which is set to 22ml a day - in practice, the daily deviation is less than just 0.2ml (i.e. - 21.85ml, 21.92ml, 22.05ml, etc.). In practice, these tiny deviations do not make themselves apparent in my chemistry testing of my reef. In fact, my Iron has ridden the last 3+ years extremely stable at my target of 0.1ppm. And I would expect they wouldn’t be measurable in your smaller system either, however, I do want to be very honest and transparent about this.

On the flip side, I can imagine some of that latency is also from the Linux OS itself, as Linux is not an atomic real time OS, so it’s possible even with direct hardwired GPIO from a ReefPi, you might still see some tiny bit of latency & variation in what actually gets dosed.

One way around this would be to implement the doser control logic directly on the ESP chip of the Sonoff 4CH itself & simply use HA to push the variable runtime parameters to the ESP chip. I just never got around to doing this myself, because I’ve not come across the need to have that level of accuracy.

But you do have lots of options to choose from for your own implementation here, I think. :slight_smile:

I hope that helps in deciding which way you want to go. Again, keep us updated what you decide and how your implementation goes. I always appreciate hearing about other people’s journeys with this, and if you feel inclined to do so, feel free to share a few pictures of your Tank and your HA solution implementation.

1 Like

@cowboy, although I have nothing to do with aquarium automation, I really do appreciate how you have explained your setup and the redundancies that are in place. I was wondering if you have done any posts or blogs on the methods you have done for these?

Just on here on the Home Assistant Community Forum, on Facebook in different Aquarium and HA groups (i.e.- ReefControl, MyReef, etc.) and was just invited to start blogging on the ReefHacks website.

2 Likes

Hi @cowboy , sorry, I mean’t have you posted anywhere about how you did the redundancies for HA itself, not so much about the aquarium automations, if that makes sense?

That’s what I was answering you about - the redundancies. I’ve described them in the above thread, and on the other forums I described in my first reply to you.

When I first sat out to do this, nobody in the home automation / aquarium automation sphere was doing anything like this; commercial or open source. In 2017, most people on HA were still just automating lights and media centres; High Availability seemed over kill for a lot of people just beginning with HA, which was also a much smaller user base back then. Most were just annoyed SmartThings and some other IoT clouds weren’t even 95% available. Aquariums are a little more mission critical than lights and media centres however, and so that’s why I posted & described my builds there; that’s where the larger potential audience came from.

On the flip side, I knew home automation user base would come, eventually. Once HA gets embedded in people’s lifestyles, when support for more mission critical devices like Diabetic and Health monitors got picked up & depended on, more interests would eventually come for HA-HA implementations. And it finally seems interest in Application Level redundancies is growing, finally.

What’s your use case? What kind of High Availability solution are you looking for?

My setup is by no means as critical as yours. I run all the house automation, which includes lighting, fans, etc. Some purely rely on MQTT & HA to function (I know, I know, it was a mistake). So I guess breaking out as many things as possible would be better to ensure the load on HA is low. I would eventually like to have the DB & MQTT server on it’s own. If there was a way of HA having a redundancy itself, then I would also look at that.

For the time being, I have 2 separate machines running that I could duplicate what I need to on each. One currently runs Proxmox and the other Unraid.

I guess if you have anything written up on your methods of doing what you have done that I can also implement, I would be interested :wink:

Apologies in advance if the answer is yes, but did you happen to read the reply I made to Xinil just 10 days ago? In fact, it was right before your initial post to me. The context function is about liquid dosing pumps, but the method is easily carried over to lights, sprinkler systems, media centres, coffee pots, deep fryers, etc.

And for what it’s worth, I don’t think pure HA / MQTT is a mistake (although I’m curious why you think so?). In some ways, it makes the setup of Application Layer implementation of High Availability a bit easier, as all instances of HA can be subscribed to the same topic (say Temp data from a specific probe which controls your home heater/AC) and your default primary HA VM node can be setup to act on that data, while your secondary HA VM node will only act on that Temp data when the primary HA node is offline, or fails to function properly within a certain time period.

And that latter point, is one of the strengths of Application layer High Availability offers over and above VM based High Availability. With VM based High Availability, I can set triggers to reboot a VM if that VM crashes or kernel panics, but it doesn’t give me the ability to sanity check the logic and functioning health of the Application itself. Application layer High Availability can be setup to constantly sanity check and monitor the functioning health of another node’s application and/or OS VM.

This is very similar to the Space Shuttle or Crew Dragon Flight control systems - 5 redundant computers which all are running the same program, one of which is controlling (primary) and the other 4 are monitoring (secondary) nodes, the later of which are all sanity checking the performance operation of the controlling node. The moment they see the primary go out of expected functional operation - whether do to a complete system crash or a more subtle dropped CPU instruction due to a low voltage condition (which is common on RaspberryPi’s using either underpowered or failing USB Power supplies), one of the designated secondaries will jump in, demote the primary, and take control as the new primary.

And we can do this, we can replicate this kind of functionality, easily and natively inside Home Assistant, because HA gives us the possibility to monitor and control another HA node almost just as if it was a light bulb or smart switch itself, but one that offers a lot more data input to make smart decisions about that HA node’s health and operational state.

The only major difference between my primary and my 4 secondaries is that my secondary nodes have additional bits of conditional evaluation functions in their HA automations to not trigger unless the condition has been meet that a failure scenario has occurred.

BTW, this is another strength of Application Layer High Availability - you can get very granular in deciding what functionality should be mission critical and offered as High Availability, while excluding others that are not. For instance, I like having my dozen security cameras streamed to HA’s interface, and my wife likes the dozen or so Sunset & Beach cams streaming to the HA interface. I have this setup on my VM based instance of Home Assistant, which can easily handle this. But do not have these cameras streaming setup on my RaspberryPi nodes, because they would overload the CPU of RPI’s and they are not mission critical. With VM based High Availability, it’s pretty much all or nothing.

However, there is a role for VM High Availability to play here: Your MQTT server, and this is a perfect scenario for that, because we are not really concerned with MQTT storing data, but we are concerned with MQTT always being available somewhere on the network.

If you want to go down this path, and you haven’t done so yet, I’d go ahead and split your DB servers onto their own VMs on their respective VM hosts, and do the same for the MQTT server. Setup duplicate MQTT servers on both VM hosts under their own VMs that are exactly identical down to the same IP address. The only difference between them is that one should be designated a primary MQTT server that is set to always boot up and run, and the secondary MQTT server will never run, unless the primary MQTT server has crashed / been offline. Unfortunately, I don’t run Proxmox or Unraid, so I’m not exactly sure of how to technically implement that on those platforms, so you might have to research to find out if this is even possible to use the native feature sets of these VM hosts to do that. If it’s not, let me know, and I think I have a work around for that - which is a way I had solved this issue when I first began my use of HA and everything was on only a pure RaspberryPi environment (and before my incorporation of VMs for my primary, primary DB server and MQTT server).

In the meantime, if you want to give me a copy and paste example of an automation you’d like to configure as a High Availability function, I can help show you through that how to set this up in your automations & sensors.

BTW, I’m in the middle of painting my house at the moment, and am limited to periods of rain for time to post here. :slight_smile: If I don’t reply very quickly, that means it’s sunny / not raining here and I’m out painting - I will eventually return, provided I don’t fall off the scaffold and injure or kill myself. :slight_smile:

2 Likes

We would be over the moon if you use our platform to enlighten other hobbyists about how much more is actually possible with aquarium automation. The prices the category captain charges are astronomical and I wish more aquarists knew there were alternatives.

1 Like

@Faze_One Just an update to the last time we talked; revarnishing the exterior of the front side of my house has taken longer than expected due to weather / forgetting how long it takes to sand hard wood (compared to say, pine), and other surprise repairs that had to be made. As soon as I’m wrapped up with that, and enjoy a little bit of remaining holiday time with my wife, I’ll get on that.

Hi Ciwyn, any luck yet with the RA?