[Solved] Weigh kingfishers with HX711, any thoughts on how to store results?

A quick introduction; I’m from the Netherlands (technician, but not a programmer, but starting to learn) and photograph and study kingfishers (Alcedo atthis) which visit my garden a few times a day. I did this for a few years with an ESP32, HX711 weigh sensor and pushed the results to my phone. As it was time consuming to set the results in Excel for further processing, I’m now trying to integrate it in HA. I have setup an ESP32 with a HX711, that’s working correctly, but I struggle a bit:

  • I need a quick response, sometime the kingfisher lands, spots a fish and dives straight away, so I set the interval at 1s.
  • when it rains, the branch where the bird will sit on slowly adds up weight to 2-10 grams, normally I correct for this every 30 min (scale.tare) if there is no weight above 20 grams (so there is no bird on the branch).
  • let’s just say that for convenience a kingfisher weighs 50 gram

I’m now a bit overwhelmed by all options, I checked the filter possibilities, spent hours googling, but still don’t know in which direction I can go best.
I can just store the value every second, but that will get a big database I guess?
The kingfisher visits the garden a few times a day, so I was thinking to only store those values. But than I can’t compare the measured value (lets say 53.4 grams) to the value a few seconds before (which would have been 3.4 grams because of rain). I can’t find something like that in the filters.

Is there a possibility to keep my original way of correcting this in Arduino and then integrate it in ESPHOME, or is that not the way to go? Can it be done with filtering?

My sensor settings in yaml now:

sensor:
  - platform: hx711
    name: "HX711 Value"
    dout_pin: GPIO18
    clk_pin: GPIO19
    gain: 128
    update_interval: 1s
    # ... Other HX711 options
    filters:
      - calibrate_linear:
          - 942656 -> 0
          - 1051425 -> 50
    unit_of_measurement: gram
    accuracy_decimals: 1

For who’s interested, some results (which differ very from what ‘the book’ tell) at my (dutch) website: ijsvogels.nl/gewicht

2 Likes

Cool project. Here’s some thoughts.

Make a “copy sensor” from the HX711 and use a median filter to get the median of the last (say) 10 values. Maybe even much more like 300. This is a “longer term sensor”. This should return the weight a few seconds before the bird lands (even if it has been on the branch for a few seconds).

Then have another template sensor which is the HX711 - the copy sensor. Put a delta filter on this (say 80% of the lightest bird your expecting, let’s say 35g). Now the sensor will only pass values that change by more than 35g.That’s your weight data. It should be your “responsive and calibrated filter”. You can just push large changes only with low latency this way to HA rather than every second.

I’m not sure if my logic is quite right but perhaps it gives you ideas to try.

1 Like

Thank you Mahko! I think I get your idea, will try to work this out tonight! :grinning:

1 Like

I searched and googled for a few hours but I think I miss something (apart from braincells :laughing:). How and where can I copy the sensor?

It’s here. It’s a slightly newer thing. Before this you would typically use a template sensor. The copy sensor approach is cleaner in my opinion.

1 Like

Yes, I did find that, but still don’t understand how and where to put the code? I think I’m completely overloooking something… I’m using the Home Assistant web interface/dashboard.

It should be something like this in your ESPHome Yaml. Code is untested.

sensor:
  - platform: hx711
    name: "HX711 Raw Value"
    id: hx711_value_raw     #Need an id to refer to it in code elsewhere.
    internal: false  #Toggle this to send this data to HA or just keep it internal. Maybe False intially and then True later after testing/minitoring. 
    dout_pin: GPIO18
    clk_pin: GPIO19
    gain: 128
    update_interval: 1s
    filters:
      - calibrate_linear:
          - 942656 -> 0
          - 1051425 -> 50
    unit_of_measurement: gram
    accuracy_decimals: 1

#Long Term Senspr
  - platform: copy
    source_id: hx711_value_raw #Point to sensor above
    id: hx711_value_moving_median
    name: "HX711 Moving Median"
    internal: false
    accuracy_decimals: 0
    filters:
    #Use moving median to create "longer term sensor"
      - median:
          window_size: 10
          send_every: 1
          send_first_at: 10 # Will take ~ 10secs for first value to show up. Adjust as required.

  #Subtract the background / long term weight from the more responsive weight using a template sensor / lambda 
  - platform: template
    id: hx711_value_calibrated_delta
    name: "HX711 Calibrated Delta"
    accuracy_decimals: 0
    internal: false
    lambda: |-
      return (id(hx711_value_raw).state) - (id(hx711_value_moving_median).state); 
    update_interval: 1s
    filters:
      - or: #Send  a value to HA every throttle seconds, or when the value changes by more thna 34. 
        - throttle: 300s
        - delta: 34.0
1 Like

Damn, I just add it to that sensor :grimacing: :rofl: I couldn’t find anything about it and because it is the first project with yaml, I wasn’t aware of it, I was trying to copy/duplicate the integration :grin:

Thanks, just uploaded it, it is not working completely right but that will come tomorrow, you helped me out a lot, big thanks!! :smiley:

1 Like

I think I found the copy sensors slightly confusing when I first looked at them too. Now you know and it’s easy;)

1 Like

Just out of curiosity - whith the setup shown in the photo, wouldn’t the pressure vary with the place where the bird sits? Closer to the joint being lower pressure than at the far end of the stick?

1 Like

That’s almost with everything in life, once you know… :joy:

I was afraid of that when starting this project too, but luckily it doesn’t matter at all. I think because there is no turning point like with a lever. Otherwise I could have solved that with a ping sensor to measure the birds position.

As for now I see it is working great, the delta is exactly 50 gr each (my calibration weight), but when the throttle of 300s (which for testing I changed to 10) is expired, the delta becomes 0. So I have to dive deeper :upside_down_face:

Not sure exactly what you mean but I can maybe help a bit more if you get stuck.

I like to visualise all of my dev sensors in Grafana so i can see how they are behaving…

I mean when the bird is longer than 300 seconds on the branch, the long term sensor will then be no longer p.e. 0.7 grams, but becomes 50.7 grams, resulting in that the calibrated delta is no longer 50, but 0, when the bird leaves, it will be -50.
But tonight I’m gonna take a look at custom sensor component and give that a try, so I’m more flexible and can integrate my existing sketch, which worked always great.
I’m not familiar with Grafana, will check that out later too :smiley:

1 Like

Oh right yes I see. I can already think of a few ways to clean that up. I think there’s more than one way you could solve this problem.

Like you could just look at large positive/negative changes (arrive/depart events) and build an occupancy binary sensor based on that.

And you could get the values of the large changes (before and after) and work with that.

When it becomes occupied you publish the delta to a template sensor. When it becomes unoccupied you publish “0” to the sensor.

I seem to recall accessing the “previous value” is slightly clunky, but am pretty sure it can be done.

Pffft, the more I read, the more I get lost… :flushed: They say yaml makes it more easy, but C++ was much more easy in my opinion. I found also some topics about HX711 and Tare, p.e. also this: Tare functionality for HX711 · Issue #594 · esphome/feature-requests · GitHub

I’m running out of time now for this week, have to do important work for friday, will work on it again next weekend. I hoped I could integrate my Arduino sketch, but seems it doesn’t work that way.
This also looks promissing: ESPHome-Smart-Scale/esphome_hx711_smart_scale.yaml at ff224fb9237d67aeaa87694524277fb3d7fa9950 · markusressel/ESPHome-Smart-Scale · GitHub

1 Like

Got a pretty ‘brilliant’ idea cough a few hours ago, I just put an Arduino nano between the HX711 to manage that part and communicate via I2C (or something) to the ESP32 board :upside_down_face:
I saves me a lot of hassle, maybe not the way a real programmer would choose, but for me it saves me a lot of time.

Tried this and it works beautifully!
ESPHome-Smart-Scale/esphome_hx711_smart_scale.yaml at ff224fb9237d67aeaa87694524277fb3d7fa9950 · markusressel/ESPHome-Smart-Scale · GitHub (ESPHome-Smart-Scale/esphome_hx711_smart_scale.yaml at ff224fb9237d67aeaa87694524277fb3d7fa9950 · markusressel/ESPHome-Smart-Scale · GitHub)

1 Like

Nice.

A few more kingfisher photos please? :wink: