[Custom component] Bird Buddy smart bird feeder

CumpsD · February 1, 2023, 9:31am

Small question, what does collect_postcard do exactly when you compare it to the BB app? Does it simply “accept” a postcard and all the pictures and adds it to the BB collection? Or something else?

madcoder · February 1, 2023, 2:15pm

Yes, that’s right. It’s like opening the postcard in the BB app, and adding the photos to your collections. Depending on which option you use for strategy, it might keep only fully recognized sightings, or it might select the top species guess. See GitHub - jhansche/ha-birdbuddy: Home Assistant integration for Bird Buddy smart bird feeders

CumpsD · February 1, 2023, 4:20pm

Sadly enough, it seems they are not in the event, I just had a sighting with a video in the BB app, but this is what HA received (replaced a bunch of id’s with xxx to make it shorter)

this:
  entity_id: automation.birdy_download_new_postcard_sighting
  state: 'on'
  attributes:
    last_triggered: '2023-01-29T15:51:09.344520+00:00'
    mode: queued
    current: 0
    max: 10
    id: 'xxx'
    friendly_name: '[Birdy] Download new sighting, notify and add to calendar'
  last_changed: '2023-02-01T09:49:09.436693+00:00'
  last_updated: '2023-02-01T09:49:09.436693+00:00'
  context:
    id: xxx
    parent_id: null
    user_id: null
trigger:
  id: '0'
  idx: '0'
  alias: null
  platform: event
  event:
    event_type: birdbuddy_new_postcard_sighting
    data:
      postcard:
        id: xxx
        createdAt: '2023-02-01T15:38:06.004Z'
        __typename: FeedItemNewPostcard
      sighting:
        feeder:
          id: xxx
          name: Birdy
          state: READY_TO_STREAM
          __typename: FeederForOwner
        medias:
          - id: xxx
            createdAt: '2023-02-01T15:37:52.013Z'
            thumbnailUrl: >-
              https://media.app-api-graphql.app-api.prod.aws.mybirdbuddy.com/media/feeder/xxx/media/xxx/THUMBNAIL.jpg?Expires=1675.....
            contentUrl: >-
              https://media.app-api-graphql.app-api.prod.aws.mybirdbuddy.com/media/feeder/xxx/media/xxx/CONTENT.jpg?Expires=1675.....
            __typename: MediaImage
          - id: xxx
            createdAt: '2023-02-01T15:37:52.013Z'
            thumbnailUrl: >-
              https://media.app-api-graphql.app-api.prod.aws.mybirdbuddy.com/media/feeder/xxx/media/xxx/THUMBNAIL.jpg?Expires=16753.....
            contentUrl: >-
              https://media.app-api-graphql.app-api.prod.aws.mybirdbuddy.com/media/feeder/xxx/media/xxx/CONTENT.jpg?Expires=167.....
            __typename: MediaImage
        sightingReport:
          reportToken: >-
            xxx
          sightings:
            - id: xxx
              matchTokens:
                - xxx
                - xxx
              __typename: SightingRecognizedBird
              color: YELLOW
              text: Whoa! Can't get enough? 1st time today!
              count: 3
              icon: HEARTS
              shareableMatchTokens:
                - xxx
                - xxx
              species:
                id: xxx
                iconUrl: >-
                  https://assets.cms-api-graphql.cms-api.prod.aws.mybirdbuddy.com/asset/icon/bird/xxx_great%20tit.svg
                name: Great Tit
                __typename: SpeciesBird
                isUnofficialName: false
                mapUrl: >-
                  https://assets.cms-api-graphql.cms-api.prod.aws.mybirdbuddy.com/asset/map/bird/xxxGreat%20Tit-01.svg
          __typename: SightingReport
        __typename: SightingCreateFromPostcardResult
    origin: REMOTE
    time_fired: '2023-02-01T15:40:00.044800+00:00'
    context:
      id: xxx
      parent_id: null
      user_id: null
  description: event 'birdbuddy_new_postcard_sighting'

no MediaVideo

But if I then check in the Media browser, I do see there is a video.

madcoder · February 1, 2023, 5:10pm

I just had a sighting with a video in the BB app

So you opened the app and clicked to open the postcard, where you saw a video along with 2 images? The important question is, what did you do after viewing the image/videos? Did you finish this postcard (click “Add to Collections” after viewing), or abandon it (the “Yes, fly away” button)?

Worth noting, that a postcard can only be “finished” once: so if you click the green “Add to collection” button in the app, then the HA integration won’t see it as a new postcard, and if the integration finishes it with the collect_postcard service, then the BB app shows it as a “sighting” feed item, rather than a “new postcard” feed item. However, you can open and view a postcard in the app, without “finishing” it, if you close out of the postcard sighting screens without clicking the green button at the end.

From your trace data:

createdAt: '2023-02-01T15:38:06.004Z'
...
createdAt: '2023-02-01T15:37:52.013Z'
...
time_fired: '2023-02-01T15:40:00.044800+00:00'

This means the media were captured (rather uploaded) at 15:37:52. BB servers converted those sighting media into a “new postcard” feed item 14 seconds later at 15:38:06. And the HA integration polled the feed and fired the event 2 minutes later at 15:40:00 (polling interval is 10 minutes). If your automation also calls the collect_postcard service, then you would have had to view the postcard in the app within that 2 minute window. If it doesn’t call the service, then you likely finished the postcard in the app.

But if I then check in the Media browser, I do see there is a video.

Was it the same video from the same sighting?

If you clicked “Add to Collections” from the BB app, then that explains why you see it, because the app saved it. The app took the place of the HA integration’s collect_postcard service.
If you abandoned the postcard, and the video still appeared after the HA integration’s automation collected the postcard, then that would seem to imply that the integration worked correctly, but does not explain why the MediaVideo did not appear in sighting.medias.

I may need to review the traffic logs when viewing a postcard with video from the app (something I’ll need to do, because the process is quite involved). I don’t know of any reason that it wouldn’t work. I’m not filtering out anything from the sighting data, so what you saw in the automation trace is what we receive. The only thing that I can think of would be if the “sightingCreateFromPostcard” query takes an input flag to include video items as well. But I didn’t see that change in the graphql schema, so I don’t think that’s the case.

CumpsD · February 2, 2023, 12:46am

The automation ran in HA at 15:40, no collect_postcard service is called in the automation. The event data pasted earlier is what HA received (no video in it).

I opened the BB app later that day (can’t remember the exact hour) and got the new postcard animation with 2 photos and 1 video. I viewed them and added all 3 to collection.

The video which appears in the media browser is the same as in the BB app:

These are the 3 I saw in the BB app and which I saved to my collection.

I have no ideas in which direction to debug sadly enough. One thing I was thinking of: maybe BB servers were still processing the video which is why it is not in the event?

madcoder · February 2, 2023, 2:27am

OK. So something about the app’s postcard is able to retrieve and process the video, but the integration is not. I don’t think it’s a problem of timing, since it was a full 2 minutes from creating the postcard, to the automation running.

Also I checked the query we use when converting the postcard to sighting data, and it does support and expect MediaVideo in the response: pybirdbuddy/birds.py at e7f455d9a1903f2828fe7bab05cfa4673d5f97a9 · jhansche/pybirdbuddy · GitHub

Line 4: the graphQL mutation that’s invoked
Line 12: the expected response type
Lines 17-18: the .medias array, referencing MediaFullFields
Line 33: the MediaFullFields fragment definition, which contains
Line 37-38: contentUrl for MediaImage
Line 41-42: contentUrl for MediaVideo

So if the postcard result included a MediaVideo object, it would appear here with a contentUrl field.

However, looking back through the schema file, there was an update to the schema on 12/24 Update schema · jhansche/pybirdbuddy@547d6a9 · GitHub which added a new videoMedia field to the SightingCreateFromPostcardResult object! So, even though the .medias field contains a list of Media types (which includes both MediaImage and MediaVideo), the videoMedia field is new, and was not present when I first started on converting the postcard. Very likely they did this in order to avoid breaking old app versions (which would not have been able to understand the MediaVideo type in the response list), and since there is only one video to go with a postcard and its other photos, they instead created a new optional response field containing just the video.

So, this will be an easy fix, we’ll just merge the .videoMedia field into the .medias array; and possibly include that in the finish call.

CumpsD · February 2, 2023, 10:16am

Nice on finding out! Do you want me to create a GH issue for it?

madcoder · February 2, 2023, 3:35pm

Whoops, I just did before I saw this. You’re welcome to open a new one if you want, and I’ll close mine.

CumpsD · February 2, 2023, 3:36pm

No that’s ok

Sergeantpup · February 4, 2023, 4:08pm

This aligns with my observations. Now that I have the second one visible in HA and the cameras permanently mounted, I’ll keep an eye on their association.

I understand thanks

I realized this today when I saw it was in HA. Nice!

In the media tab, it basically made a copy of all the pictures and put them under the other feeder (which isn’t great). If I were to clear all pictures out of BB from both feeders (I already have copies) would the media tab display correctly in showing only the birds that were captured on each feeder going forward?

It’s a small sacrifice I’m willing to make to get it cleaned up in HA going forward. I just want to make sure it will work the way I expect at the other end

madcoder · February 4, 2023, 6:49pm

Don’t do that! I just released v0.0.11 today, and that version removes the feeder directories from the media browser entirely. The images aren’t easily grouped by feeder, so now they will just appear by account and then species, like they do in the BB app. If we do later start storing each image (probably not likely tbh), then we can bring back the feeder grouping. But in either event, you won’t have to make any changes in the app.

See Release v0.0.11 · jhansche/ha-birdbuddy · GitHub for more notes. This also introduces the videoMedia field in the sighting data, so you can use that to download the videos (@CumpsD). But I’m still not sure if that will work to save the videos to your BB collections. I’ll have to do more testing of that, but I’ve only had one sighting in the last ~4 weeks, so that makes it very hard to test anything

Sergeantpup · February 4, 2023, 7:29pm

I got the new version and the media tab is much cleaner now. The absolute joy I get from just getting to scroll through and watch some of these goobers

the butter butts have taken over the SE US the last 2 weeks.

CumpsD · February 6, 2023, 12:44pm

Waiting for a bird here as well

0.0.11 changed the states to lowercase as well by the way, which broke my scripts, since it was all upper case before, just something for the release notes maybe?

madcoder · February 6, 2023, 4:06pm

It’s in the release notes: translation keys can no longer be all caps, it fails hassfest verification. So they had to change to lowercase. I guess the release notes could be more clear about what that means…
Release notes updated to clarify.

CumpsD · February 6, 2023, 4:11pm

Sorry, missed that

Sergeantpup · February 9, 2023, 4:54pm

Just wanted to offer validation feedback. I initiated the update of one BB from BB. HA kept up with the status updates and when the update was complete, the update notification banner went away.

The 2nd BB I initiated update from HA. The BB was not at 40% charge yet. It waited until it hit 40% then immediately resumed update. Update status was accurately reflected in BB and in HA. Update banner in HA went away when update was complete.

All that to say, working as intended and I could cross validate all steps regardless of where I initiated the update from. Thanks for such awesome work!

CumpsD · February 9, 2023, 7:43pm

Firmware 1.2.2 just became available for my feeder. Haven’t upgraded yet, can’t find release notes on what they changed

madcoder · February 10, 2023, 2:58am

1.2 was a major game changer for battery usage!

1.2.2 is likely just a bug fix release. I haven’t got the patch yet.
They post release notes in the Facebook group. I’m not sure if they have notes posted on their website or not. If I can find a way to reliably link to release notes I’ll add that in the update entity.

Sergeantpup · February 11, 2023, 6:39pm

Hello,

I discovered Home Assistant does this cool thing with the aggregation of the Recent Visitor sensor data:

The problem with both my feeders, is they’re still reporting more Can’t Decide Which Bird, than I’d expect.

I’m diligently going into every bird I get a postcard for in the app and adding the ID. I re-read back through this thread and the best I could make out of it is, I thought these were disabled.

I’m not sure if I’m misunderstanding or if I’m observing behavior I shouldn’t be but the net is, I’m seeing more ‘can’t decide which bird’ than I think I should be in HA and every picture in my BB app is identified with a species.

Can you tell me if I’m doing something wrong?

madcoder · February 11, 2023, 7:29pm

You’re not doing anything wrong. The Recent Visitor entity reports only what BB thinks the bird should be when the postcard arrives. If it can’t confidently recognize the species, the entity will fall back on reporting the sighting type (in this case “SightingCantDecideWhichBird”) as the value (probably not a good fallback).

I could improve this by doing the same kind of inference we do with the collect_postcard strategy=best_guess, where it will look through the possible suggestions and report it as the suggestion with the highest confidence. Even that might ultimately be wrong though.

Also any changes you make in the app after the postcard arrives will not affect this entity, because the entity only looks at the incoming postcard data, it does not follow any modifications you’ve made after the postcard is opened. Alternatively, if you did modify the species in the BB app and you want the recent visitor entity to reflect the change, you could reload the BB integration, because that will cause it to scan the Feed again, which is how it populates the initial value on startup.

I’ll see if I can make a couple changes:

On CantDecideWhichBird, make a best guess inference using the same best_guess strategy we use in the collect_postcard service.
Periodically monitor the Feed items, and update the recent visitor entity with the last recognized (or unlocked) bird sighting feed item – the same thing it does on startup, but also let it do this after the postcard is opened, especially if the current recent visitor entity value is CantDecideWhichBird. That way if you modified the species in the app, it would recover and update the visitor species to whatever you selected.