Sending image entity to generative AI

Configuration
1

I’m trying to use this new service to send an image for evaluation.

If I send a static file, like this it works:

service: google_generative_ai_conversation.generate_content
data:
  image_filename: /media/snapshot.png
  prompt: Can you describe what you see?

But I would like to send an entity image (from Frigate), like this:

service: google_generative_ai_conversation.generate_content
data:
  prompt: Can you describe what you see?
  image_filename: >-
    /api/image_proxy/image.exterior_person?token=xxxxxxx

And I’m getting this error:

Failed to call service google_generative_ai_conversation.generate_content. Cannot read `/api/image_proxy/image.exterior_person?token=xxxxxxx`, no access to path; `allowlist_external_dirs` may need to be adjusted in `configuration.yaml`

I’ve tried adding ‘/api/’, ‘/api/image_proxy/’ and the configuration ends up invalid:

Configuration errors
Invalid config for 'homeassistant' at configuration.yaml, line 53: Not a directory 'allowlist_external_dirs->0', got '/api/image_proxy/'

Is there a way for me to send an image entity in this service?
I’ve seen some people saving a snapshot through a service call and then sending the snapshot, but in the case of an entity image it is not possible.

In short, how do I send my image saved in an HA entity to this service? Thanks in advance.

1 Like
2

I’m not positive, but based on another post on here, I don’t think you actually give it the file itself, you give it the URL to the file.

3

Using the same link that works through the browser like this:

https://my.ha.com/api/image_proxy/image.exterior_person?token=xxxx

Ends up giving me the same external_dirs error. I wonder if only files under the local www folder can be used.

4

I believe if you want to allow HTTP access to directories other than /config/www then you need to add those directories to configuration.yaml.

No different than any other webserver, the www folder is the root of where the webserver serves files from.

5

What you’re saying makes sense, but the whole “/api/image_proxy” is not really a directory, right?

Thanks again

6

I’d have to go and look. It might be, but if it is, it’s not a subdirectory of /www.

7

The docs suggest to me it has to be an actual file, not a web path to generate or capture a file.

Maybe you need to save the output of that API request to a file, and then send it to the AI engine ?

8

I am running into this same issue. Any hints on how to do this?

9

Theres a service called camera.snapshot.

You oculd use this to save the image to a file, and then send that image off to the AI engine.

Something like:

- service: camera.snapshot
    data:
      filename: /local/doorbell.jpg
    target:
      entity_id: camera.doorbell
- service: google_generative_ai_conversation.generate_content
    data:
      image_filename:  /local/doorbell.jpg
      prompt: Can you describe what you see?
10

If camera snapshot doesnt work in your case, this likely will:

doorbell_snapshot:
  alias: Doorbell Snapshot
  sequence:
    - service: downloader.download_file
      data:
        url: >-
          http://[enter your HA IP]:8123{{state_attr('image.doorbell_event_image','entity_picture')}}
        filename: visitor.jpg
        overwrite: true
2 Likes
11

You mean the same docs that are so clearly written that there is an entire thread of people guessing about how to make this work?

Ah, yes. Clearly they are quite useful. Lol

12

yeah — I agree.
I contemplated submitting a PR saying “Note: THis has to be a file, on the file system” or something to that effect.

13

Wait, so you know that for certain? Your previous posts read a bit like speculation…

14

In the playing around with it i did, thats how it behaves.

The documention infers it, but doesnt explicity state it.

1 Like
15

Thanks a lot for this!

The snapshot option is not available for image entities, so the downloader integration solved the main problem! It’s working like this:

New event from frigate
Download snapshot from image entity
Send downloaded snapshot to AI
Send notification with AI text

Ideally the image entity should be directly compatible without an extra integration in between, but the results are the same.

16

Would you mind sharing the exact code you are using for the process? I’d like to mess around with this a bit as well.

Thanks!

17

Sure, here it is. First I added this to the config folder:

downloader:
  download_dir: /config/www/downloads

And then I made an script like this:

alias: AI - describe
sequence:
  - service: downloader.download_file
    data:
      overwrite: true
      filename: "{{ select_image.split('.')[-1] }}.jpg"
      url: https://yourha.com{{state_attr(select_image,'entity_picture')}}
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
  - service: google_generative_ai_conversation.generate_content
    data:
      image_filename: /config/www/downloads/{{ select_image.split('.')[-1] }}.jpg
      prompt: Prompt text here
    response_variable: ai
    continue_on_error: true
  - service: notify.mobile_app
    data:
      data:
        ttl: 0
        priority: high
        channel: camera_stream
        image: https://yourha.com{{state_attr(select_image,'entity_picture')}}
      title: Alert!  {{state_attr(select_image,'friendly_name')}}
      message: "{{ai.text}}"
mode: queued
icon: mdi:robot-outline
fields:
  select_image:
    selector:
      entity:
        filter:
          - domain: image
    name: Image
    description: Select an image
    required: true
max: 10

You’ll need to select an image entity, then the script will download it to the configured folder, send it to Google AI and send a notification with the text generated. For a camera entity some extra modifications might be necessary

18

Has anyone come up with an alternative approach to grab an external image, where the image URL requires authentication ?

It appears, unfortunately, that the downloader.download_file service doesnt work when the URL requires credentials.

For the moment ive built something outside of HA, on my local network, that proxies the request, but I’d rather not need that.