Custom Component for object detection and image analysis with Azure computer vision API

Share your Projects! Custom Integrations
1

I made a custom Component for Home assistant to get image data from Azure Computer Vision Cognitive Service.

Detailed documentation: Azure Custom Component - Github

TLDR; You can easily integrate azure computer vision services and detect objects in an image. I personally use this for human detection when nobody is in the home and motionEye detects motion in an image.

How to setup:

azurecv:
  api_key: !secret azurecv_key
  url: !secret azurecv_url

How to call service:

- service: azurecv.analyze
  data_template:
    filepath: '/share/motioneye/Camera2/2021-01-30/20-13-23.jpg'
    haContext: 'Camera Name'
    detect: 'dog,person'

Response is available as an event on the home assistant bus. Sameple automation to listen to response:

- alias: Notify when person detected on a cam
  initial_state: true
  trigger:
    platform: event
    event_type: azurecv_image_analyzed
  condition:
    - condition: state
      entity_id: input_boolean.home_armed
      state: 'on'
    - condition: template
      value_template: >
        {{ trigger.event.data.detection.person == True }}
  action:
    - service: script.notify_home_alarm
      data_template:
        message: >
          Person Detected on {{ trigger.event.data.haContext }}
1 Like
2

More data is available as a response from azure. Sample event data:

{
    "event_type": "azurecv_image_analyzed",
    "data": {
        "caption": {
            "text": "a dog sitting on a wood floor in a living room",
            "confidence": 0.49372708797454834
        },
        "tags": "floor,indoor,chair,wall,table,coffee table,living,room,couch,desk,kitchen & dining room table,house,studio couch,flooring,sofa bed,bed,hardwood,furniture,cluttered",
        "objects": "chair,dog,chair,chair",
        "detection": {
            "dog": true,
            "person": false
        },
        "method": "analyze",
        "haContext": "Camera Name",
        "filePath": "/share/motioneye/Camera2/2021-01-30/20-13-23.jpg"
    },
    "origin": "LOCAL",
    "time_fired": "2021-01-31T04:23:57.394243+00:00",
    "context": {
        "id": "dbc796a1c247bc81afa8353632d7fa40",
        "parent_id": null,
        "user_id": null
    }
}
3

Just wanted to say thank you for creating this! Being far more comfortable in Azure than AWS world I was looking for an alternative for Rekognition and this is exactly what I needed.

Haven’t had the time to set this up yet but looking forward to it!

4

I’m currently trying this integration, but I can’t get it to work. When I try to send an event, using Dev tools, I never get a response from Azure. On Azure, in monitoring, I also don’t see any request. My HA set-up isn’t giving any logs?

Thank you!

{
    "event_type": "azurecv_image_analyzed",
    "data": {
        "fileurl": "https://www.allport.be/images/inspiration/flex-2-poorten-1.jpg",
        "haContext": "camera"
    },
    "origin": "REMOTE",
    "time_fired": "2022-01-07T09:47:58.463301+00:00",
    "context": {
        "id": "xxxxxx501c1b991525d6a0896fe7df7e",
        "parent_id": null,
        "user_id": "xxxx11efa4d45c58f158ff3d7da8622"
    }
}

Edit, it seems that there is an error.

2022-01-07 10:26:14 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [547924148032] Error handling message: Unknown error

Traceback (most recent call last):

File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 27, in _handle_async_response

await func(hass, connection, msg)

File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 527, in handle_execute_script

await script_obj.async_run(msg.get("variables"), context=context)

File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1260, in async_run

await asyncio.shield(run.async_run())

File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 363, in async_run

await self._async_step(log_exceptions=False)

File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 381, in _async_step

await getattr(self, handler)()

File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 584, in _async_call_service_step

await service_task

File "/usr/src/homeassistant/homeassistant/core.py", line 1495, in async_call

task.result()

File "/usr/src/homeassistant/homeassistant/core.py", line 1534, in _execute_service

await self._hass.async_add_executor_job(handler.job.target, service_call)

File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run

result = self.fn(*self.args, **self.kwargs)

File "/config/custom_components/azurecv/__init__.py", line 123, in onAnalyze

response = azureCV.analyze(

File "/config/custom_components/azurecv/AzureCV.py", line 44, in analyze

response = self.parser.polish(data)

File "/config/custom_components/azurecv/ParseResponse.py", line 3, in polish

dataObj = self.clean(data)

File "/config/custom_components/azurecv/ParseResponse.py", line 12, in clean

if 'requestId' in rawData:

TypeError: argument of type 'NoneType' is not iterable