Face and person detection with Deepstack - local and free!

[DivanX10]

If you wish, you can always do it. For example, I took the first step to implement my plan (the plan itself is described below). Watch video

Next, it remains to buy a camera in this room and try to realize what you have planned

In the plans, but not yet done yet. This is turning off the sound on the playing device if I answered an incoming call. I plan to do so. In Russia we have our own smart speaker with a voice assistant Yandex Alice, an analogue of your Alexa. Our speakers can play the same music in all rooms where there are speakers. If I am in any room and if I answered an incoming call, the music will stop playing in that room.

Текст на русском (Text in Russian)

При желании вы всегда сможете это сделать. Я к примеру сделал первый шаг для реализации своего плана (сам план описан ниже). Смотрим видео

Далее осталось купить камеру в эту комнату и попробовать реализовать задуманное

В планах, но пока не сделал еще. Это выключение звука на играющем устройстве, если я ответил на входящий вызов. Планирую сделать так. У нас в России есть своя умная колонка с голосовым помощником Яндекс Алиса, аналог вашей Алексы. Наши колонки умеют играть одну музыку и во всех комнатах, где стоят колонки. Если я нахожусь в какой либо комнате и если я ответил на входящий звонок, то музыка прекратит играть именно в этой комнате.

[jaaneo (Alejandro S)]

I don’t have permission to view the video :’(

[DivanX10]

I made a video of how you can still implement deepstack in automation. The speech is in Russian, but you can include subtitles with translation into your native language

Home Assistant. Deepstack. Домашняя сигнализация

Example of automation

# Automation No. 1. Security: Deepstack. Turning off the alarm. Human presence detected

alias: >-
  Security: Deepstack. Turning off the alarm. Human presence detected
description: >-
  If someone is at home with the alarm turned on, the system will ask look into the camera for 
  identification. If successful, the guard will be removed. In case of failure, you can remove the guard 
  using a code. If not successful in all cases, a notification will be sent to the host
trigger:
  - type: motion
    platform: device
    device_id: b4d846ac15ab22d4e7f1456a5ced452d
    entity_id: binary_sensor.koridor_datchik_dvizheniia_01_occupancy
    domain: binary_sensor
condition:
  - condition: state
    entity_id: alarm_control_panel.home_alarm
    state: armed_away
  - condition: state
    entity_id: input_boolean.security_system_identification
    state: 'off'
  - condition: template
    value_template: >-
      {{is_state("binary_sensor.koridor_datchik_dveri_glavnaia_dver_contact",
      "off") and (now() -
      states.binary_sensor.koridor_datchik_dveri_glavnaia_dver_contact.last_changed).seconds
      < 60 }}
action:
  - service: counter.reset
    target:
      entity_id: counter.security_system_identification
  - service: input_boolean.turn_on
    target:
      entity_id: input_boolean.security_system_identification
  - service: browser_mod.navigate
    data:
      navigation_path: /lovelace-tablet/10
  - service: light.turn_on
    target:
      entity_id: light.galaxy_tab_a_2016_screen
  - service: media_player.play_media
    target:
      entity_id: media_player.tablet_samsung_a_2016
    data:
      media_content_type: music
      media_content_id: notification human presence detected filipp.mp3
  - wait_for_trigger:
      - platform: state
        entity_id: media_player.tablet_samsung_a_2016
        from: playing
        to: 'off'
    continue_on_timeout: false
  - service: script.start_face_detection
  - wait_for_trigger:
      - platform: state
        entity_id: image_processing.detect_face_eufy_camera
        to: unknown
    continue_on_timeout: true
    timeout: '5'
  - service: counter.increment
    target:
      entity_id: counter.security_system_identification
mode: single

================================================================================================================
# Automation No. 2. Security: Deep stack. Turning off the alarm. The identity has not been established

alias: 'Security: Deep stack. Turning off the alarm. The identity has not been established'
description: >-
  If someone is at home with the alarm turned on, the system will ask look into the camera for 
  identification. If successful, the guard will be removed. In case of failure, you can remove the guard 
 using a code. If not successful in all cases, a notification will be sent to the host
trigger:
  - platform: state
    entity_id: counter.security_system_identification
    from: '0'
    to: '1'
    id: Number 1
  - platform: state
    entity_id: counter.security_system_identification
    from: '1'
    to: '2'
    id: Number 2
condition:
  - condition: state
    entity_id: sensor.persons_names_hall
    state: unknown
  - condition: state
    entity_id: input_boolean.security_system_identification
    state: 'on'
action:
  - choose:
      - conditions:
          - condition: trigger
            id: Number 1
        sequence:
          - service: media_player.play_media
            data:
              media_content_type: music
              media_content_id: notification failed to recognize the identity filipp.mp3
            target:
              entity_id: media_player.tablet_samsung_a_2016
          - wait_for_trigger:
              - platform: state
                entity_id: media_player.tablet_samsung_a_2016
                to: 'off'
                from: playing
            continue_on_timeout: false
          - service: script.start_face_detection
          - wait_for_trigger:
              - platform: state
                entity_id: image_processing.detect_face_eufy_camera
            continue_on_timeout: true
            timeout: '5'
          - service: counter.increment
            target:
              entity_id: counter.security_system_identification
      - conditions:
          - condition: trigger
            id: Number 2
        sequence:
          - service: media_player.play_media
            data:
              media_content_type: music
              media_content_id: notification the persons face is not recognized filipp.mp3
            target:
              entity_id: media_player.tablet_samsung_a_2016
          - wait_for_trigger:
              - platform: state
                entity_id: media_player.tablet_samsung_a_2016
                to: 'off'
                from: playing
            continue_on_timeout: false
          - service: counter.reset
            target:
              entity_id: counter.security_system_identification
          - service: timer.start
            data:
              duration: '0'
            target:
              entity_id: timer.security_system_identification
    default: []
mode: restart

================================================================================================================
# Automation No. 3. Security: Deepstack. Turning off the alarm. Identity is established

alias: >-
  Security: Deep stack. Turning off the alarm. Identity is established
description: >-
  If someone is at home with the alarm turned on, the system will ask look into the camera for 
  identification. If successful, the guard will be removed. In case of failure, you can remove the guard 
  using a code. If not successful in all cases, a notification will be sent to the host
mode: single
trigger:
  - platform: state
    entity_id: sensor.persons_names_hall
condition:
  - condition: state
    entity_id: input_boolean.security_system_identification
    state: 'on'
  - condition: state
    entity_id: alarm_control_panel.home_alarm
    state: armed_away
action:
  - choose:
      - conditions:
          - condition: state
            entity_id: sensor.persons_names_hall
            state: user 01
        sequence:
          - service: media_player.play_media
            target:
              entity_id: media_player.tablet_samsung_a_2016
            data:
              media_content_type: music
              media_content_id: notification the identity is established user 01 welcome home.mp3
          - service: script.security_system_identity_is_established
      - conditions:
          - condition: state
            entity_id: sensor.persons_names_hall
            state: user 02
        sequence:
          - service: media_player.play_media
            target:
              entity_id: media_player.tablet_samsung_a_2016
            data:
              media_content_type: music
              media_content_id: notification the identity is established user 02 welcome home.mp3
          - service: script.security_system_identity_is_established #the script description is below
    default: []


# Script script.security_system_identity_is_established
alias: 'Security: Identification. Identity is established'
sequence:
  - service: alarm_control_panel.alarm_disarm
    data:
      code: '1234'
    target:
      entity_id: alarm_control_panel.home_alarm
  - service: input_boolean.turn_off
    target:
      entity_id: input_boolean.security_system_identification
  - service: timer.cancel
    target:
      entity_id: timer.security_system_identification
  - delay:
      hours: 0
      minutes: 0
      seconds: 5
      milliseconds: 0
  - service: counter.reset
    target:
      entity_id: counter.security_system_identification
mode: single
icon: mdi:security

================================================================================================================
# Automation No. 4. Security: Deep stack. Turning off the alarm. Time is up

alias: 'Security: Deep stack. Turning off the alarm. Time is up'
description: >-
  If someone is at home with the alarm turned on, the system will ask look into the camera for 
  identification. If successful, the guard will be removed. In case of failure, you can remove the guard 
  using a code. If not successful in all cases, a notification will be sent to the host
trigger:
  - platform: state
    entity_id: timer.security_system_identification
    from: active
    to: idle
    id: Время истекло
  - platform: state
    entity_id: alarm_control_panel.home_alarm
    to: disarmed
    id: Security is off
condition:
  - condition: state
    entity_id: input_boolean.security_system_identification
    state: 'on'
  - condition: state
    entity_id: sensor.persons_names_hall
    state: unknown
action:
  - choose:
      - conditions:
          - condition: trigger
            id: Time is up
          - condition: state
            entity_id: alarm_control_panel.home_alarm
            state: armed_away
        sequence:
          - service: media_player.play_media
            data:
              media_content_type: music
              media_content_id: >-
                notification the time is up please leave the apartment
                filipp.mp3
            target:
              entity_id: media_player.tablet_samsung_a_2016
          - service: input_boolean.turn_off
            target:
              entity_id: input_boolean.security_system_identification
      - conditions:
          - condition: trigger
            id: Security is off
        sequence:
          - service: timer.cancel
            target:
              entity_id: timer.security_system_identification
          - service: input_boolean.turn_off
            target:
              entity_id: input_boolean.security_system_identification
          - service: media_player.play_media
            data:
              media_content_type: music
              media_content_id: notification the security system is turned off filipp.mp3
            target:
              entity_id: media_player.tablet_samsung_a_2016
    default: []
mode: single

[DivanX10]

Great news.

The author of the deepstack client integration has added new features, namely, photo editing in the photo gallery. We can now delete photos, rename the name for the photo.

GitHub - t0mer/deepstack-trainer: deepstack trainer is a Flask powerd, easy to...

Download and install the latest version of the deepstack 1.0.2 client

[Eric9453]

Do you know how to update deepstack-trainer?
I’d already installed 1.01 in docker of synology.
Thank you.

[DivanX10]

I have discovered a new project DeepStack_ActionNET and it mentions about custom models. Tell me, how can this case be integrated into a home assistant? And how to properly configure custom models in docker. I kind of figured out how to configure it, but there is no clear understanding of how to configure it yet. How to recognize what an object is doing through a home assistant, as in the case of objects?

How I set up custom models in docker, but it was done without proper instructions and it cannot be called an instruction

Created folders /modelstore/detection
$ mkdir -p /modelstore/detection

Stopped the container DeepStack_Server

In the settings of the DeepStack_Server container, in the volume section, I specified the paths for connecting folders

And then I specified a variable. Here I do not know whether it is necessary to specify or not. This is not written in the instructions, although it is mentioned for Windows

An entry appeared in the log v1/vision/custom/actionnetv2. It turns out that the model is hooked

Текст на русском (Text in Russian)

Я обнаружил новый проект DeepStack_ActionNET и там упоминается про custom models. Скажите, как можно это дело интегрировать в домашний помощник? И как правильно настроить custom models в docker. Я вроде бы как бы понял, как надо настраивать, но пока нет четкого понимания как это надо настраивать. Как распознать, что делает объект через домашний помощник, как в случае с объектами?

Как я настроил custom models в docker, но это сделано без должной инструкции и нельзя это назвать инструкцией

Создал папки /modelstore/detection
$ mkdir -p /modelstore/detection

Остановил контейнер DeepStack_Server

В настройках контейнера DeepStack_Server, в разделе том я указал пути для подключения папок

И тут я указал переменную. Тут я не знаю, нужно ли указывать или нет. Об этом не пишется в инструкции, хотя об этом упоминают для Windows

В журнале появилась запись v1/vision/custom/actionnetv2. Получается, что модель подцепилась

[Ierlandfan]

Trying to setup Deepstack and folder_watcher like motion-activated-image-capture-and-classification-of-birds

If i copy/paste the automation snippets:

- action:
   data_template:
     file_path: ' {{ trigger.event.data.path }} '
   entity_id: camera.dummy
   service: camera.local_file_update_file_path
 alias: Display new image
 condition: []
 id: '1520092824633'
 trigger:
 - event_data:
     event_type: created
   event_type: folder_watcher
   platform: event
- id: '1527837198169'
 alias: Perform image classification
 trigger:
 - entity_id: sensor.last_added_file
   platform: state
 condition: []
 action:
 - data:
     entity_id: camera.dummy
   service: image_processing.scan
- action:
 - data_template:
     message: Class {{ trigger.event.data.id }} with probability {{ trigger.event.data.confidence
       }}
     title: New image classified
     data:
       file: ' {{states.camera.local_file.attributes.file_path}} '
   service: notify.pushbullet
 alias: Send classification
 condition: []
 id: '1120092824611'
 trigger:
 - event_data:
     id: birds
   event_type: image_processing.image_classification
   platform: event

I get the error:

* while parsing a block collection in "/config/automations.yaml", line 1, column 1 expected <block end>, but found '<block mapping start>' in "/config/automations.yaml", line 20, column 2
* mapping values are not allowed here in "/config/automations.yaml", line 26, column 16
* mapping values are not allowed here in "/config/automations.yaml", line 26, column 15
* while parsing a block collection in "/config/automations.yaml", line 1, column 1 expected <block end>, but found '<block mapping start>' in "/config/automations.yaml", line 21, column 2

Any idea?

[nordeep]

@Ierlandfan

Any idea?

Seems incorrect spaces(indent) in automations.yaml.
Check project on GitHub - HASS-Machinebox-Classificationbox/bird_project at master · robmarkcole/HASS-Machinebox-Classificationbox · GitHub has correct syntax.

[DivanX10]

Instructions on how to install the server and client side of DeepStack in the Portainer CE Home Assistant

Why do I need to install Portainer CE and how should I install it?

Instructions in English

This text will be hidden

Installing the DeepStack server

Download the image deepquestai/deepstack
deepquestai/deepstack:gpu – for hardware with a graphics card
deepquestai/deepstack:cpu – for hardware without a graphics card - for the processor

1. First you need to download the image deepquestai/deepstack:cpu (I choose to use the processor)
   

   

2. Creating a new container for the server DeepStack
   

   

3. When creating a container, specify the following
   • Specify the name of the container
   • Specifying the image for the container
   • Specify the ports. The required port is 5000, and any port can be specified instead of port 5100
   • Start the container so that it creates the primary settings and then turn it off
   

   

After the first launch of the container to create settings, the container must be turned off for further configuration

When the container is turned off, click on it and get inside the container, where you need to click on "Duplicate/Edit”

4. At the very bottom, go to the Env section (digit 1) and add two lines, click on Add an environment variable (digit 2)
   VISION-DETECTION = True
   VISION-FACE = True
   

   

5. After you have specified two lines, you need to apply the parameters and launch the container by clicking the button Deploy the container
   

   

6. Click on Replace, after which the DeepStack server container will start
   

   

7. The Deep Stack_Server container is running
   

   

8. In the browser, enter the IP address and port. In my case http://192.168.1.108:5100
   If everything is done correctly, we will receive a welcome screen saver that the DeepStack server is running and activated. This completes the setup.
   

   

We put the client DeepStack (deepstack-trainer)

Download the image techblog/deepstack-trainer

1. First you need to download the image techblog/deepstack-trainer:1.0.2
   (as of the current date 30.10.2021 this is the latest version of the image)
   

   

2. Create a new container for the client DeepStack
   

   

3. When creating a container, specify the following
   • Specify the name of the container
   • Specifying the image for the techblog/deepstack-trainer container:1.0.2
   • Specify the ports. The required port is 8080, and any port can be specified instead of port 5150
   • Start the container so that it creates the primary settings and then turn it off
   

   
   
   
   

4. At the very bottom, go to the section Env (digit 1), find the line DEEPSTACK_HOST_ADDRESS and add the IP address and port of the Deepstack server (IP address of the Home Assistant). Specify via http:// . In my case it is http://192.168.1.108:5100
   

   

5. Click on Replace, after which we will launch the DeepStack client container
   

   

6. The DeepStack_Client_Trainer container is running
   

   

7. In the browser, enter the IP address and port. In my case http://192.168.1.108:5150
   If everything is done correctly, the DeepStack client page will open. This completes the setup.
   

   

Инструкция на русском (Instructions in Russian)

Инструкция, как ставить серверную и клиентскую часть DeepStack в Portainer Home Assistant

Почему нужно ставить Portainer CE и как его надо ставить?

Ставим сервер DeepStack

Скачать образ deepquestai/deepstack
deepquestai/deepstack:gpu – для железа с графической картой
deepquestai/deepstack:cpu – для железа без графической карты - для процессора

1. Для начала надо скачать образ deepquestai/deepstack:cpu (я выбираю использовать процессор)
   

   

2. Создадим новый контейнер для сервера DeepStack
   

   

3. При создании контейнера указываем следующее
   • Указываем имя контейнера
   • Указываем образ для контейнера
   • Указываем порты. Обязательный порт 5000, а вместо порта 5100 можно указать любой порт
   • Запустите контейнер, чтобы он создал первичные настройки и после этого выключите его

После первого запуска контейнера для создания настроек, контейнер нужно выключить для дальнейшей настройки

Когда контейнер будет выключен, нажмите на него и попадем во внутрь контейнера, где нужно нажать на “Duplicate/Edit”

4. В самом внизу, переходим в раздел Env (цифра 1) и добавляем две строчки, нажимаем на Add an environment veriable (цифра 2)
   VISION-DETECTION = True (распознавать объекты)
   VISION-FACE = True (распознавать лица)
   

   

5. После того как указали две строчки, необходимо применить параметры и запустить контейнер, нажав на кнопку Deploy the container
   

   

6. Нажимаем на Replace, после чего у нас запустится контейнер сервер DeepStack
   

   

7. Контейнер DeepStack_Server запущен
   

   

8. В браузере вводим IP адрес и порт. В моем случае http://192.168.1.108:5100
   Если все сделано верно, то получим приветственную заставку о том, что сервер DeepStack запущен и активирован. На этом настройка закончена.
   

   

Ставим клиент DeepStack (deepstack-trainer)

Скачать образ techblog/deepstack-trainer здесь

1. Для начала надо скачать образ techblog/deepstack-trainer:1.0.2
   (на текущую дату 30.10.2021 это последняя версия образа)
   

   

2. Создадим новый контейнер для клиента DeepStack
   

   

3. При создании контейнера указываем следующее
   • Указываем имя контейнера
   • Указываем образ для контейнера techblog/deepstack-trainer:1.0.2
   • Указываем порты. Обязательный порт 8080, а вместо порта 5150 можно указать любой порт
   • Запустите контейнер, чтобы он создал первичные настройки и после этого выключите его
   

   
   
   
   

4. В самом внизу, переходим в раздел Env (цифра 1), находим строчку DEEPSTACK_HOST_ADDRESS и добавляем IP адрес и порт сервера Deepstack ( IP адрес Home Assistant). Указывать через http:// . В моем случае это http://192.168.1.108:5100
   

   

5. Нажимаем на Replace, после чего у нас запустится контейнер клиент DeepStack
   

   

6. Контейнер DeepStack_Client_Trainer запущен
   

   

7. В браузере вводим IP адрес и порт. В моем случае http://192.168.1.108:5150
   Если все сделано верно, то откроется страница клиента DeepStack. На этом настройка закончена.
   

   

[Ierlandfan]

Did anyone figured out how to tied both options together?
I am struggling with this:
(Folder wachter) (Works)
Motion detected > Send image for object recognition (Works)
If summary contains “person” send image for face recognition (No idea)
If summary contains “car” send image for logo recognition (No idea)

Face recognition and logo recognition are working manually
Can someone give me some pointers. I probably need some if else functions.

[robmarkcole (Robin)]

@DivanX10 you need to configure custom_model arg and the targets

[DivanX10]

I don’t understand you. Can you show by example how to set it all up correctly in the screenshots to make this custom model work? Here I asked a question and the question remained unanswered. I set it up, an entry appeared in the logs, and then how do I specify everything? There are a lot of questions, but no answers. Please make an instruction with screenshots of how to use custom models.

Текст на русском (Text in Russian)

Не понимаю вас. Можете показать на примере как правильно все это настроить на скриншотах, чтобы заработал этот custom model? Вот тут я задал вопрос и вопрос остался не отвеченным. Я настроил, в логах появилась запись, а дальше как мне указывать то все? Вопросов много, а ответов нет. Пожалуйста сделайте инструкцию со скриншотами как можно использовать пользовательские модели.

[Anemoi (Robert)]

Questions for you all.

How are you handling triggering the deepstack image processing service?

Previously I had an automation that ran the image processing every 2 seconds but I always felt this was a waste of resources. So set up motion events using MQTT from Blue Iris. This works but seems to be missing stuff. For example in the past as I walked to my detached garage the two second loop would pick me up in the camera and I would get the alert message of positive person detection. I would then get a second one when the car backed into the drive way (same camera). Now using the blue iris motion events I am only getting the second detection and the image is of my car much further out of the garage then previous which leads me to think there is some delay.

Is there a third option I am missing or should I just go back to the 2 second loop?

[manalishi]

@DivanX10 and @robmarkcole
I’ve investigated deepstack with object and face detection now for a few days and would like to share my findings:

1. Hardware:
   a) Jetson Nano 4GB is not sufficient for object and face detection, since deepstack requires cpu power
   b) Intel i7 with GTX 1060 workstation and i7 Notebook with M1200 Graphic performance works fine
   c) VMWare esxi on xeon 5560 cpu with ubuntu 20.04 and docker on an nfs ssd datastore without gpu support is still ok. Object detection below 1sec, face 3sec. Didn’t check on Windows VM yet

Sorry guys, need to apologize. My mistake in defining the start command.
2. Windows vs Docker
a) Windows CPU/GPU on a physical host doesn’t support object, face and scene, but does support custom models
b) Windows CPU on a VM doest work at all (python errors during http request)
c) Docker on Ubuntu does support object, face and scene, but doesn’t support custom models.

3. Deepstack Versions:
   a) Windows: gpu-2021.09.1, cpu-2021.09.1
   b) Jetson: jetpack-2021.09.1
   c) esxi VM: cpu-x5-beta

I got stuck at the same point like DivanX10 when it’s about to configure “custom_model” arg in order to make the custom models work on docker. Different to his experience I didn’t see my custom model listed in the log when starting the container.

my docker start env:

-e VISION-DETECTION=True -e VISION-FACE=True -e VISION-SCENE=True -e MODELSTORE-DETECTION="/modelstore/detection" -v /mnt/datastore/custom_models:/modelstore/detection

my windows start env : UPDATED

–MODELSTORE-DETECTION “C:\DeepStack\cust_models” --VISION-DETECTION True --VISION-FACE True --VISION-SCENE True

Can someone correct my docker command in a way it will use my custom model?. I tried to find “ARG” in docker docs but couldn’t make it work.
Should object, face and scene work on windows?

tnx

[elmo2002]

Hi all, I’m kind of new to home assistant so just finding my feet. I would like to run deep stack to allow me to carry out object detection and ideally facial recognition after a few failed break in attempts.
I’m really confused how I install deep stack as I am running Home assistant on a Windows 10 machine, within Oracle VM VirtualBox with a ubuntu (64-bit) operating system.

I have tried to install Deep stack CPU version for windows but it does seem to start up and when i run the image processing in HA it times out. Everything i seem to read shows how to install Deep stack in docker. However I do not have and am not running docker.

Im lost and confused so any guidance would be very very greatly appreciated. Thanks

[checking12 (Sander)]

how do you run de app , dont forget to add the start up param to activate endpoints

@manalishi , what you mean with no support in windows, there is native app that runs fine (see screenshot just here above)
I have it running on windows 10 with old gen i5 and 12Gb mem (where most is occupied by MySql and oracle virtual box machine)
on average, responses are below 1 second:

[manalishi]

give this version a try. works on my ubuntu20.04 VM
deepquestai/deepstack:cpu-x5-beta:cpu-x5-beta

[manalishi]

it doesn’t load the object detection

I7-2600K 16GB RAM

[manalishi]

Custom Models are loaded

[elmo2002]

@manalishi How do I go around adding that in my VM? This is where I get confused. My apoligies if I sound completely stupid.