Face and person detection with Deepstack - local and free!

Fezile · August 2, 2019, 6:48am

I am currently trying to build a Docker container on a Raspberry 3B but have not been successful. Would mind sharing your Dockerfile

Yoinkz · August 2, 2019, 10:19am

Maybe this is a dumb question, but what if I want to use both Deepstack-object and Deepstack-face?

I have tried with this so I hope that is correkt:
docker run -e VISION-DETECTION=True -e VISION-FACE=True -v localstorage:/datastore -p 5000:5000 deepquestai/deepstack

So finally I tried teaching the deepstack engine with a face, copied a photo to the Ubuntu server with docker and deepstack and then went back to Home Assistant and used the teach face service. When I click call Service - nothing happens.

What should I expect? How can I check whether it actually worked?

bline79 · August 3, 2019, 1:16pm

Are you seeing anyhthing in the docker logs? For example:

[GIN] 2019/08/03 - 13:15:46 | 200 | 1.349301452s | 172.17.0.1 | POST /v1/vision/detection
[GIN] 2019/08/03 - 13:15:46 | 200 | 1.888751449s | 172.17.0.1 | POST /v1/vision/detection
[GIN] 2019/08/03 - 13:15:56 | 200 | 756.543856ms | 172.17.0.1 | POST /v1/vision/detection
[GIN] 2019/08/03 - 13:15:57 | 200 | 1.381159845s | 172.17.0.1 | POST /v1/vision/detection
[GIN] 2019/08/03 - 13:15:57 | 200 | 1.87328408s | 172.17.0.1 | POST /v1/vision/detection

How about the home assistant logs? Both should show activity if things are working properly.

bline79 · August 3, 2019, 2:37pm

@robmarkcole Great work on this! It was very straightforward to get object detection running.

I’m having issues getting the state as sensor data, mainly due to my lack of skill with templating.

I’ve tried the following, none of which return the count of person(s) detected:

    person_count_driveway:
      friendly_name: 'Number of people in Driveway'
      value_template: '{{ states.driveway_person_detector.state | int}}'
      unit_of_measurement: 'bodycount'

    person_count_driveway:
      friendly_name: 'Number of people in Driveway'
      value_template: '{{ states.driveway_person_detector.state }}'
      unit_of_measurement: 'bodycount'


    person_count_driveway:
      friendly_name: 'Number of people in Driveway'
      value_template: '{{ states.driveway_person_detector.state }}'
      unit_of_measurement: 'bodycount'

Can anyone shed some light on how to achieve this?

I’d also love to get sensors on the other objects detected as well but am not sure how to pull the json array properly.

Yoinkz · August 3, 2019, 7:00pm

I don’t see anything in the log that I can see on the Home Assistant webinterface.
I do get a nice count of faces and with the object detection I do also get the persons and all other “known” objects.
So for sure I know that there is connection, I just don’t know why it doesn’t take my picture, because it doesn’t recognize me atfer I run the command teach as described.

Is it possible to teach faces directly on the Ubuntu machine where I run Deepstack within Docker?

robmarkcole · August 4, 2019, 5:08am

@bline79
This should work

person_count_driveway:
      friendly_name: 'Number of people in Driveway'
      value_template: '{{ states.driveway_person_detector.state }}'
      unit_of_measurement: 'bodycount'

It is a quirk of home assistant that a unit_of_measurement is required for a sensor to be plottable as a graph.

@Yoinkz it doesn’t matter which machine you train from. If you are training via the teach service then the most likely problem is that home assistant cannot access the image files, check they are in a directory you have whitelisted

bline79 · August 4, 2019, 3:34pm

I’m not having much success here trying to get this as a usable sensor value.

Capture

Capture2

Is it possible to somehow use the object detected as the unit_of_measurement? ie: person, dog, car etc? In addition to not being able to pull in the state info for the target, I can’t seem to get create a sensor for the “all predictions” objects. It would be great to keep metrics of the objects detected for future automatons.

robmarkcole · August 4, 2019, 4:57pm

You could break out each of the metrics using a template sensor, with the caveat that only state changes are every recorded by home assistant. Really this is not a fully fledged camera monitoring solution.
It’s strange you don’t have any sensor data for your template sensor, what does the main image_procssing.deepstack sensor show? The actual units of unit_of_measurement don’t have any meaning to HA

Yoinkz · August 4, 2019, 6:21pm

@robmarkcole Hey Rob, that could be the case. I’m just not sure I understand what exactly I should whitelist.
My Home Assistant runs on a NUC and my Docker runs on another Machine running Ubuntu.
When I want to teach the DeepStack AI with a photo, should I then place the photo on the Hassio machine or the Deepstack machine and what dir should I then whitelist ?

EDIT: I whitelisted the directory on my Hass installation where I have saved the photo, restarted HA and then tried the teach command again.
I still didn’t get any response and I couldn’t find it in the log, but when I tested it by doing processing the picture, it immediately recognized the photo and provided me with the correct name!

Thanks! This is just awesome!!!

bline79 · August 4, 2019, 6:42pm

Here’s the full state data:

I’m okay with it just recording state changes, that information would be great to see over time. Just struggling with how to get this data usable in sensor form. I’m just not sure where I’m going wrong here.

ender7 · August 4, 2019, 9:13pm

This is a great component @robmarkcole. Thanks for your work. I am seeing a strange issue. I have done the following troubleshooting:

Set up coral REST server on my rpi4 with coral USB accelerator
Checked the server locally by running curl with sample image. Processing successful.
Tested from my home assistant server (non-localhost). Processing successful.
Added RTSP generic camera like so:

camera:
   - platform: generic
     name: "driveway"
     still_image_url: http://192.168.1.252/snap.jpeg

Added image processing:

image_processing:
  - platform: deepstack_object
    ip_address: 192.168.1.131
    port: 5000
    scan_interval: 20000
    save_file_folder: /config/www/deepstack_person_images
    target: person
    confidence: 50
    source:
      - entity_id: camera.driveway
        name: driveway_detector

Restarted HA
I can now see my snapshots in HA. about 1 fps.
Searched thread for similar issue (couldn’t find any).

However, even though everything appears to be working the driveway sensor remains in a detection = unknown state and no images are hitting the REST server as far as I can tell from debug. Not sure where else to look to see why the image processor is not sending the snap.jpeg images from my camera to the REST server.

Any pointers would be greatly appreciated.

bline79 · August 4, 2019, 9:36pm

This addon is not setup to analyse RTSP (streaming) camera data. You need to analyze snapshots.

robmarkcole · August 4, 2019, 9:58pm

@ender7 select a suitable scan_interval or use an automation to call the scan service

ender7 · August 4, 2019, 11:23pm

Thanks, that worked! Didn’t know the interval was in seconds. Assumed milliseconds.

I was even able to get rid of my RTSP snapshot streams. The component took input from my ‘platform: uvc’ Unifi cameras and analyzed no problem. Turns out I didn’t need to save RTSP snapshots.

Yoinkz · August 5, 2019, 12:00pm

Could anyone perhaps share some of their automations when it comes to notifying either on ios or android?

Currently I try to create an automation when a motion detecter is triggered, then the image processing fires both for

- alias: '[Backyard] Android Notification - Motion w. picture'
  trigger:
    platform: state
    entity_id: binary_sensor.motion_sensor_xxxxxxxxxxxxxxxxxxxxxxxxx
    to: 'on'
  condition:
    - condition: state
      entity_id: alarm_control_panel.house
      state: 'armed_away'
  action:
    - delay: 1
    - service: image_processing.scan
      entity_id: image_processing.backyard_face
    - service: image_processing.scan
      entity_id: image_processing.backyard_object
    - delay: 2
    - service_template: '{% if (image_processing.backyard_object.state | int) > 0 %} notify.android {% endif %}'
      data_template:
        message: "(Known / Unknown) person detected at the front door"
        image: 'http://xxx.xxx.xxx.xxx:8123/local/deepstack_person_images/backyard/deepstack_latest_person.jpg?{{now().second}}'

But, as you can see I call both the deepstack_face and deepctack_object. My idea was to sent the notification if the deepstack_object recognizes a person in the picture, but I would like if the face is a known one to be a part of the message.

Does someone have an idea of how to accomplish that?

robmarkcole · August 5, 2019, 12:20pm

My own is

sensor:
  - platform: template
    sensors:
      room_persons:
        friendly_name: "People in room"
        unit_of_measurement: 'Persons'
        value_template: "{{states.image_processing.room_main.state}}"

robmarkcole · August 5, 2019, 12:47pm

Very exciting announcement on the Deepstack forum by @OlafenwaMoses, this is really a game changer

cmille34 · August 5, 2019, 2:05pm

@robmarkcole Great work on this and your other custom components! I got the email from Deepstack this morning announcing the unlimited instances and it sparked my memory to finally get around and ask if this component would work to detect which car(s) are in a driveway? I have a black sedan and my wife has a red SUV. Could this component be used with the camera feed above my garage to detect which of the (2) vehicles are in the drive way? If not this component, either of your other ones? Thanks!

robmarkcole · August 5, 2019, 2:37pm

@cmille34 you can detect the presence of a car with the standard object detection model. To identify if its a red or a black car, you would need to create a custom model, please see:

cmille34 · August 5, 2019, 3:22pm

Thanks! Would you recommended the Deepstack or Machinebox component for this type of use?