Face and person detection with Deepstack - local and free!

[dougbaptista (Douglas Machado Baptista)]

Ignore it, it worked. It is unknown when there is no person at that moment, I thought it would change to 0.

I saw that it is in HACS, so in HassOS / Hass.Io is it already possible?

great job.

[MSmithHA]

Cool stuff! Can’t wait to get started! I have 6 rtsp wyze cams and will grow to 15. My plan was to install Blue Iris on a 6 year old Core i3 Windows 10 laptop with 8 GB ram (cheap laptop when I got it) and using Deepstack for processing motion events on each camera in realtime to determine if there was a person detected and if so getting an event fired. I would also prefer to have facial recognition as well. Blue Iris will also be my NVR, so there will continuous 24x7 recording of them. Would I be better off:

a) Running Deepstack alongside Blue Iris on the same laptop, or
b) Running Deepstack alongside Blue Iris on the same laptop but adding an Intel NCS to, or
c) Buying a dedicated RaspPi 4 with an Intel NCS solely for running Deepstack or
d) Forget about it…can’t be done without a much beefier setup. Max # of cameras would be [please fill in].

Thanks! Nice work!

[danbutter]

I doubt 15 wifi cams will do well with any type of nvr.
You should buy Poe cams.
Running blue iris on a laptop doesn’t seem like a great idea…how will you hook up extra storage? Or won’t you be recording any of your cams?
Lots of people have issues with deepstack running on win 10. Not saying it won’t work, but Linux seems to be preferred.
I think you need to go over to ipcamtalk.com and spend a week in the fourms. You can learn more about everything I said here from that forum.

[typeonegative (Gunnar)]

can you share it as code please?

[MSmithHA]

Hmmm…Ok. The only reason I was going with Win 10 was because it was a Blue Iris requirement. I wonder though if MotionEyeOs is sufficient if I offload the work to DeepStack for person detection and have that be my gateway for deciding if I should get a notification.

I do plan to record but plan to continuously upload to AWS S3. I’ll have a lambda that automatically rotates out the old.

If I put Deepstack of R Pi 4 with Intel NSC, how many simultaneous camera streams could it handle for processing person detection (preferably also with face detection)? Not necessarily at parallel but with keeping the latency of all low (ideally < 3 seconds). Actually though, it’ll only process motion events at night time and while I have 2 cats and lots of windows, it should still be manageable traffic. So I think it should be able to handle it correct?

Why POE? Are wifi cams less efficient or are you saying it’ll clog up the wifi network? Could I scale by adding a second wifi router at a different channel? How about 6 wifi cams?

Will checkout ipcamtalk.com

[danbutter]

The way I use deepstack (and tensorflow) in HA is to use my cameras to detect motion and then call the image processing service via an automation.
I have never tried to do detection continuously. I think that would be quite a load on the computer that deepstack is running on. Not even sure this is made for or capable of that.
Sound more like you want to check out a couple other threads on image processing:

Local realtime person detection for RTSP cameras

I wanted to integrate nest-like person detection into homeassistant with my existing IP camera so I could do things like turn on exterior/interior lights while the alarm was armed. Ended up working well, so I thought I would post here. It detects a person in <1s.

Motion AI - Detection and classification of entities

Wow! This is exhaustive. I have so many questions, but the most important ones: how many cameras can this reasonably support on a Pi4? how much latency for detection events have you noticed on a Pi4? Does it make sense to have an entire HA instance dedicated to this or should my regular installation support most of this? Do you use any kind of NVR separately from this?

Object detection for video surveillance

Watsor Watsor detects objects in video stream using deep learning-based approach. Intended primarily for surveillance it works in sheer real-time analysing the most recent frame to deliver fastest reaction against a detected threat. What it does Performs smart detection based on artificial neural networks significantly reducing false positives in video surveillance. Capable to limit detection zones using mask image with alpha channel. Supports multiple hardware accelerators such as The Coral U…

And yeah POE cams are wired and therefore better performance than wifi cams. Keeping as much wired as possible is the recommended method.

[jon_r (Jonathan)]

A big thank to @robmarkcole for your hard work on this, also the guys at Deepstack.
After some initial issues I’m now running the latest GPU Beta in Docker on Ubuntu, (i3 8Gb RAM, running in the background of my desktop machine at the minute) everything working well, alerts on face detect etc.

The thing I’m now stuck on is how to view previous events in a simple way on Lovelace, preferably some sort of table with timestamp & identity, and ideally where if you click on it you then get the snapshot displayed. Anyone seen anything suitable? Does anyone have any better ideas?

[robmarkcole (Robin)]

I am not aware of a UI like that, would be a pretty substantial software engineering task. One idea I had was to sit a proxy between HA and deepstack, and use that to capture all events and images, and have that build a timeline. However front end is not my area of expertise, an example UI is here

[luciandf (Lucian)]

@robmarkcole Hi, I just stumbled on your project and it looks amazing. I have a few questions if you don’t mind:

1. I run home assistant in a virtual machine in proxmox. how would I go about adding your module there?
2. If your module requires a separate machine, can I install it in a virtual machine in proxmox?

Thank you

[Joca]

I have my HA and Deepstack working in same NUC with proxmox in 2 separated VM.

[robmarkcole (Robin)]

I am not familiar with proxmox so cannot comment, if it is using docker you should be fine

[jon_r (Jonathan)]

Hi
I have an error in HA logs, yet everything appears to work
Every time my automation runs that calls image_processing.scan service I get an error, as per below. However, it does recognise faces and triggers notifications as requested. Any idea what’s causing the error and how to stop it?

Log Details (ERROR)

Logger: custom_components.deepstack_face.image_processing
Source: custom_components/deepstack_face/image_processing.py:212
Integration: deepstack_face (documentation)
First occurred: 16:07:55 (1 occurrences)
Last logged: 16:07:55

Depstack error : Error from request, status code: 400

Automation yaml:

- id: '1594190607624'
  alias: Face recognition
  description: ''
  trigger:
  - platform: time_pattern
    seconds: /5
  condition: []
  action:
  - data: {}
    entity_id: image_processing.face_counter
    service: image_processing.scan

Configuration yaml:

image_processing:
  - platform: deepstack_face
    ip_address: 192.168.1.100
    port: 5000
    timeout: 10
    detect_only: False
    save_file_folder: /config/snapshots/
    save_timestamped_file: True
    show_boxes: True
    source:
      - entity_id: camera.my_esp32_camera
        name: face_counter

NB slowing the rate the service is called does not affect the errors, it still errors every time (just less often, obviously!) Running the service manually causes the same error.

Anyone any idea what’s causing it?

Thanks in advance

Jonathan

[Tomer_Klein]

Just moved it to Telegram Bot

[nightwood (Simon)]

I’m banging my head on the wall with this one and I’m sure it’s something simple.

I’m trying to use save_file_folder to save the processed image for later use (in a notification ideally) however I can’t get anything written to the folder.

Deepstack is installed on docker per readme. I’m able to call manually (via curl) and also I’ve had a correct detection of a car using deepstack_object component from the camera feed, so I’m assuming the issue isn’t at that end.

Relevant config is:

homeassistant:
  whitelist_external_dirs:
    - /config/snapshots

…

image_processing:
  - platform: deepstack_object
    ip_address: localhost
    port: 5000
    save_file_folder: /config/snapshots/
    api_key: <redacted>
    # scan_interval: 60
    save_timestamped_file: True
    #roi_x_min: 0.35
    #roi_x_max: 0.8
    #roi_y_min: 0.4
    #roi_y_max: 0.8
    targets:
      - person
      - car
    source:
      - entity_id: camera.driveway_camera

I’ve tried triggering image_processing.scan from Developer Tools and also via automation

- id: '1595416418089'
  alias: Driveway Motion Detected
  description: ''
  trigger:
  - entity_id: binary_sensor.driveway_motion_190
    platform: state
    to: 'on'
  condition: []
  action:
  - data:
      entity_id: image_processing.deepstack_object_driveway_camera
    service: image_processing.scan

But no luck. Any suggestions on what I’m missing?

Edit: I see in some of the older screenshots that save_file_folder is an attribute of the entity, however I can’t see that on mine. Not sure if that’s relevant or if it’s been dropped in later versions.

[robmarkcole (Robin)]

Likely either a permissions issue or there is no image to be saved

[nigel1 (Nigel)]

Only difference I see in comparison to mine, is the final “/” in
homeassistant:
whitelist_external_dirs:
- /config/snapshots**/**

[nightwood (Simon)]

Thanks for the speedy reply @robmarkcole!

Beyond whitelisting I’m not sure what other permissions might be required. Can you point me in the right direction for what to check?

Under what conditions might there not be an image to save? I’m assuming that if deepstack is able to detect a car there’s an image that’s been analysed and is able to be saved. Am I missing a step in that process?

[MSmithHA]

When we call image_processing.scan, what happens exactly? Does it just take a single snapshot from the camera and look at that one image? Or does it do multiple? Or does it use the camera snapshot url? How does that part work?

[nightwood (Simon)]

Not sure exactly why but it’s now working.

I changed two things:

1. Updated whitelist_external_dirs to allowlist_external_dirs in line with updated documentation. I don’t see that in 0.113 release notes but the documentation seems to have changed at the same time
2. I have an object to detect, so it’s possible that files are only written when objects are detected. @robmarkcole can you confirm?

Either way, happy camper now!

[robmarkcole (Robin)]

Yes images saved only when there is a positive detection