Facial recognition & room presence using Double Take & Frigate

Share your Projects!
1

Double Take Double Take Docker Pulls Discord

Double Take

Unified UI and API for processing and training images for facial recognition.

3683×1515 638 KB

Why?

There’s a lot of great open source software to perform facial recognition, but each of them behave differently. Double Take was created to abstract the complexities of the detection services and combine them into an easy to use UI and API.

Features

  • Responsive UI and API bundled into single Docker image
  • Ability to password protect UI and API
  • Support for multiple detectors
  • Train and untrain images for subjects
  • Process images from NVRs
  • Publish results to MQTT topics
  • REST API can be invoked by other applications
  • Disable detection based on a schedule
  • Home Assistant Add-on

Supported Architecture

  • amd64
  • arm64
  • arm/v7

Supported Detectors

Supported NVRs

Use Cases

Frigate

Subscribe to Frigate’s MQTT topics and process images for analysis.

mqtt:
  host: 192.168.1.1

frigate:
  url: http://192.168.1.1:5000

When the frigate/events topic is updated the API begins to process the snapshot.jpg and latest.jpg images from Frigate’s API. These images are passed from the API to the configured detector(s) until a match is found that meets the configured requirements. To improve the chances of finding a match, the processing of the images will repeat until the amount of retries is exhausted or a match is found.

When the frigate/+/person/snapshot topic is updated the API will process that image with the configured detector(s). It is recommended to increase the MQTT snapshot size in the Frigate camera config.

cameras:
  front-door:
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      height: 500

If a match is found the image is saved to /.storage/matches/${filename}.

Home Assistant

Trigger automations / notifications when images are processed.

If the MQTT integration is configured within Home Assistant, then sensors can be created from the topics that Double Take publishes to.

sensor:
  - platform: mqtt
    name: David
    icon: mdi:account
    state_topic: 'double-take/matches/david'
    json_attributes_topic: 'double-take/matches/david'
    value_template: '{{ value_json.camera }}'
    availability_topic: 'double-take/available'

857×464 86.3 KB

Github

Docker Hub

https://hub.docker.com/r/jakowenko/double-take

I’ve been trying to come up with a room presence solution for the past few months and recently created a project that’s working very well for me.

Prior to my solution I’ve tried using beacons, BLE, and a few other options. These methods did not produce the results I was looking for or required the user to have their phone or some other device on them. In a perfect world, the user wouldn’t have wear or do anything, right? Well what about facial recognition?

I recently started using Frigate which allowed me to detect when people were in a room, but what if I had friends or family over? I needed a way to distinguish each person from the images Frigate was processing. This led me to looking at Facebox, CompreFace, and DeepStack. All of these projects provide RESTful APIs for training and recognizing faces from images, but there was no easy way to send the information directly form Frigate to the detector’s API.

I tried using Node-Red and built a pretty complicated flow with retry logic, but it quickly became painful to manage and fine-tune. Being a developer I decided to move my Node-Red logic over to it’s own API, which I then containerized and named Double Take.

Double Take is a proxy between Frigate and any of the facial detection projects listed above. When the container starts it subscribes to Frigate’s MQTT events topic and looks for events that contain a person. When a Frigate event is received the API begins to process the snapshot.jpg and latest.jpg images from Frigate’s API. These images are passed from the API to the detector(s) specified until a match is found above the defined confidence level. To improve the chances of finding a match, the processing of the images will repeat until the amount of retries is exhausted or a match is found. When a match is found a new MQTT topic is published with the results. This then allowed me to have a two node flow in Node-Red for taking the results and pushing them to a Home Assistant entity.

Double Take can also use multiple detectors at the same time to try to improve the results. From my testing at home I’ve found CompreFace and DeepStack to produce the best results, but I’ve also added support for Facebox. If you don’t use Frigate, then you can still utilize the Double Take API and pass any image to it for facial recognition processing.

I would love feedback on Double Take if anyone tries it or hear about any feature requests! I’ve been using this method for a few weeks now with excellent results.

63 Likes
2

Hello,
Thank you for sharing your amazing work.
With room-presence being such a hot requirement, I wonder why your post hasn’t received any traction.

Can this setup function on a Rpi4, 4GB?

Cheers!

1 Like
3

i am fascinated too! :slight_smile: Will check this :slight_smile:

4

Thank you for the kind words! I’m not sure why this post hasn’t gained much traction, but I still was excited to share what I came up with hoping someone else would find it useful.

My Double Take container is a pretty lightweight RESTful API at it’s core and should run fine on a RPi4. It’s just a proxy between the facial detectors that are supported.

The facial detection services though do require more horsepower. I would checkout the docs for DeepStack, CompreFace, and Facebox and see which one suits your needs.

If you need any help getting it setup, let me know. I’m happy to help!

2 Likes
5

Thank you! Let me know if you have any problems getting it setup.

1 Like
6

Well i just setup deepstack face recognition. now I am stuck :stuck_out_tongue:

7

Do you use Frigate? If not, what does your full setup look like and what are you hoping to achieve?

Double Take has a /recognize endpoint where you can pass any image URL for facial detection processing. This is most likely what you’ll want to use if you don’t use Frigate.

8

I am running frigate yes… what if deepstack is not running on standard port 5000 but 5030 ? because in docker run command there’s only the frigate url. no ports to configure.

I run frigate addon

9

I include all the available options at the bottom of the README. You can pass any URL for Frigate and DeepStack. Just make sure to include the ports when you set them as environment variables.

10

Sorry but I don’t get it that well. How would I configure the env? Would you mind posting an example here with custom ports for frigate and deepstack?

11

I would suggest using a docker-compose file, but the same would apply with just a docker run command. In my example below replace http://123.123.123.123:PORT with your info and then update MQTT_HOST to the IP you’re running it on.

version: '3.7'

services:
  double-take:
    container_name: double-take
    image: jakowenko/double-take
    restart: unless-stopped
    environment:
      DETECTORS: deepstack
      MQTT_HOST: 123.123.123.123
      FRIGATE_URL: http://123.123.123.123:PORT
      DEEPSTACK_URL: http://123.123.123.123:PORT
    ports:
      - 3000:3000
12

awesome i have it installed
and it worked
Can you share with your configuration?
have a lucky day

13

Are we agree that frigate port is 5000 by default? My docker-compose file is:

version: '3.7'

services:
  double-take:
    container_name: double-take
    image: jakowenko/double-take
    restart: unless-stopped
    environment:
      DETECTORS: deepstack
      MQTT_HOST: 192.168.1.41
      MQTT_USERNAME: xxx
      MQTT_PASSWORD: xxx
      FRIGATE_URL: http://192.168.1.41
      DEEPSTACK_URL: http://192.168.1.41:5030
    ports:
      - 3000:3000

but get

frigate status error: Request failed with status code 404


frigate is not responding

any idea here?

EDIT Works with this docker-compose:

version: '3.7'

services:
  double-take:
    container_name: double-take
    image: jakowenko/double-take
    restart: unless-stopped
    environment:
      DETECTORS: deepstack
      MQTT_HOST: 192.168.1.41
      MQTT_USERNAME: xxx
      MQTT_PASSWORD: xxx
      FRIGATE_URL: http://192.168.1.41:5000
      DEEPSTACK_URL: http://192.168.1.41:5030
    ports:
      - 3333:3000

seems like if you define one port un deepstack URL u also have to add the port to all other URLs :slight_smile:
Now it is running. BUT the biggest part is missing: How can we use it now in Home assistant :slight_smile: ?

I created a MQTT-in and home assistant entity which is created but can you please show us how to configure this mqtt and home assistant entity?

1 Like
14

Testing right now as I already wanted to build something similar to add face recognition to my frigate setup :slight_smile:

Just one suggestion from my side is to be able to limit detection per camera (based on frigate events messages?).

Great work @Jako

15

I use Node-Red to handle the Home Assistant entity, but I’m pretty sure you can use Home Assistant to subscribe to the topic directly, right? I can provide a better sample later tonight with how I’m doing it in Node-Red.

Matches should be published to this topic double-take/matches/:name

16

Thank you!

What was you use case behind the limit per camera? For specific cameras you wouldn’t want to do the facial recognition on?

17

thanks … will wait your nodered flow then :slight_smile:

18

Exactly, to exclude some cameras.

Some have too low resolution/quality to be able to get reliable info for face recognition. Similar with some cameras that are capturing faces from (very) far, thus faces are small.

For person detection with frigate this works OK, I would just like to save some cpu cycles by not doing face recognition on mentioned cameras :slight_smile:

19

Totally makes sense. I’ll work to add an option to exclude cameras and keep you updated.

20

Would you mind posting your nodes for nodered?