Facial recognition & room presence using Double Take & Frigate

[Jokerigno]

Hi all,

I just installed this as my first tentative to have face recognition. I have installed frigate on my unraid server few weeks ago and it works flawlessy. It recognizes person and I setup an automation to notify and send pictures. I wanted also to add face detection to the equation so I installed deepstack and double-take in unraid. It “seems” to work but in DT I cannot see any result so far.

I thinkt that it works becausein DeepStack log I have a lot of logs like this

[GIN] 2021/05/07 - 18:54:29 | 200 | 243.860701ms | 172.17.0.1 | POST /v1/vision/face/recognize
[GIN] 2021/05/07 - 18:54:30 | 200 | 163.965155ms | 172.17.0.1 | POST /v1/vision/face/recognize
[GIN] 2021/05/07 - 18:54:35 | 200 | 694.119873ms | 172.17.0.1 | POST /v1/vision/face/recognize
[GIN] 2021/05/07 - 18:54:36 | 200 | 1.264500757s | 172.17.0.1 | POST /v1/vision/face/recognize
[GIN] 2021/05/07 - 18:54:36 | 200 | 310.042525ms | 172.17.0.1 | POST /v1/vision/face/recognize

So I think that DT is sending something to it from frigate. Bui DT webpage is empty, it just says “No files found”. Any tip?

EDIT: I have to add SAVE_UNKNOWN=true to start seeing something.

[jonnyrider (Jonathan Rider)]

I’ve started a brand new Compreface profile, and this isn’t bringing through any random celebrities now. Thanks for the tip, I’m using Deepstack and Compreface together now, seems to be working well. I’m going to leave it for a couple of days and see how the accuracy improves with some training.

Thanks

[Eoin]

Not sure where to start troubleshooting this one. I have everything up and running except the publishing of the double-take mqtt topic.

I can see double-take connecting to my mqtt server, it is detecting mqtt messages from Frigate and processing the images with deepstack, but when it detects a face nothing is being published under the topic double-take. The below is my config, any suggestions?

{
	"CONFIDENCE": 70,
	"DEEPSTACK_URL": "http://192.168.1.166:32768",
	"DETECTORS": [
		"deepstack"
	],
	"FRIGATE_IMAGE_HEIGHT": 800,
	"FRIGATE_URL": "http://192.168.1.166:5000",
	"LATEST_RETRIES": 20,
	"MQTT_HOST": "mqtt://192.168.1.166:1883",
	"MQTT_PASSWORD": "PASSWORD",
	"MQTT_TOPIC": "frigate/events",
	"MQTT_TOPIC_CAMERAS": "double-take/cameras",
	"MQTT_TOPIC_MATCHES": "double-take/matches",
	"MQTT_USERNAME": "USERNAME",
	"PORT": 3000,
	"PURGE_MATCHES": 48,
	"PURGE_UNKNOWN": 10,
	"SAVE_UNKNOWN": true,
	"SNAPSHOT_RETRIES": 20,
	"STORAGE_PATH": "./.storage",
	"TZ": "UTC",
	"VERSION": "0.3.2"
}

[Jako]

Does the detection produce a match result? Everything in your config looks ok and since it’s receiving Frigate messages it should be fine.

Could you try to hit the API manually with an image that will produce a match? This will also publish to the MQTT topic. Could you then post the results of the output.

http://localhost:3000/api/recognize?url=http://YOUR-IMAGE-URL.com/image.jpg&results=all

[Eoin]

EDIT: I moved double-take to a different box, from my NAS to the same NUC HA is running on. MQTT now working as expected.

Hi Jako,

Ok output form direct API call:

{"id":"2f79b8cd-b468-4178-af32-a02bf7cf9cc7","duration":0.65,"timestamp":"2021-05-11T06:20:08.983Z","attempts":1,"camera":"double-take","zones":[],"matches":[{"name":"eoin","confidence":73.45,"match":true,"box":{"top":34,"left":254,"width":39,"height":47},"type":"manual","duration":0.43,"detector":"deepstack","filename":"9a6d355a-7aa8-4a41-9022-ba2736f5537d.jpg"}]}

and from message log:

2021-05-11T06:20:08.334Z
processing double-take: 2f79b8cd-b468-4178-af32-a02bf7cf9cc7
manual attempt 0
response:
{
	"id": "2f79b8cd-b468-4178-af32-a02bf7cf9cc7",
	"duration": 0.65,
	"timestamp": "2021-05-11T06:20:08.983Z",
	"attempts": 1,
	"camera": "double-take",
	"zones": [],
	"matches": [
		{
			"name": "eoin",
			"confidence": 73.45,
			"match": true,
			"box": {
				"top": 34,
				"left": 254,
				"width": 39,
				"height": 47
			},
			"type": "manual",
			"duration": 0.43,
			"detector": "deepstack",
			"filename": "9a6d355a-7aa8-4a41-9022-ba2736f5537d.jpg"
		}
	]
}
2021-05-11T06:20:08.987Z
done processing double-take: 2f79b8cd-b468-4178-af32-a02bf7cf9cc7 in 0.65 sec

Still no topic being published under ‘double-take’

[Jako]

Awesome, glad you got it working!!!

[Jako]

I’ve been working on updating the API to accept configuration variables via a YAML file rather than passing them directly to the container as environment variables. Wanted to get some feedback, but here is what a more complex config would look like.

server:
  port: 3000

mqtt:
  host: localhost
  password: test
  topics:
    matches: double-take/matches
    frigate: frigate/events

detectors:
  compreface:
    url: http://localhost:8000
    key: xxx-xxx-xxx-xxx-xxx
  deepstack:
    url: http://localhost:8001

frigate:
  url: http://localhost:4000
  image:
    height: 500
  attempts:
    latest: 15
    snapshot: 0
  cameras:
    - office
  zones:
    - name: zone-name
      camera: office

confidence:
  match: 50
  unknown: 30

save:
  matches: true
  unknown: true

purge:
  matches: 48
  unknown: 12

[Eoin]

Would this be a YAML file living within the HA config directly, something like frigate.yml?

Personally, it doesn’t make a lot of difference to me, editing the config of the container or the yaml file is about the same level of effort. But overall, I like the idea of moving to yaml, it keeps things consistent with other addons.

[jonnyrider (Jonathan Rider)]

Sounds good, I would prefer a YAML file, it’s much easier to make changes on the fly.

It also paves the way for an add-on!

[Jako]

@Eoin @jonnyrider, thanks for the feedback! Just wanted to get some ideas what others thought. I think it’ll allow future customizations to be easier and a little more organized rather than just cramming everything in as ENV vars like I am now.

You’ll just need to mount your config.yml file to the container. I’ll make sure it’s all documented before I push out a beta build.

volumes:
      - ${PWD}/config.yml:/double-take/config.yml

[Jako]

I’ve deployed the configuration changes to the beta build. README updated to show the configurable options in the updated format. If you don’t want to override any of the default values, you don’t need to pass those options in the config.yml file.

[Jako]

I’ve added partial HA integration via their REST API

If you’re using the beta build you can now pass a Home Assistant config object.

home_assistant:
  url: http://192.168.1.1:8123
  token: xxx.xxx-xxx

It doesn’t look like this method officially registers the entity within HA, since it disappears via a restart, but the next time a detection is found it is recreated. So for the time being, I’m using this and dropping the Node-Red → HA flow I had before on my personal setup.

Curious if others would find this useful.

[jonnyrider (Jonathan Rider)]

Sounds good, I’ve just updated my config to use this, will let you know if I run into any issues. The automatic creation of entities is really useful!

I’ve stopped using Deepstack now as it is much slower than Compreface and less accurate. Also, I was using an automation (below) that wasn’t firing because Deepstack brought back a lower confidence percentage and so not all the conditions were met:

- alias: Doorbell face detected
  trigger:
  - platform: mqtt
    topic: double-take/cameras/doorbell
  condition:
  - '{{ trigger.payload_json[''matches''][0][''confidence''] > 80 }}'    
  - '{{ ( as_timestamp(now()) - as_timestamp(state_attr(''automation.person_face_detected'',
    ''last_triggered'')) |int(0) ) > 30 }}'  
  action:
  - service: notify.mobile
    data_template:
      message: '{{ now().strftime(''%H:%M'') }} - {{trigger.payload_json[''matches''][0][''name'']}}
        detected with confidence of {{trigger.payload_json[''matches''][0][''confidence'']}}%'
      data:
        image: /local/doubletake/matches/{{ trigger.payload_json['matches'][0]['filename']
          }}
        priority: high

I store my matches in the www folder of HA so that I can send notifications with the image. I tried this with the /media folder but I couldn’t get it to work!

hopefully someone will find the above automation useful.

[rfsurfr (Andrew Peppercorn)]

@Jako - Huge kudos for double-take! I’ve just got everything installed successfully and thought I’d share a happy user over here. Getting successfully recognized and presence detection following.

Regarding your question about preferred integration to HA, the direct config using token as opposed to linking thru Node Red certainly caught my attention. Either way though, awesome work and highly grateful for your innovations from here in Charlotte!

-AP

[Jako]

@rfsurfr thank you for the kind works, that really means a lot. Happy to hear you are finding the HA setup useful, I’ve been using that on my main home setup as well and it’s been working great. It’s allowed me to drop Node-Red for the entity creation step completely.

If you have any feature requests or suggestions feel free to drop them in here. Happy to keep making this as useful as possible for everyone.

[sender]

Hi, I want to give this a go… I have:

Home assistant (with MQTT)
frigate with coral running on separate docker container on separate machine form home assistant.

Both work “well”

Now I wanted to start another container on a separate machine with double take and I think I need another container (deepstack?) to run alongside.

I wish to run double take separate from home assistant and frigate but i can run along with other containers.

What I tried:
I have a machine with docker installed and I am trying to use docker compose to start double take container.

I installed docker-compose:

curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

I follow docker compose manual from here:

Try Docker Compose

Created this:

@thundergreen

EDIT Works with this docker-compose:

version: '3.7'

services:
  double-take:
    container_name: double-take
    image: jakowenko/double-take
    restart: unless-stopped
    environment:
      DETECTORS: deepstack
      MQTT_HOST: 192.168.1.41
      MQTT_USERNAME: xxx
      MQTT_PASSWORD: xxx
      FRIGATE_URL: http://192.168.1.41:5000
      DEEPSTACK_URL: http://192.168.1.41:5030
    ports:
      - 3333:3000

ofcourse with my credentials but not with deepstack correctly as I do no have that yet…

And when I try it it come with:

root@dockerhost ~/doubletake# ls
Dockerfile  app.py  docker-compose.yml  requirements.txt
root@dockerhost ~/doubletake# docker-compose -up
-bash: /usr/local/bin/docker-compose: Permission denied
root@dockerhost ~/doubletake#

So… you can see I am a beginner… what do I need to do

[Jako]

Hey @sender, the best place to start would be the docker compose section in the README. The configuration has changed quite a bit in the last few weeks and the example quoted in your reply won’t work anymore. Everything is handled through a mounted config.yml file now. I have an example of this in the README too.

Try running docker-compose -v and see what comes back. Seems like there’s issues with your Docker compose install.

Let me know if you’re still having issues.

[mwolter]

Wow this is a great project and has so many possibilities. I think you’re right in facial recognition being the best type of room presence. Custom actions per user, etc.

My question is, how do the others in your household feel about camera inside the house? Most people are used to motion sensors for alarms but a camera looks significantly different can feel a lot more intrusive.

Curious what you’ve done to overcome this? Do you disguise them or hide them? Do you find others in your household become accustomed?

I realize this isn’t within the scope of your project but having two cameras on each entry/exit from the room counting how many and who enters and exits would be next level for home automation and truly make a real time “smart home” possible.

[Jako]

Hey @mwolter thanks for the feedback. You bring up some great questions. My girlfriend wasn’t the biggest fan of the cameras inside the house at first, but I’ve slowly been working her over with various home automation projects. Her favorite is a simple motion sensor in the hallway that turns on the lights haha.

I have a few of my cameras placed where they are hard to see, but some are also more visible. None of them are in super private areas though, the one in my bedroom only looks at the stairs to see who is walking up them.

Ideally I’d have the cameras and other IoT devices on a different VLAN or something to lock them down a bit more or buy better cameras that only work locally. Currently I’m using Eufy and Wyze cameras inside my house. I’ve got two Nest cameras on the outside and unfortunately I can’t do anything cool with them yet. I really wish I could pipe my doorbell video feed into Frigate and Double Take.

[mwolter]

Thanks for the insight. This is definitely one of the hurdles to be navigated. My goal would be to use your project and have two cameras on each entrance to key rooms. Preferably they would be mounted high on the doorway, pointed downward to only video people entering and exiting. Then use that to keep track of room occupancy and trigger automations. Most likely node red will be integrated somehow. One hurdle to overcome will be running Ethernet to the cameras as I distain Wifi.

Thanks again and good luck with the project!