Local realtime person detection for RTSP cameras

polarbearshirt · September 20, 2019, 5:21am

Thank you freshcoast. You were right. I needed to update the MQTT address in my frigate config file.

polarbearshirt · September 20, 2019, 6:05am

Thank you @blakeblackshear for putting this together. Here is my best shot at a noob’s guide to using a Raspberry Pi 4 to do person detection. I’m still working on home assistant integration but this doesn’t seem very difficult. (Edit: It was harder than I expected although now that I have it setup, it’s pretty awesome.)

Step 1: Flash Hypriot image onto SD card using balenaEtcher
Step 2: Update Password:

passwd

Step 3: root privilege

sudo -i

Step 4: update your install

apt-get update && apt-get upgrade -y

reboot

Step 6: Install extra software for hassio

apt-get install -y jq curl avahi-daemon dbus apparmor-utils network-manager apt-transport-https ca-certificates socat software-properties-common gnupg-agent

reboot

Step 7: install hass.io for RPi4

curl -sL https://raw.githubusercontent.com/home-assistant/hassio-installer/master/hassio_install.sh | bash -s -- -m raspberrypi4

Step 8: check Hassio install at 192.168.xx.xx:8123
Step 9: Setup Home Assistant
Step 10: Install Mosquitto from add ons and add username and password
Step 11: Install Configurator from add ons
Step 12: Download frigate files to /home/pirate

git clone https://github.com/blakeblackshear/frigate.git

Step 13: Update config file
Step 14: Comment out intel hardware acceleration lines in DockerFile so it looks like this

 libgcc1 
 # VAAPI drivers for Intel hardware accel
# libva-drm2 libva2 i965-va-driver vainfo \
# && rm -rf /var/lib/apt/lists/*

Step 15: Build Frigate container

docker build -t frigate .

Step 16: Create docker-compose.yml file and save it in main frigate directory. Like so:

nano docker-compose.yml

copy and paste the following.

version: "3"
services:

  frigate:
    container_name: frigate
    restart: unless-stopped
    privileged: true
    image: frigate:latest
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /home/pirate/frigate/config:/config
    ports:
      - "5000:5000"
    environment:
      RTSP_PASSWORD: "password"

Then control+o, enter, control+x
Step 17: Run frigate container

docker-compose up

Step 18: Integrate with Home Assistant by copying the camera and binary_sensor code from @blakeblackshear’s readme example to the Home Assistant config file. As of this writing it appears the automation code is intended to live inside the automations.yaml file which is where it was generated when I created my first automation via the Configuration->Automation tab.
For those coming from the ipcamtalk world like me, here are a few tips on Home Assistant:

The configuration.yaml file that is created by default on the most recent version of HA won’t look the same as most of the examples you’ll find. It doesn’t have or need the extra lines of code to work properly. You’ll end up needing to edit the file a lot less than most of the examples and videos you’ll find.
If you want to add multiple cameras you’ll just copy the “camera” and “binary_sensor” sections and add a space and an identifier like so

camera 2:
  - name: Camera Last Person 2
    platform: mqtt
    topic: frigate/<camera2_name>/snapshot

binary_sensor 2:
  - name: Camera2 Person
    platform: mqtt
    state_topic: "frigate/<camera2_name>/objects"
    value_template: '{{ value_json.person }}'
    device_class: motion
    availability_topic: "frigate/available"

For Android users you won’t need a “native” app from the Play store. It’s possible to setup your configuration with duckdns.org via a Hass.io add-on so that you can view your HA install remotely. Once setup you can “add to home screen” the https page you have created and it’ll function just like a normal app would. The lovelace UI is very good from what I’ve seen and is working better than my attempt at setting up “HA Client”. For notifications I’m using html5 although, telegram and pushbullet are apparently good options as well.

polarbearshirt · September 20, 2019, 10:14pm

One thing I’d really like to do is have the ability to create an automation that is triggered by an individual bounding box. For instance if you have two bounding boxes within a single camera view, one covering the front door and another box covering the driveway, it would be nice to be able to trigger off of the front door box (deliveries during the day) separately from the driveway box (prowler at night). I might be able to figure this out but if someone knows how already please let me know.

The other thing I’m trying to figure out is how to create an automation that is triggered only if a person is detected in a box on one camera shortly before a person is detected in a box on another camera. For instance if camera1 covers the outside of a fence and camera2 covers the inside of the fence, trigger an action if a person is detected on camera1 shortly before a person is detected on camera2 (someone hops over the fence). Any ideas?

internetix · October 1, 2019, 7:58pm

Thank for that, this has been very useful. I managed to get the object detection working

And thanks to @blakeblackshear for developing and sharing such a great piece of software!

Darbos · October 6, 2019, 2:25pm

Hey guys, I set this up few weeks back and it ran solid for the whole time. Well I rebooted everything and completely forgot how I did this lol. Literally been trying for days to get it back. I finally have it so my pictures are getting to ha and alerts working again. But strange part is I can’t get to the url to setup the regions. I don’t see it mentioned in the docs it changed but at the point I gotta ask… Thx

Darbos · October 6, 2019, 2:27pm

I realize the pictures are coming over mqtt now too, but i thought the url woud still work to adjust the regions? im using http://ipaddress:5000/camname

polarbearshirt · October 6, 2019, 3:41pm

The regions are adjusted in your frigate config file which is usually located in the /home/“username”/frigate/config folder.

Darbos · October 6, 2019, 10:15pm

Thanks. I am looking for the url that shows the video with the region boxes, so I can make sure my regions are where they need to be.

Darbos · October 6, 2019, 10:49pm

My fault. just noticed i forgot to expose the port in my compose file. added the port and back in buisness. thanks

Mark-Mills · October 11, 2019, 2:46pm

Hello,
Thank you @blakeblackshear for a really interesting project and a useful addition to my home automation.
I followed @polarbearshirt instructions and got things up and running… of sorts.
The image at port 5000 is very jerky and delayed by about 30 seconds. Cars going past can jump from one side of the screen to the other and people are missed as they ‘jump’ over the detection areas.
I’m using a Raspberry Pi 4 and a new security camera, I just bought for this project, that is outputting H265 video at 1920x1080. I tried changing the take_frame to 5 but this didn’t make an improvement.
Any ideas where I could be going wrong? Maybe H265 is not supported.
Regards
Mark

jon102034050 · October 11, 2019, 3:12pm

what FPS are you running the camera at? Honestly, I havent tried H265 because I had issues capturing clips from my cameras then sending them over Telegram.

Mark-Mills · October 11, 2019, 3:32pm

Thanks for the reply. To be honest I haven’t been able to work out what the FPS are from the camera. I looked in VLC but this is not one of the fields shown in ‘Media Information’. But, it does look very smooth to me so I assume quite high. Which is why I changed the ‘take_frame’ value.

jon102034050 · October 11, 2019, 3:49pm

Curious, is the camera and pi4 on wifi or wired ethernet? I’d try, just for kicks, bumping down to h264 if you can.

Mark-Mills · October 11, 2019, 4:25pm

It’s all on wired. I can’t see how to change the output to H264 (the camera is a Jennov A73WG20) but your suggestion gave me an idea of using some of the other output streams. Stream 2 is still H265 apparently , but the resolution is 640x480 which seems to not suffer the same stuttering. I’ll leave it running to see if it identifies people as before.

Mark-Mills · October 11, 2019, 4:27pm

and the lag is now minimal too. A big step forward, thank you!

Darbos · October 13, 2019, 2:01am

Hey I was looking through the GitHub and noticed someone mentioned you can put a different model to be able to detect cars. Is there some steps to what I would need to do to make that happen? Thank you!

scstraus · October 13, 2019, 12:14pm

Models trained on the ImageNet database could potentially recognize more categories of cars like trucks, as they are differentiated in the DB… Best would probably be something like the MobileNet V2 with Imagenet from here:

Though you should be able to recognize a generic “car” category with the included model. You will just have to set the configuration to send recognition of the additional category in the Frigate configuration.

Darbos · October 13, 2019, 6:24pm

Thanks man! That last sentence sounds easy lol. Where do you make the change from person to car?

scstraus · October 13, 2019, 6:45pm

I haven’t done it myself, but I recall seeing other people doing it either in this thread or in the github chat. IIRC, it was done in the parameters that can be sent to the detection engine in recent versions. You can find the details if you look for them.

Darbos · October 13, 2019, 7:02pm

Ok thanks for the direction. I’ve been through the thread and gitHub. I don’t have chat or anything but I’ll see if I can find something similar n copy. I’ll post back if I do. Thanks again.