Person Detection with Blue Iris, Google Coral, and Frigate v0.8.0

EDIT 01-27-2020: Frigate 0.8.0 has been released. I’ve updated the instructions below to reflect the latest version since there were a ton of changes. Still works great!

EDIT 12-15-2020: I just noticed that Frigate has a 0.8.0 beta release, complete with NVIDIA support. The config made some significant breaking changes. At some point I’ll write another version of this that incorporates the changes. Waayyy less CPU usage when you’re using the GPU version of ffmpeg.

Background:

I’ve written a few articles on improving person detection, typically with Blue Iris. I also recently purchased a Google Coral USB Accelerator after reading about it in a few forums: low power, excellent detection, and fast performance even compared to GPUs. Well, I kind of put the cart before the horse and ordered one before knowing what I could do with it. The first hint should be “it doesn’t directly integrate into Blue Iris”. We’ll get around that in this article.

Summary:

To be honest, there isn’t a lot of “Home Assistant” in this article since we’re only using Home Assistant add-ons. However, the Frigate add-on was designed to integrate directly into HA, so you can always run with that and improve upon both. Specifically though, we’ll be using the following:

I’m going to assume you have Node-RED and Mosquitto broker up and running, likely in HA somewhere. Frigate is a bit different. You can run it as an addon to Home Assistant. However, I have HA running in a VM, and felt like I was adding too many layers of complexity. Instead, I used a stand-alone and separate Arch Linux server running docker with docker-compose installed. The Google Coral was also plugged into this same server so the Frigate docker container can access it once we fire it up.

To summarize:

  • BI-WIN: Blue Iris is running on Windows 10.
  • HASSOS-VM: Home Assistant, Mosquitto, and Node-RED are running on Home Assistant OS in a VM.
  • DOCKER-LINUX: Frigate is running on a separate physical server that is running Docker.

Frigate

Again, this is on the stand-alone server that the Google Coral is plugged into, i.e. DOCKER-LINUX. A full list of the Frigate configuration can be found here. You will likely need to modify this docker-compose file. This is the docker-compose.yml that I use to fire up Frigate:

version: '3.8'

services:
  frigate:
    image: blakeblackshear/frigate:0.8.0-amd64nvidia
    runtime: nvidia
    shm_size: '1g'
    container_name: frigate
    privileged: true
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate:/config
      - /storage01/frigate/clips:/media/frigate/clips
      - /storage01/frigate/recordings:/media/frigate/recordings
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: '1g'
    ports:
      - 5000:5000
      - 1935:1935
    restart: unless-stopped

You also need to create the Frigate configuration file before bringing up the container. This is for my particular environment with 4 cameras, but hopefully it’ll get you pointed in the right direction. One item of note: always use camera substreams for detection. It’s way less intensive on the CPU, and provides the same level of detection from the Google Coral perspective. The RTSP streams in the example are the Hikvision URLs for the substream view. It’ll likely be different for different vendors.

You also get the best of both worlds with the latest version of Frigate: it can use the substream for detection and record the full stream when something occurs. Granted, this doesn’t matter if you’re using Blue Iris for recording, but I like to have a bit of redundancy and sync the triggered full stream Frigate clips into Google Drive.

The config file is /opt/frigate/config.yml in my setup:

logger:
  default: info

detectors:
  coral:
    type: edgetpu
    device: usb

mqtt:
  host: my.mqtt.server
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: mqttuser
  password: mqttpassword

clips:
  max_seconds: 300
  retain:
    default: 14

record:
  enabled: True
  retain_days: 7

snapshots:
  retain:
    default: 14

ffmpeg:
  global_args: -hide_banner -loglevel fatal
  input_args: -c:v h264_cuvid
  output_args:
    detect: -f rawvideo -pix_fmt yuv420p
    record: -f segment -segment_time 600 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    rtmp: -c copy -f flv

objects:
  track:
    - person

cameras:
  fyc:
    width: 640
    height: 480
    ffmpeg:
      inputs: 
        - path: rtsp://username:[email protected]/Streaming/Channels/1
          roles:
            - record
            - clips
        - path: rtsp://username:[email protected]/Streaming/Channels/2
          roles:
            - detect
            - rtmp
    fps: 6
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True
  fdc:
    width: 704
    height: 480
    ffmpeg:
      inputs:
        - path: rtsp://username:[email protected]/Streaming/Channels/1
          roles:
            - record
            - clips
        - path: rtsp://username:[email protected]/Streaming/Channels/2
          roles:
            - detect
            - rtmp
    fps: 6
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True
  byc:
    width: 704
    height: 480
    ffmpeg:
      inputs:
        - path: rtsp://username:[email protected]/Streaming/Channels/1
          roles:
            - record
            - clips
        - path: rtsp://username:[email protected]/Streaming/Channels/2
          roles:
            - detect
            - rtmp
    fps: 6
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True
  bdc:
    width: 704
    height: 480
    ffmpeg:
      inputs:
        - path: rtsp://username:[email protected]/Streaming/Channels/1
          roles:
            - record
            - clips
        - path: rtsp://username:[email protected]/Streaming/Channels/2
          roles:
            - detect
            - rtmp
    fps: 6
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True

Next you’ll do a docker-compose up -d to bring up the Frigate container. Frigate should be up and running, and is 100% independent of Home Assistant.

At this point you can also check if the detection is working as expected. You should be able to hit the URL http://DOCKER-LINUX.your.lan:5000 and see the Frigate UI.

1

If everything has been done correctly, Frigate will now be (1) looking for people in the camera stream, (2) saving clips of those detections, and (3) sending MQTT data to the HA Mosquitto broker about the detection being triggered.

Blue Iris

This is on BI-WIN. I really, really like Blue Iris and I’ve used it for a long time. However, I don’t want to pay for their cloud-based AI service for doing people detection, especially when I can keep it all local instead. Not much to say here other than this:

  • I’m assuming that you have Blue Iris already configured.
  • We’ll be replacing the native BI motion detection with Google Coral.
  • In my case, BI still records everything for 1-2 weeks of continuous video.
  • Important: you must have Blue Iris talking to MQTT via my other guide. You could even strip that down further and just connect to the MQTT server instead of creating any triggers. BI just needs to be connected to MQTT.

Node-RED

This is on HASSOS-VM and is basically to glue that holds everything together. Detection follows this workflow:

  1. Frigate and Google Coral monitor all the RTSP streams for person detection.
  2. If a person is detection, it sends an MQTT event to the Mosquito broker.
  3. Node-RED is subscribed to those events on the Mosquitto broker.
  4. When a person is detected, it (1) sends an alert with an image capture via Pushover and (2) generates an MQTT event that causes Blue Iris to trigger for that same camera, even with BI’s native motion detection disabled.

Please note you’ll need those two NPM modules above installed in Node-RED. This flow will not work without them.

Here’s the screenshot:

And here’s the flow:

[{"id":"5e1bc9f8.5aedb8","type":"mqtt in","z":"4dc6f85c.c0d998","name":"MQTT Frigate Person","topic":"frigate/+/person","qos":"2","datatype":"auto","broker":"e620c1bc.88c94","x":180,"y":140,"wires":[["bb1c15f5.103ce8"]]},{"id":"bb1c15f5.103ce8","type":"switch","z":"4dc6f85c.c0d998","name":"ON","property":"payload","propertyType":"msg","rules":[{"t":"gt","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":350,"y":140,"wires":[["eafb0d48.6319c"]]},{"id":"9d20343d.57a028","type":"mqtt-in-dynamicsub","z":"4dc6f85c.c0d998","name":"MQTT Grab Snapshot","qos":"2","datatype":"auto","unsubscribe-after-first-msg-recv":false,"debug-subscribe":false,"broker":"e620c1bc.88c94","x":180,"y":220,"wires":[["f783d925.65af88"]]},{"id":"eafb0d48.6319c","type":"change","z":"4dc6f85c.c0d998","name":"Rewrite Payload","rules":[{"t":"set","p":"topic","pt":"msg","to":"$.topic & \"/snapshot\"","tot":"jsonata"},{"t":"set","p":"camera_name_person","pt":"global","to":"$match(topic, /frigate\\/(.*?)\\/./).groups[0].$string()","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":140,"wires":[["9d20343d.57a028","9c1ecd43.253e4"]]},{"id":"f783d925.65af88","type":"file","z":"4dc6f85c.c0d998","name":"Write JPG","filename":"/config/www/camera_snapshot/snapshot-latest-person.jpg","appendNewline":false,"createDir":false,"overwriteFile":"true","encoding":"none","x":370,"y":220,"wires":[[]]},{"id":"9c1ecd43.253e4","type":"function","z":"4dc6f85c.c0d998","name":"Create BI MQTT","func":"var camera_name = global.get('camera_name_person') || 0;\nmsg.topic = \"BlueIris/admin\"\nmsg.payload = \"camera=\" + camera_name + \"&trigger=1\"\nmsg.qos = \"0\"\nmsg.retain = \"false\"\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":700,"y":140,"wires":[["de0e68d.902c498"]]},{"id":"de0e68d.902c498","type":"mqtt out","z":"4dc6f85c.c0d998","name":"Trigger BI","topic":"","qos":"","retain":"","broker":"e620c1bc.88c94","x":860,"y":140,"wires":[]},{"id":"e620c1bc.88c94","type":"mqtt-broker","name":"HASS MQTT","broker":"hass.naternet.local","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

Quick explanation on each of these.

MQTT Frigate Person: This subscribes to the wildcard MQTT topic frigate/+/person to monitor for person events on all of your cameras. It’s a quick ON or OFF.

ON: It looks for anything >0 which means a person was detected.

Rewrite Payload: Any of the Node-RED MQTT ingest objects expect the msg to contain the MQTT topic and payload. This modifies the topic to add /snapshot on the end. This also adds the actual camera name to the flow value camera_name_person so we can use it in a few other places.

MQTT Grab Snapshot: This is the Node-Red add-on NPM that I’d mentioned before. It simply grabs the person detection snapshot from the Mosquitto broker.

Write JPG: It does just that and write the MQTT JPG to /config/www/camera_snapshot/snapshot-latest-person.jpg on Home Assistant.

Now, moving to the top line of the flow:

Create BI MQTT: this structures an MQTT event that causes Blue Iris to trigger that particular camera for alerting. Take a look at the function for more details. It’s pretty clear.

Trigger BI: This sends out the MQTT event that Blue Iris will automatically trigger the camera in Blue Iris. You do not need to have motion detection enabled in Blue Iris.

All Done!

Sample video recording that was triggered in Blue Iris by the same detection:

Conclusion

Hope this helps! I’ve fully switched over to this method and went from hundreds of alerts per day to just a small handfull of high-fidelity alerts. From what I’ve seen, Google Coral’s object detection has been phenomenal.

18 Likes

Amazing. This worked great. BI has RTSP streams built in so you don’t need to integrate with HA at all. If you go the BI -> Frigate direct route the nobuffer option needs to be disabled. I would also remove the threshold values as the latest version adjusted that so 0.85 is way too high to trigger anything

filters:
    person:
      min_score: 0.5
      threshold: 0.85`

see here https://github.com/blakeblackshear/frigate/releases/tag/v0.8.0-beta2

2 Likes

Thanks. I’m actually running beta3 with the NVIDIA build in docker and it’s working great, although I had to redo most of my config. At some point I’ll need to update the guide. Either way it blows BI out of the water for object detection.

Are you able to use h265 for cameras in BI? I have tried, but UI3 then no longer displays clips.

Yep, one of my Hikvision cameras can do both h264 and h265+ at various resolutions. I’ve never had a problem with them in BI.

recording works, but trying to view clips using webserver UI3 doesn’t work.
EDIT: solved. Had to restart BI.

what hardware are you using for blue iris + frigate? I’ve been putting deep consideration into this for a while now, but I’d want to enable person detection for at least 6 cameras and have concerns about the load. I also have a couple cameras that I cannot configure a substream, so I’ll be dealing with 25fps 1080 streams. I’m thinking it might be worth transcoding live to reduce load on Frigate, but it might be a wash.

I’ve got 5 cameras but they’re all using substreams. BI is running on my Windows desktop which has an AMD 3900X, but it’s way overkill just for that. The Frigate docker container is running on an i7 but uses the NVIDIA build, which runs against a Quadro P620. Each stream is < 2% CPU utilization for the substreams which is pretty damn amazing. It records a full 1920x1080 stream too.

I’m going to rewrite this for the 0.8 release once it’s out. Lots of breaking changes to the configuration, but the betas and RCs have been running flawlessly.

@TaperCrimp can you show how you trigger BI thru MQTT? What is the typical command?
EDIT: found it.
Send for exemple payload: camera=x&trigger=1
to /admin topic.

I am currently using rstp server in docker to gather all the streams from cameras.
then BI & frigate connect to this.
I would like ti disable motion detection in BI and rely only on frigate.
Then listening on MQTT events from frigate (already setup), trigger Blueiris recording.

my last implementation was based on BI and deepstack.ai
but i switched to frigate because of the great integration into home assistant.

Frigate has an excelled add-on and an integration (via) HACS.
the docker container will run in your home assistant platform. no additional (external) docker implementation is needed.

because of that big value i don’t use BlueIris anymore.
My hikvision/annke c800 cameras are having rtsp streams. So you i can stream it directly to frigate and not over BI.

i am using frigate with one camera only and my cpu utilization on my odroid n2+ is still below 5%,.

the only reason for me running frigate outside of my hassio instance is, if i would have to many cameras or high resolution streams which would impact my hassio performance.

the frigate integration will add all sensors and entities which are needed for automations etc.

below are my entities generated by the frigate integration…

I actually use both and prefer Frigate in HA for viewing clips. However, I keep Blue Iris in play since it’s better for reviewing lots of video at once, e.g. scanning through 4 hours of footage across 5 cameras. TL;DR is that I use Frigate for clips and Blue Iris for continuous footage.

anyone successfully mapped the Coral Edge TPU M2 to docker?

1 Like

you are right and thanks for your guide.
after playing around with my config. now my stream is going from the cameras -> BI -> Frigate.
This because the new Annke c800 (hikvision) has only h.265.

Nice! Any way to train Frigate? I am looking for something to train image detection for ups, FedEx…

Do you have an example on how to integrate push notification to your flow like you did with the Amazon ai?

This is what I have presently so how would I remove amazon and get Frigate to push notification:

Your flow has no pushover nodes, so how do you use the pushover?

I did this but not sure if it works:

This is the code sent to my pushover alert:

var camera_name = global.get('camera_name_person') || 0;
msg.topic = "BlueIris/admin"
msg.payload = "camera=" + camera_name + "&trigger=1"
msg.device = "GalaxyS10Plus";
msg.priority = 1;
msg.image = "/config/www/camera_snapshot/snapshot-latest-person.jpg";
msg.qos = "0"
msg.retain = "false"
return msg;

How did you get the ffmpeg to work in docker-compose?

That’s probably a better question for the Frigate thread.

I’d have to dig through since I change it a while ago, but this is the function node that I feed into a Pushover alert:

var camera_name = global.get('camera_name_person') || 0;
msg.payload = "camera: " + camera_name;
msg.topic = "Person Detected";
msg.device = "MyDevice";
msg.priority = 1;
msg.image = "/config/www/camera_snapshot/snapshot-latest-person.jpg"
return msg;

Looks like you’re pretty close though.

1 Like

This worked for me with one small change.

msg.image → msg.attachment

It looks like a lot of you are running Frigate container on a Linux box. Is it possible to install docker desktop on a Windows 10 box and run the frigate container?

Sorry if I’m way off base here. I have Blue Iris running on a windows box already, and it would be really convenient to just use that PC to run Frigate.

1 Like