EDIT 01-27-2020: Frigate 0.8.0 has been released. I’ve updated the instructions below to reflect the latest version since there were a ton of changes. Still works great!
EDIT 12-15-2020: I just noticed that Frigate has a 0.8.0 beta release, complete with NVIDIA support. The config made some significant breaking changes. At some point I’ll write another version of this that incorporates the changes. Waayyy less CPU usage when you’re using the GPU version of ffmpeg.
Background:
I’ve written a few articles on improving person detection, typically with Blue Iris. I also recently purchased a Google Coral USB Accelerator after reading about it in a few forums: low power, excellent detection, and fast performance even compared to GPUs. Well, I kind of put the cart before the horse and ordered one before knowing what I could do with it. The first hint should be “it doesn’t directly integrate into Blue Iris”. We’ll get around that in this article.
Summary:
To be honest, there isn’t a lot of “Home Assistant” in this article since we’re only using Home Assistant add-ons. However, the Frigate add-on was designed to integrate directly into HA, so you can always run with that and improve upon both. Specifically though, we’ll be using the following:
I’m going to assume you have Node-RED and Mosquitto broker up and running, likely in HA somewhere. Frigate is a bit different. You can run it as an addon to Home Assistant. However, I have HA running in a VM, and felt like I was adding too many layers of complexity. Instead, I used a stand-alone and separate Arch Linux server running docker with docker-compose installed. The Google Coral was also plugged into this same server so the Frigate docker container can access it once we fire it up.
To summarize:
- BI-WIN: Blue Iris is running on Windows 10.
- HASSOS-VM: Home Assistant, Mosquitto, and Node-RED are running on Home Assistant OS in a VM.
- DOCKER-LINUX: Frigate is running on a separate physical server that is running Docker.
Frigate
Again, this is on the stand-alone server that the Google Coral is plugged into, i.e. DOCKER-LINUX. A full list of the Frigate configuration can be found here. You will likely need to modify this docker-compose file. This is the docker-compose.yml that I use to fire up Frigate:
version: '3.8'
services:
frigate:
image: blakeblackshear/frigate:0.8.0-amd64nvidia
runtime: nvidia
shm_size: '1g'
container_name: frigate
privileged: true
volumes:
- /dev/bus/usb:/dev/bus/usb
- /etc/localtime:/etc/localtime:ro
- /opt/frigate:/config
- /storage01/frigate/clips:/media/frigate/clips
- /storage01/frigate/recordings:/media/frigate/recordings
- type: tmpfs
target: /tmp/cache
tmpfs:
size: '1g'
ports:
- 5000:5000
- 1935:1935
restart: unless-stopped
You also need to create the Frigate configuration file before bringing up the container. This is for my particular environment with 4 cameras, but hopefully it’ll get you pointed in the right direction. One item of note: always use camera substreams for detection. It’s way less intensive on the CPU, and provides the same level of detection from the Google Coral perspective. The RTSP streams in the example are the Hikvision URLs for the substream view. It’ll likely be different for different vendors.
You also get the best of both worlds with the latest version of Frigate: it can use the substream for detection and record the full stream when something occurs. Granted, this doesn’t matter if you’re using Blue Iris for recording, but I like to have a bit of redundancy and sync the triggered full stream Frigate clips into Google Drive.
The config file is /opt/frigate/config.yml in my setup:
logger:
default: info
detectors:
coral:
type: edgetpu
device: usb
mqtt:
host: my.mqtt.server
port: 1883
topic_prefix: frigate
client_id: frigate
user: mqttuser
password: mqttpassword
clips:
max_seconds: 300
retain:
default: 14
record:
enabled: True
retain_days: 7
snapshots:
retain:
default: 14
ffmpeg:
global_args: -hide_banner -loglevel fatal
input_args: -c:v h264_cuvid
output_args:
detect: -f rawvideo -pix_fmt yuv420p
record: -f segment -segment_time 600 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
rtmp: -c copy -f flv
objects:
track:
- person
cameras:
fyc:
width: 640
height: 480
ffmpeg:
inputs:
- path: rtsp://username:[email protected]/Streaming/Channels/1
roles:
- record
- clips
- path: rtsp://username:[email protected]/Streaming/Channels/2
roles:
- detect
- rtmp
fps: 6
snapshots:
enabled: True
timestamp: False
bounding_box: True
clips:
enabled: True
fdc:
width: 704
height: 480
ffmpeg:
inputs:
- path: rtsp://username:[email protected]/Streaming/Channels/1
roles:
- record
- clips
- path: rtsp://username:[email protected]/Streaming/Channels/2
roles:
- detect
- rtmp
fps: 6
snapshots:
enabled: True
timestamp: False
bounding_box: True
clips:
enabled: True
byc:
width: 704
height: 480
ffmpeg:
inputs:
- path: rtsp://username:[email protected]/Streaming/Channels/1
roles:
- record
- clips
- path: rtsp://username:[email protected]/Streaming/Channels/2
roles:
- detect
- rtmp
fps: 6
snapshots:
enabled: True
timestamp: False
bounding_box: True
clips:
enabled: True
bdc:
width: 704
height: 480
ffmpeg:
inputs:
- path: rtsp://username:[email protected]/Streaming/Channels/1
roles:
- record
- clips
- path: rtsp://username:[email protected]/Streaming/Channels/2
roles:
- detect
- rtmp
fps: 6
snapshots:
enabled: True
timestamp: False
bounding_box: True
clips:
enabled: True
Next you’ll do a docker-compose up -d to bring up the Frigate container. Frigate should be up and running, and is 100% independent of Home Assistant.
At this point you can also check if the detection is working as expected. You should be able to hit the URL http://DOCKER-LINUX.your.lan:5000 and see the Frigate UI.
If everything has been done correctly, Frigate will now be (1) looking for people in the camera stream, (2) saving clips of those detections, and (3) sending MQTT data to the HA Mosquitto broker about the detection being triggered.
Blue Iris
This is on BI-WIN. I really, really like Blue Iris and I’ve used it for a long time. However, I don’t want to pay for their cloud-based AI service for doing people detection, especially when I can keep it all local instead. Not much to say here other than this:
- I’m assuming that you have Blue Iris already configured.
- We’ll be replacing the native BI motion detection with Google Coral.
- In my case, BI still records everything for 1-2 weeks of continuous video.
- Important: you must have Blue Iris talking to MQTT via my other guide. You could even strip that down further and just connect to the MQTT server instead of creating any triggers. BI just needs to be connected to MQTT.
Node-RED
This is on HASSOS-VM and is basically to glue that holds everything together. Detection follows this workflow:
- Frigate and Google Coral monitor all the RTSP streams for person detection.
- If a person is detection, it sends an MQTT event to the Mosquito broker.
- Node-RED is subscribed to those events on the Mosquitto broker.
- When a person is detected, it (1) sends an alert with an image capture via Pushover and (2) generates an MQTT event that causes Blue Iris to trigger for that same camera, even with BI’s native motion detection disabled.
Please note you’ll need those two NPM modules above installed in Node-RED. This flow will not work without them.
Here’s the screenshot:
And here’s the flow:
[{"id":"5e1bc9f8.5aedb8","type":"mqtt in","z":"4dc6f85c.c0d998","name":"MQTT Frigate Person","topic":"frigate/+/person","qos":"2","datatype":"auto","broker":"e620c1bc.88c94","x":180,"y":140,"wires":[["bb1c15f5.103ce8"]]},{"id":"bb1c15f5.103ce8","type":"switch","z":"4dc6f85c.c0d998","name":"ON","property":"payload","propertyType":"msg","rules":[{"t":"gt","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":350,"y":140,"wires":[["eafb0d48.6319c"]]},{"id":"9d20343d.57a028","type":"mqtt-in-dynamicsub","z":"4dc6f85c.c0d998","name":"MQTT Grab Snapshot","qos":"2","datatype":"auto","unsubscribe-after-first-msg-recv":false,"debug-subscribe":false,"broker":"e620c1bc.88c94","x":180,"y":220,"wires":[["f783d925.65af88"]]},{"id":"eafb0d48.6319c","type":"change","z":"4dc6f85c.c0d998","name":"Rewrite Payload","rules":[{"t":"set","p":"topic","pt":"msg","to":"$.topic & \"/snapshot\"","tot":"jsonata"},{"t":"set","p":"camera_name_person","pt":"global","to":"$match(topic, /frigate\\/(.*?)\\/./).groups[0].$string()","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":140,"wires":[["9d20343d.57a028","9c1ecd43.253e4"]]},{"id":"f783d925.65af88","type":"file","z":"4dc6f85c.c0d998","name":"Write JPG","filename":"/config/www/camera_snapshot/snapshot-latest-person.jpg","appendNewline":false,"createDir":false,"overwriteFile":"true","encoding":"none","x":370,"y":220,"wires":[[]]},{"id":"9c1ecd43.253e4","type":"function","z":"4dc6f85c.c0d998","name":"Create BI MQTT","func":"var camera_name = global.get('camera_name_person') || 0;\nmsg.topic = \"BlueIris/admin\"\nmsg.payload = \"camera=\" + camera_name + \"&trigger=1\"\nmsg.qos = \"0\"\nmsg.retain = \"false\"\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":700,"y":140,"wires":[["de0e68d.902c498"]]},{"id":"de0e68d.902c498","type":"mqtt out","z":"4dc6f85c.c0d998","name":"Trigger BI","topic":"","qos":"","retain":"","broker":"e620c1bc.88c94","x":860,"y":140,"wires":[]},{"id":"e620c1bc.88c94","type":"mqtt-broker","name":"HASS MQTT","broker":"hass.naternet.local","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
Quick explanation on each of these.
MQTT Frigate Person: This subscribes to the wildcard MQTT topic frigate/+/person to monitor for person events on all of your cameras. It’s a quick ON or OFF.
ON: It looks for anything >0 which means a person was detected.
Rewrite Payload: Any of the Node-RED MQTT ingest objects expect the msg to contain the MQTT topic and payload. This modifies the topic to add /snapshot on the end. This also adds the actual camera name to the flow value camera_name_person so we can use it in a few other places.
MQTT Grab Snapshot: This is the Node-Red add-on NPM that I’d mentioned before. It simply grabs the person detection snapshot from the Mosquitto broker.
Write JPG: It does just that and write the MQTT JPG to /config/www/camera_snapshot/snapshot-latest-person.jpg on Home Assistant.
Now, moving to the top line of the flow:
Create BI MQTT: this structures an MQTT event that causes Blue Iris to trigger that particular camera for alerting. Take a look at the function for more details. It’s pretty clear.
Trigger BI: This sends out the MQTT event that Blue Iris will automatically trigger the camera in Blue Iris. You do not need to have motion detection enabled in Blue Iris.
All Done!
Sample video recording that was triggered in Blue Iris by the same detection:
Conclusion
Hope this helps! I’ve fully switched over to this method and went from hundreds of alerts per day to just a small handfull of high-fidelity alerts. From what I’ve seen, Google Coral’s object detection has been phenomenal.
