So my config didn’t have a global object at all. Only at the camera level. So I thought cars where not being tracked. I put the global object in without the car/truck settings and the CPU is staying at 8% with a walk test.
As a test. Before that change. I also moved my camera up a bit where it was seeing the road. Doing that my CPU went from 8% is 35% in about 30 minutes. So I guess it tracks every car and the CPU will get out of control fast. I did put a mask on the road. But still was catching them. Maybe needed a bigger mask. Not sure.
Would like to have cars detected. But will wait till stationary object detection is a thing.
BTW, I moved my setup to a new machine (A Lenovo ThinkCentre M700 i5) running bare metal Ubuntu 20.04, and I got this in the logs as well as frequent “Restarting detection process” messages which I didn’t have on the old i3 laptop:
F :1133] HandleQueuedBulkIn transfer in failed. Not found: USB transfer error 5 [LibUsbDataInCallback]
Googling around it seems as if it means the Coral is not getting enough power - I had to move USB ports to one that shows a little battery pic (assuming it means you can charge phones from it). Works now!
I also had trouble with the Coral not being picked up (“No EdgeTPU device detected”) – after running the getting started instructions it worked (steps 1 - 3).
Edit: I can also say that USB3 does make a massive difference over USB2:
I’m trying out the “save_clips” image, and I added the new config entries to my camera config to save the clips.
Config for one camera:
cameras:
front:
ffmpeg:
################
# Source passed to ffmpeg after the -i parameter. Supports anything compatible with OpenCV and FFmpeg.
# Environment variables that begin with 'FRIGATE_' may be referenced in {}
################
input: rtsp://<redacted>/Streaming/Channels/3
#################
# These values will override default values for just this camera
#################
# global_args: []
# hwaccel_args: []
# input_args: []
# output_args: []
################
# The expected framerate for the camera. Frigate will try and ensure it maintains this framerate
# by dropping frames as necessary. Setting this lower than the actual framerate will allow frigate
# to process every frame at the expense of realtime processing.
################
fps: 2
################
# This will save a clip for each tracked object by frigate along with a json file that contains
# data related to the tracked object. This works by telling ffmpeg to write video segments to /cache
# from the video stream without re-encoding. Clips are then created by using ffmpeg to merge segments
# without re-encoding. The segments saved are unaltered from what frigate receives to avoid re-encoding.
# They do not contain bounding boxes. 30 seconds of video is added to the start of the clip. These are
# optimized to capture "false_positive" examples for improving frigate.
#
# NOTE: This will only work for camera feeds that can be copied into the mp4 container format without
# encoding such as h264. I do not expect this to work for mjpeg streams, and it may not work for many other
# types of streams.
#
# WARNING: Videos in /cache are retained until there are no ongoing events. If you are tracking cars or
# other objects for long periods of time, the cache will continue to grow indefinitely.
################
save_clips:
enabled: True
#########
# Number of seconds before the event to include in the clips
#########
pre_capture: 30
################
# Configuration for the snapshots in the debug view and mqtt
################
snapshots:
show_timestamp: False
################
# Camera level object config. This config is merged with the global config above.
################
objects:
track:
- person
- car
filters:
person:
min_area: 3300
max_area: 16000
threshold: 0.75
car:
max_area: 20000
threshold: 0.5
I’ve also adjusted my docker-compose file to map the clips and cache folders:
The cache has 3 .mp4 files (one for each camera), which are all increasing in size, but I don’t see anything in the clips folder. In the logs I see this:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55623137f4c0] moov atom not found
/cache/backyard-20200805094855.mp4: Invalid data found when processing input
bad file: backyard-20200805094855.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5568ed73d4c0] moov atom not found
/cache/cars-20200805094848.mp4: Invalid data found when processing input
bad file: cars-20200805094848.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x565462c2f4c0] moov atom not found
/cache/front-20200805094840.mp4: Invalid data found when processing input
bad file: front-20200805094840.mp4
Is that your entire config? What are your ffmpeg parameters? vsync drop prevents the segment feature from working. Your cameras must output in a format compatible with mp4 natively. Frigate doesn’t decode and encode for the cache.
Nope, not the whole config. Here’s the ffmpeg params:
web_port: 5000
mqtt:
host: <redacted>
topic_prefix: frigate
# client_id: frigate # Optional -- set to override default client id of 'frigate' if running multiple instances
user: <redacted> # Optional -- Uncomment for use
password: <redacted> # Optional -- Uncomment for use
#################
# Default ffmpeg args. Optional and can be overwritten per camera.
# Should work with most RTSP cameras that send h264 video
# Built from the properties below with:
# "ffmpeg" + global_args + input_args + "-i" + input + output_args
#################
ffmpeg:
global_args:
- -hide_banner
- -loglevel
- panic
hwaccel_args:
- -hwaccel
- vaapi
- -hwaccel_device
- /dev/dri/renderD128
- -hwaccel_output_format
- yuv420p
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer
- -flags
- low_delay
- -strict
- experimental
- -fflags
- +genpts+discardcorrupt
- -vsync
- drop
- -rtsp_transport
- tcp
- -stimeout
- '10000000'
- -use_wallclock_as_timestamps
- '1'
output_args:
# - -vf
# - mpdecimate
- -f
- rawvideo
- -pix_fmt
- rgb24
I don’t do anything per camera.
So yes, I see I do do the vsync drop. I removed it and immediately noticed that it is now creating multiple clips instead of one big clip per camera in the cache folder.
Is blakeblackshear/frigate:save_clips-8e78760 the best image to use for this?
EDIT: OK it is now working – thanks Blake!
Quick question – when does the cache clear up? I see it expand pretty much on all my cameras, even the ones with very little movement. I do see it clears up though.
Frigate keeps the cache around as long as there is an object being tracked within 90 seconds of the video clip. It does not handle long running events at all, so if you had a car that was being tracked for hours (even stationary), it would retain the cache for hours and ultimately assemble a clip hours long for that car before removing the cache.
Maybe there’s a better way, but I’ve created an action that saves the snapshot prior to the notification.
- id: '1585636351359'
alias: NOTIFY Front Door Person
description: ''
trigger:
- entity_id: binary_sensor.front_door_camera_motion
platform: state
to: 'on'
condition: []
action:
- data:
filename: /tmp/snapshot_front_door_camera_last_person.jpg
entity_id: camera.front_door_camera_last_person
service: camera.snapshot
- data:
data:
images:
- /tmp/snapshot_front_door_camera_last_person.jpg
message: Person detected at front door
title: person
service: notify.email
Since my frigate instance is on a completely separate host, saving the snapshot to the HA config directory wouldn’t be an option.
@ebendl, @blakeblackshear: I really like the idea of using frigate for object detection but keeping a ‘real’ highres video of objects detected.
Challenges:
Requires permanent recording for ‘lookback’ functionallity
The HA stream component isn’t really stable for me - I guess that would be the right way. Is that just me?
Potentially frigate gets a ‘lowres’ stream only for performance reasons - video should be highres.
I didn’t see save_clips earlier (cool) - but from your description I’m not sure its a perfect fit.
The save_clips videos didn’t play smoothly for my Foscam / Reolink - blakeblackshear said that might be part of how they are recorded.
Alternate approach:
Start permanent 120 second recordings, e.g. with openRTSP which doesn’t seem to involve transcoding (<2% CPU)
Start this e.g. every 60 seconds, keep a 10min history
use network share / memory (/dev/shm) - consider implications on required memory / disk wearout
Have those recordings, e.g. <camname_date_%H%M.mpg> available via http or nas mount
Create automation on frigate notification - e.g. send via telegram, store to nas, …
Logic to pick right recording to be discussed, e.g. rounddown(trigger.ts, min)
When adding openRTSP to the frigate container I’m able to store the data within the container. The save_clip stuff didn’t play well for me - this does starting with the first keyframe.
cat Dockerfile
FROM blakeblackshear/frigate:dev
RUN apt-get update
RUN apt-get -y install livemedia-utils
I’m still trying to get my head around how to do this best - maybe something more generic with stream plugin is better, not sure. I’d assume with the ease of integration of frigate & the performance with coral this might be something more people could be interested in.
I do a lot of this with my setup. I use ffmpeg to record 1 minute segments of my high resolution feed and some bash scripting to move those into YYYY/MM/DD/HH folders. Then I use a bash script to clear out mp4 files older than 2 weeks. Then I make that folder available through nginx so I can view the clips. The save_clips feature was designed to capture videos as frigate sees them so I can collect false/true positive examples for testing. However, I did design it in a way that I can have frigate output events that can then be used to assemble mp4 “playlists” of the video clips I store with ffmpeg. As is, you could use ffmpeg to write full resolution videos to the cache directory and frigate will slice and assemble them for you. I plan to bring these things together so there is a “frigate-nvr” project for storing and capturing camera feeds that works alongside frigate itself.
Hey @blakeblackshear. Lately I’ve been playing with the FFmpeg camera component in hass and I’ve been noticing a number of the problems that used to plague me and others in frigate happening (streams turning into a smear, streams shutting down and starting up again).
It seems you have done a really effective job of finding workarounds or a setup for whatever has been causing all the stream issues that people used to have (at least you’ve solved all the ones I had)…
I wonder if you would consider adding some of your accumulated knowledge into the ffmpeg camera component in homeassistant? I would love to see it work as well as frigate does so that I can stop using the stream component and generic camera in homeasstant which adds a 10 second delay to the stream.
You are really at the forefront of making all this stuff work. Thanks again for a rock solid component.
What are you trying to accomplish by switching to the ffmpeg camera component?
I have never used the ffmpeg camera because I don’t want to decode the video feed an extra time, but I am guessing some of the input parameters just need to be utilized to get similar results. The stream component will always have a delay because it is creating an HLS stream to reduce CPU load and devices like the chromecast only work with HLS streams. I force my cameras to have an I-frame every second so my delay is minimized.
I run a single instance of ffmpeg to generate an HLS stream and store 1 minute segments of my cameras for long term storage like this. Then I setup my cameras like this:
- name: Backyard Camera
platform: generic
still_image_url: http://default-frigate:5000/back/latest.jpg
stream_source: http://nvr-nginx/stream/back.m3u8
It is a bit wasteful that HA creates an HLS stream from an HLS stream, but it works well enough. My HLS stream has a 3-4s delay and HA adds another 1s delay (probably because my I-frame rate is set to same FPS) to recreate the stream again. Not too bad with a 5s delay, but I don’t view the stream often.
Thanks for sharing your expertise on this, I’ve been fighting with it for more than a year.
So my goal is trying to get rid of that 10-20 second delay. I also have an I-frame every second but the delay is still 10-20 seconds. I don’t actually need any storage or chromecast or any of that… All I need is cameras that show up properly in the UI without delay so that I can use them for video doorbell, monitoring the kids while they are outside and the like. When I use the generic component without stream turned on, I get near real time feeds but they don’t work well on my tablets or iOS devices. The feed often turns to a broken image.
FFmpeg ones seem much more reliable but I just noticed they are eating all my CPU so I can’t keep all the streams running properly. Probably this is where my problems are actually coming from rather than the component itself, I just didn’t expect my CPU to jump from 20% to 100% just by changing the camera component.
Your idea is a very interesting one… So let me understand, how do you run a single instance of ffmpeg? Do you have just one camera set as ffmpeg and the rest set as generic? Do you run the stream component?
I haven’t found any way to run the stream component without the lag, so I’m now trying to find a solution where I don’t use it but still get near real time cameras that will display properly in the UI and not use all my CPU…
I run ffmpeg independent of HASS in a separate container, then use the generic camera to point at the HLS stream from that camera. The stream component then makes an HLS stream from my HLS stream. I didn’t see any difference in latency with and without the stream component enabled.
Either way, it’s not going to be as low latency as you want. I am eventually trying to achieve the same thing by using jsmpeg, but it will require a custom lovelace card and an addon with ingress at least. I also plan to add it as a capability in frigate so you can avoid decoding the video stream more than once.
At the moment, I don’t know if there is really a way to get low latency video feeds in HASS. It is just a fundamentally difficult problem with HTML5.
If I turn off the stream component, I am able to get what I consider to be low enough latency streams (~2 seconds) from hass using either the generic or ffmpeg component when they are displayed as (I believe) MJPEG in frontend rather than HLS. Framerate suffers a bit, but even .5-1 FPS is okay to me. But without the stream component, I have the problem with them not displaying reliably on some devices (generic camera) or using all my cpu (ffmpeg camera component).
It’s only when I turn the stream component on that the lag goes up to 10-20 seconds, but the CPU goes way down and they display nicely on all devices.
So, let me ask you. When you run your own ffmpeg container, and the generic camera component, do you get lower latency or lower CPU usage than you would running the FFMPEG camera in hass against the same cameras? Roughly how much latency do you get? I’m assuming you still need use the stream component in hass to deal with the HLS stream, right?
ATM it seems that my only way t is to get a stronger hass machine to run lots of FFMPEG on.
Nice work on the latest RC @blakeblackshear! I’m playing around with the clips and they seem to be working great so far. Does the cache directory clean itself up? Or should I be throwing up a crontab to clean this up periodically?
