Local realtime person detection for RTSP cameras

[denver]

that’s just how I felt when I found out this

[ravensteijn]

when i set my cam from 1080 to 4k i get the error below. When i set RTMP to off it works without errors.

ffmpeg:
  global_args:
    - '-hide_banner'
    - '-loglevel'
    - error

i get this error:

ffmpeg.voordeur.clips_rtmp     ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.voordeur.clips_rtmp     ERROR   : Error writing trailer of rtmp://127.0.0.1/live/voordeur: Connection reset by peer
watchdog.voordeur              INFO    : Terminating the existing ffmpeg process...
watchdog.voordeur              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.voordeur.clips_rtmp     ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.voordeur.clips_rtmp     ERROR   : Error writing trailer of rtmp://127.0.0.1/live/voordeur: Connection reset by peer
watchdog.voordeur              INFO    : Terminating the existing ffmpeg process...
watchdog.voordeur              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.voordeur.clips_rtmp     ERROR   : av_interleaved_write_frame(): Broken pipe
ffmpeg.voordeur.clips_rtmp     ERROR   : Error writing trailer of rtmp://127.0.0.1/live/voordeur: Broken pipe
watchdog.voordeur              INFO    : Terminating the existing ffmpeg process...
watchdog.voordeur              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.voordeur.clips_rtmp     ERROR   : av_interleaved_write_frame(): Broken pipe
ffmpeg.voordeur.clips_rtmp     ERROR   : Error writing trailer of rtmp://127.0.0.1/live/voordeur: Broken pipe
watchdog.voordeur              INFO    : Terminating the existing ffmpeg process...
watchdog.voordeur              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.voordeur.clips_rtmp     ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.voordeur.clips_rtmp     ERROR   : Error writing trailer of rtmp://127.0.0.1/live/voordeur: Connection reset by peer
watchdog.voordeur              INFO    : Terminating the existing ffmpeg process...
watchdog.voordeur              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.voordeur.clips_rtmp     ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.voordeur.clips_rtmp     ERROR   : Error writing trailer of rtmp://127.0.0.1/live/voordeur: Connection reset by peer
watchdog.voordeur              INFO    : Terminating the existing ffmpeg process...
watchdog.voordeur              INFO    : Waiting for ffmpeg to exit gracefully...

[danbutter]

Yeah…I think that’s because you are triggering on any event coming from frigate if you followed that template. There is a method to kepp an automation from running if it has run in the last x period of timr, but I can’t find it on my phone. I’ll try to post it when I get home. (Assuming I can remember of course) Maybe someone else has it in the meantime.

[wills106 (Will)]

I don’t really know much about ffmpeg…

But I was getting a lot of Green Screen type corruption when I first tried 4K.
So I don’t know if some cameras are sending not quite compliant video or there is a buffer error in 4K.

Adding the following prevented that:

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
  input_args:
    - '-avoid_negative_ts'
    - make_zero
    - '-fflags'
    - nobuffer
    - '-flags'
    - low_delay
    - '-strict'
    - experimental
    - '-fflags'
    - +genpts+discardcorrupt
    - '-use_wallclock_as_timestamps'
    - '1'
    - '-an'
    - '-dn'
    - '-flags'
    - '-global_header'

Don’t know if it will make a difference as I don’t really know what it does, but worth a go?

[sdc]

I’ve been playing around with the 8.0 release - I’ve setup a camera with the role of detect and rtmp - i’ve also enabled snapshots in the config for one of the cameras.

Should i be seeing JPG snapshots automatically written to the media/clips directory or is this not how it works? Currently my media folders are empty and when I do some testing in front of the camera with the new frigate webui up, not seeing any of the detection bounding boxes, so not sure if something is off with my config.

[biggen1684]

You need to setup a condition like the following:

condition: 
      - "{{ trigger.payload_json['after']['label'] == 'person' }}"
      - "{{ ( as_timestamp(now()) - as_timestamp(state_attr('automation.driveway_notfication', 'last_triggered')) |int(0) ) > 45 }}"

I have mine set to 45 seconds here. This way I only get notifications no sooner than once every 45 seconds.

[paulstronaut (Paul)]

@sdc

Should i be seeing JPG snapshots automatically written to the media/clips directory or is this not how it works?

Can you paste your config somewhere? You need to ensure that snapshots are enabled, eg

snapshots:
  enabled: true

[paulstronaut (Paul)]

@arkoko

One question though. What means “rebuild” on the addon screen in HA? When i should use this button?

That effectively deletes the current container and any internal cache for it and re-creates the docker container. You probably never need to do this.

[sdc]

Figured this one out - didn’t realize the switches in HA needed to be turned on for detect to work on each camera! Saw someone post this in a thread on here, but might be something worth adding to the next release of the docs (apologies if i missed it).

[srk23]

Perfect. I guess if I want to make it per camera, I will have to set up individual automations for each camera, but it will do the job nicely.

[paulstronaut (Paul)]

@srk23

One observation is that if people are near the camera for any period of time, or if there are multiple people in a group, the automation will send a notification repeatedly in rapid-fire. Is there any way in an automation to limit a re-notification for a particular camera for a short period (e.g. 10 secs) to limit this behavior?

Frigate currently does one event per object detected and tracked, so I just want to confirm that this is expected behavior.

I’m not much help with built-in automations for HA, but if you’re using Node-RED, there’s a built-in throttle node that can be set to avoid sending too many notifications in your chosen time window.

[hasshoolio]

Few discussions over on github about event rate and notifications. These may help

https://github.com/blakeblackshear/frigate/discussions/393
https://github.com/blakeblackshear/frigate/discussions/559
https://github.com/blakeblackshear/frigate/discussions/382

[Melkor82 (Marco)]

i have azone seen by two cameras so i defined it in both. Before there was only one person sensor for that zone, now i got 2 different sensors one for each camera with the prefix of the original camera name but the sensor never update even though mqtt topic is still reporting the zone name without prefixes and get the correct value

[blakeblackshear]

Thanks. I will look into it.

[HITChris]

I’m in this camp as well. Prior to Blake’s cobbling this incredible platform together, NVR / object detection for me involved:

• BlueIris for recording + motion detection + mjpeg streaming
• Rob’s Rekognition component for object detection
• NodeRed for BI’s detection alerts, calling Rekognition, and subsequent interpretation/response

I think the thing holding me back from deleting that BI VM is Frigate’s 60s recordings limitation. I’m guessing 60s is partly to limit cache size, and yeah typically when you want to go and look back at something, you probably have some idea as to approximately when the target event took place. But it seems difficult to navigate large spans of time across so many individual files. Cobbling the .mp4s together outside of Frigate and presenting them elsewhere is an option I might trend towards, but for me at least it would be nice if the Frigate-based recordings structure (and subsequent presentation via HA) more followed out-of-the-box NVR softwares

[blakeblackshear]

The recordings are in 60s segments to prevent data loss. If/when ffmpeg loses connection to your camera, the process exits and the resulting file is often corrupted. Using 60s segments prevents more than a minute of data loss. There are a few possible solutions here. There may be a way to ensure ffmpeg exits cleanly without losing the rest of the file, but I have never bothered to look. I could also look at appending these 60s segments to a single file or merging them on some interval. Perhaps the best solution would be to create the mp4 dynamically on the fly and chain the segments together. That way its just a continuous video stream and you can seek wherever you want. I may end up building a custom panel for homeassistant (not using ingress) that has a much richer interface for viewing historical footage.

To be honest though, in the several years I have been using ffmpeg to save 24/7 video as 60s segments, I can count on one hand the number of times I have gone back and reviewed the footage. I feel like that time is better spent improving the realtime analysis and other features.

[Dredzed (Zak)]

Just wanted to thank Blake for his amazing work on this. Prior to using Frigate, I was recording from 3 cameras on motion detection, which was next to useless due to them being triggered by clouds/ moving trees etc. Now every clip is relevant and so much more useful! Really appreciate the hard work you have put in.

[ekos2001 (Yevgen Kostrubin)]

@davidvf

Thanks for the QSV support in beta2, it works great now!

I’m trying to enable HW acceleration for my 10th Gen processor, but do not see any CPU usage drop. Also intel-gpu-top shows nothing.
I have the same config:

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - qsv
    - -qsv_device
    - /dev/dri/renderD128

@davidvf, which OS do you have?

[davidvf]

Good catch, i didn’t know about the command intel_gpu_top. But indeed, for me it does not show anything either.
I am running:

# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.10 (Groovy Gorilla)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.10"
VERSION_ID="20.10"

EDIT:
It looks like if i change the command to the below i start seeing movement in the intel_gpu_top

ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i rtsp://192.168.0.5:7447/16gU54OAVReFohX3 -c:v h264_qsv hwaccel-test.mp4

I will try to get that in my frigate config.yml tonight…

So basically i have added in the following to the input_args and the output_args
-c:v h264_qsv

[Minglarn]

Hi…
Using rc-3 and it works great, but… I seem to have a issue with MASKs. …
I’ve made a mask that covers areas where I don’t want any kind of detection from.

However, when I check the events I see a detections all the time…

This is my config file:

 mot_parkering:
    ffmpeg:
      inputs:
        - path: >-
            rtmp://192.168.1.110/bcs/channel0_main.bcs?channel=0&stream=0&user=XXX
          roles:
            - clips
            - rtmp
        - path: >-
            rtmp://192.168.1.110/bcs/channel0_sub.bcs?channel=0&stream=0&user=XXX
          roles:
            - detect
    width: 640
    height: 480
    fps: 5
    detect:
      enabled: true
      max_disappeared: 25
    clips:
      enabled: true
    motion:
      mask:
        - '640,0,640,260,545,215,466,186,511,97,353,59,102,119,0,158,0,0'

Have I missed something?