Local realtime person detection for RTSP cameras

Hi all.

I don’t seem unable to save clips. In the “media/clips” directory I just see snapshot png/jpgs. But no videos.
I am running Frigate in a container on my NAS using a Coral USB
I think I have enabled the role for clips?
Any Ideas?

config.yml

mqtt:
  host: <MQTT server on the HA Sever>
  topic_prefix: frigate
  client_id: frigate
  user: <USER>
  password: <PASSWORD>
  stats_interval: 60
logger:
  default: debug

detectors:
  coral:
    type: edgetpu
    device: usb
clips:
  enabled: True
  pre_capture: 5
  post_capture: 5
  objects:
    - person
    - cat
    - dog
cameras:
  back:
    #save_clips:
    #  enabled: true
    #  pre_capture: 10
    ffmpeg:
      inputs:
        - path: rtsp://viewer:<PASSWORD>@<CAMIP>:554/Streaming/Channels/1
          roles:
            - detect
            - rtmp
            - clips
            - record
    objects:
      track:
        - person
        - car
        - dog
        - cat
        - bird
    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      objects:
        - person
        - cat
        - dog
      retain:
        default: 10
        objects:
          person: 15
          dog: 15
          cat: 15
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: False
      crop: False
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15
          dog: 15
          cat: 15
          bird: 2
    width: 2688
    height: 1520
    fps: 20
   

Possible relevant debug?

[2021-07-12 11:46:15] frigate.events                 ERROR   : ffmpeg version 4.3.2 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x565091caa500] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x565091caa500] moov atom not found
[concat @ 0x565091ca8740] Impossible to open 'pipe:/tmp/cache/back-20210712114542.mp4'
pipe:: Invalid data found when processing input

[2021-07-12 11:46:15] peewee.sqliteq                 DEBUG   : received query INSERT INTO "event" ("id", "label", "camera", "start_time", "end_time", "top_score", "false_positive", "zones", "thumbnail", "has_clip", "has_snapshot") VALUES (?, ?, ?, ?, ?, ?, ?, json(?), ?, ?, ?)
[2021-07-12 11:46:15] peewee                         DEBUG   : ('INSERT INTO "event" ("id", "label", "camera", "start_time", "end_time", "top_score", "false_positive", "zones", "thumbnail", "has_clip", "has_snapshot") VALUES (?, ?, ?, ?, ?, ?, ?, json(?), ?, ?, ?)', ['1626086755.710832-p6iufx', 'person', 'back', 1626086755.710832, 1626086764.66152, 0.8046875, False, '[]', '****************** LOTS of DATA************=', False, True])
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Event received: end back 1626086755.930308-qzich6
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Event received: start back 1626086771.795203-qsbmar
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Cleaning up cached file back-20210712114422.mp4
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Cleaning up cached file back-20210712114432.mp4
[2021-07-12 11:46:15] frigate.mqtt                   DEBUG   : Received mqtt message on frigate/back/person.
[2021-07-12 11:46:15] frigate.mqtt                   DEBUG   : Received mqtt message on frigate/back/person/snapshot.

Is there any way to configure frigate to use external USB mounted storage for recordings?

Hello together,
because of zmNinja - Support ends by end of 2021 i looked for alternatives…
Here i found Frigate.
I startet configuring frigate on my Unraidserver.
i’m using the frigate-nvidia docker container…
For some reasons i can’t get my Reolink RLC520 Camera’s to work properly…

Here is my Configuration:

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - cuda
    - -c:v 
    - h264_cuvid
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -c:v h264_cuvid
  output_args:
    detect: -f rawvideo -pix_fmt yuv420p -filter:v fps=fps=5
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    rtmp: -c copy -map 0 -f flv
      - -filter:v
      - fps=fps=5
mqtt:
  host: 10.0.0.65
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  stats_interval: 60
  
cameras:
  Front:
    ffmpeg:
      output_args:   
        rtmp: -c copy -map 0 -f flv
      inputs:
        - path: rtsp://admin:[email protected]/Streaming/Channels/1
          roles:
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1440
    fps: 5
    clips:
      enabled: false
      pre_capture: 5
      post_capture: 5
      objects:
        - person
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15
    motion:
      mask:
        - 0,0,1769,0,1885,0,1897,152,1669,165,1457,255,989,485,529,740,0,1083
    zones:  
      Vorgarten:
        coordinates: 0,1440,1737,1440,2560,1440,2560,0,1870,0,1878,163,1664,173,1555,219,1395,288,1156,411,788,609,704,650,63,1046,0,1079
  Garten:
    ffmpeg:
      global_args:
        - '-hide_banner'
        - '-loglevel'
        - info
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -rw_timeout
        - "5000000"
        - -use_wallclock_as_timestamps
        - "1"
      output_args:   
#        rtmp: -c:v libx264 -an -f flv -an
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy


      inputs:
        - path: rtmp://10.0.0.17/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=xxxx
          roles:
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1920
    fps: 5
    rtmp:
      enabled: True
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
      objects:
        - person
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15    
    motion:
      mask:
        - 1836,44,2441,346,2389,339,2378,404,2560,426,2560,0,0,0,0,347,113,368,79,293,563,54,1124,62,1598,73
    zones:  
      hinten_links:
        coordinates: 2560,1920,2560,418,2375,407,2378,347,2441,338,1824,49,1557,65,1125,52,1095,62,550,57,94,296,111,380,0,304,0,1920
  Gartenhaus:
    ffmpeg:
      global_args:
        - '-hide_banner'
        - '-loglevel'
        - info
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -rw_timeout
        - "5000000"
        - -use_wallclock_as_timestamps
        - "1"
      output_args:   
#        rtmp: -c:v libx264 -an -f flv -an
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy


      inputs:
        - path: rtmp://10.0.0.19/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=xxxx
          roles:
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1920
    fps: 5
    rtmp:
      enabled: True
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
      objects:
        - person
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15
    motion:
      mask:
        - 89,490,135,466,496,409,444,382,539,342,542,296,1446,32,1921,252,2132,409,2088,431,2086,623,2560,701,2560,0,0,0,0,509
    zones:  
      hinten_rechts:
        coordinates: 2560,1920,2560,1920,2560,705,2081,609,2096,415,2148,415,1918,249,1436,32,946,166,537,296,539,339,491,384,488,428,144,474,0,485,0,1920
    
objects:
  track:
    - person
    - cat
    - dog
    - car 
  filters:
    person:
      #min_area: 5000
      #max_area: 100000
      #min_score: 0.6
      threshold: 0.7
      
detectors:
  cpu1:
    type: cpu
  cpu2:
    type: cpu

First error ist the following…

frigate.video                  INFO    : Garten: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : Garten: ffmpeg process is not running. exiting capture thread...
ffmpeg.Garten.detect           ERROR   : Input #0, flv, from 'rtmp://10.0.0.17/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=27021986':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :   Duration: 00:00:00.00, start: 1626091836.851000, bitrate: N/A
ffmpeg.Garten.detect           ERROR   :     Stream #0:0: Data: none
ffmpeg.Garten.detect           ERROR   :     Stream #0:1: Video: h264, yuv420p(progressive), 2560x1920, 30 fps, 1k tbr, 1k tbn
ffmpeg.Garten.detect           ERROR   :     Stream #0:2: Audio: aac, 16000 Hz, mono, fltp
ffmpeg.Garten.detect           ERROR   : [segment @ 0x55d799e3f340] Opening '/tmp/cache/Garten-20210712141038.mp4' for writing
ffmpeg.Garten.detect           ERROR   : Output #0, segment, to '/tmp/cache/Garten-%Y%m%d%H%M%S.mp4':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :     encoder         : Lavf58.45.100
ffmpeg.Garten.detect           ERROR   :     Stream #0:0: Video: h264, yuv420p(progressive), 2560x1920, q=2-31, 30 fps, 1k tbr, 16k tbn, 1k tbc
ffmpeg.Garten.detect           ERROR   :     Stream #0:1: Audio: aac, 16000 Hz, mono, fltp
ffmpeg.Garten.detect           ERROR   : Output #1, flv, to 'rtmp://127.0.0.1/live/Garten':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :     encoder         : Lavf58.45.100
ffmpeg.Garten.detect           ERROR   :     Stream #1:0: Video: h264 ([7][0][0][0] / 0x0007), yuv420p(progressive), 2560x1920, q=2-31, 30 fps, 1k tbr, 1k tbn, 1k tbc
ffmpeg.Garten.detect           ERROR   :     Stream #1:1: Audio: aac ([10][0][0][0] / 0x000A), 16000 Hz, mono, fltp
ffmpeg.Garten.detect           ERROR   : Stream mapping:
ffmpeg.Garten.detect           ERROR   :   Stream #0:1 -> #0:0 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:2 -> #0:1 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:1 -> #1:0 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:2 -> #1:1 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:1 -> #2:0 (h264 (h264_cuvid) -> rawvideo (native))
ffmpeg.Garten.detect           ERROR   : Press [q] to stop, [?] for help
ffmpeg.Garten.detect           ERROR   : frame=    2 fps=0.0 q=-1.0 q=-1.0 q=0.0 size=N/A time=00:00:02.02 bitrate=N/A speed=3.57x
ffmpeg.Garten.detect           ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.Garten.detect           ERROR   : Finishing stream 2:0 without any data written to it.
ffmpeg.Garten.detect           ERROR   : Output #2, rawvideo, to 'pipe:':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :     encoder         : Lavf58.45.100
ffmpeg.Garten.detect           ERROR   :     Stream #2:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2560x1920 [SAR 1:1 DAR 4:3], q=2-31, 294912 kb/s, 5 fps, 5 tbn, 5 tbc
ffmpeg.Garten.detect           ERROR   :     Metadata:
ffmpeg.Garten.detect           ERROR   :       encoder         : Lavc58.91.100 rawvideo
ffmpeg.Garten.detect           ERROR   : [flv @ 0x55d799ed9700] Failed to update header with correct duration.
ffmpeg.Garten.detect           ERROR   : [flv @ 0x55d799ed9700] Failed to update header with correct filesize.
ffmpeg.Garten.detect           ERROR   : Error writing trailer of rtmp://127.0.0.1/live/Garten: Connection reset by peer
ffmpeg.Garten.detect           ERROR   : frame=    2 fps=0.0 q=-1.0 Lq=-1.0 q=0.0 size=N/A time=00:00:02.02 bitrate=N/A speed=3.39x
ffmpeg.Garten.detect           ERROR   : video:2409kB audio:4kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg.Garten.detect           ERROR   : Conversion failed!

But the Camera livestream is comming up correctly in frigate …
I have looked into the setting but the resolution is configured correctly…
Are there any errors in my config with RTMP?!

thanks in advance
chris

I am not sure what the difference between the Reolink RLC520 and my Reolink RLC-410-5MP is. However, I could only get it working reliably with RTMP not RTSP. Here is how I did my config if this helps:

  frontdriveway:
    ffmpeg:
      input_args: #Use for RTMP
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -rw_timeout
        - '5000000'
        - -use_wallclock_as_timestamps
        - '1'
      hwaccel_args:
        - -hwaccel
        - qsv
        - -qsv_device
        - /dev/dri/renderD128
      output_args:
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
      inputs:
        - path: rtmp://192.168.1.105/bcs/channel0_sub.bcs?channel=0&stream=1&user=USERNAME&password=PASSWORD
          roles:
            - detect
            - rtmp
            - record
        - path: rtmp://192.168.1.105/bcs/channel0_main.bcs?channel=0&stream=0&user=USERNAME&password=PASSWORD
          roles:
            - clips
    width: 640
    height: 480
    fps: 5
    record:
      enabled: true
      retain_days: 10
    rtmp:
      enabled: true
    detect:
      enabled: true
      max_disappeared: 60
    clips:
      enabled: true
      pre_capture: 10
      post_capture: 10
      objects:
        - person
      required_zones: []
      retain:
        default: 7
        objects:
          person: 30
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      track:
        - person

Nice i just played a bit with the resolution and set it to 2048x1536 and now it working properly…

Thanks

1 Like

Hi,
Yes i made a tutorial:
https://community.home-assistant.io/t/solved-hassos-mount-nas-network-share/

1 Like

I hate to see yellow font logs :slight_smile:

any fix/patch coming soon to take care of this?

Loving this integration… thank you!

Is there a way to enable/disable recording on a per-camera basis from Homeassistant? I see the switches for detect, clips and snapshots, but nothing similar for recording.

1 Like

As per docs

    # Optional: save clips configuration
    clips:

      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      #required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 14
        # Optional: Per object retention days
        #objects:
          #person: 15

    # Optional: 24/7 recording configuration
    record:
      # Optional: Enable recording (default: global setting)
      enabled: False
      # Optional: Number of days to retain (default: global setting)
      retain_days: 30

I am a longtime zoneminder/zmeventnotification user and I am really impressed with Frigate, really easy to setup!

You can only set width & height once per camera, and not per input, right? When using a lower resolution stream (640x360) for object detection, and a higher resolution (1920x1080) for recording/clips/rtmp, what width & height should be used in that case?
If using two inputs for a camera, can you select which of them is displayed in the web ui? Currently my low resolution stream (640x360) is the primary in the webui.

I saw that… I should have been clearer in my question. Is there a way to enable/disable recording from HA? For instance, when I’m home, I only want to record my front cameras; if I’m away I want all cameras recording.

I thought MQTT have that, but i checked and i don’t see in docs. Lets ask @blakeblackshear

Hey @allenporter I know this is from a long time ago but I’m finally at a point where I have some mental capacity for this kind of stuff. Would the HLS URL look something like this?

blob:https://<HA_External_URL>/77tg6761-acd5-4563-9ea4-976843835a41

Probably not? I’m interested in getting this going again because I heard some pretty interesting things about playing Frigate clips via a lovelace card.

Has anyone a nice lovelance card setup for viewing history events.

greetings Chris

1 Like

I use a picture card and the camera entity made by the frigate integration to display the last detection of each type. It’s super simple and you’re likely looking for something more complicated, but if you weren’t aware maybe it’ll help.

Ok can you give me a screenshot?
And maybe the code for the lovelance too? :slight_smile:

mfg Chris

Just an update, the Nest integration sometimes spits out errors with the camera RTSPS stream path. I grabbed it an it worked.

My fear is the token attached to the end of the stream path is going to expire and I’ll lose the stream.

image

type: picture-glance
title: test
entities: []
camera_image: camera.front_door_camera_person
camera_view: live

Hi @keithcroshaw , One example is when you call the camera.play_stream service to the media palyer its going to send over a url like https://home-assistant.your-domain/api/hls/<long string>/master_playlist.m3u8

1 Like

Yes, that is an expiring URL that changes every few minutes.

1 Like