Local realtime person detection for RTSP cameras

@DrSpaldo, did you ever get Authelia to work with HA? I decided to try to tackle that tonight and failed, HA seems like a challenge since I want it to work with the Android mobile app.

Hi @SpikeyGG , unfortunately I didn’t. I had a few hours spare, so I tried and failed :frowning:

I may give it another go if these kids ever give me a break!

I have been a little hesitant/cautious as I didn’t want to break my current install.

We can move the discussion over to your other thread for when I try again

what does the following show for you?

choose you camera
click Debug at the top right
then under the image, click
SHOW OPTIONS
toggle
Bounding box
Motion boxes

Does it show a clean image or do you see a bunch of boxes popping up.
When I’m viewing an image, I get constant boxes appearing.

It just so happens that I increased camera resolution on two cams me immediately after the time stamp from cameras begin trigger motion detect

As i removed the timestamp from the camera setup, now its definetly okay.So that was the problem

For anyone interested, I have posted in @SpikeyGG 's thread :

Anyone have Authelia working with HA to handle authentication? - Configuration - Home Assistant Community (home-assistant.io)

It is how I got Home Assistant working with Authelia. I used pretty much the same method for Frigate to be authenticated with Authelia (and Cloudflare, Nginx, etc) on Unraid

2 Likes

Yup, I have this issue too. It seems to get worse with time. After restarting the docker instance for Frigate, the system will record clips about 50-60% of the time… after a week of running it seems to completely stop recording clips altogether.

I still get the notifications from Home Assistant and it includes a picture from the incident but there’s no clip of it and the messages in the log file just say things like this:

frigate.events                 WARNING : Unable to create clip for out_the_back and event 1626012762.663106-3yhjy7. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for out_the_back and event 1626012747.026751-lxw7w8. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for out_the_back and event 1626012754.493196-3tgtmd. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for out_the_back and event 1626012741.619668-bl96jq. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for out_the_back and event 1626012741.619668-e9tvqg. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for east_yard and event 1626012631.266207-fhlgin. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for out_the_back and event 1626012484.739979-6djz75. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for east_yard and event 1626011021.167499-mhvsl6. There were no cache files for this event.,
frigate.events                 WARNING : Unable to create clip for east_yard and event 1626010396.646361-c2u3a7. There were no cache files for this event.

All my cameras are Reolink, the clips role is assigned to the rtmp sub channel. Like this:

rtmp://{OUT_THE_BACK_CAMERA_IP}/bcs/channel0_main.bcs?channel=0&stream=0&user={FRIGATE_RTSP_USERNAME}&password={FRIGATE_RTSP_PASSWORD}

This is the biggest thing bugging me about Frigate right now because it’s completely inconsistent on when it decides to record a clip and when it doesn’t. I have not been able to find any way to increase the logging in order to debug the problem.

1 Like

Actualy for me everything sorted out, the problem was with config.

What did you change? I still cannot seem to get mine to function properly.

Hi all.

I don’t seem unable to save clips. In the “media/clips” directory I just see snapshot png/jpgs. But no videos.
I am running Frigate in a container on my NAS using a Coral USB
I think I have enabled the role for clips?
Any Ideas?

config.yml

mqtt:
  host: <MQTT server on the HA Sever>
  topic_prefix: frigate
  client_id: frigate
  user: <USER>
  password: <PASSWORD>
  stats_interval: 60
logger:
  default: debug

detectors:
  coral:
    type: edgetpu
    device: usb
clips:
  enabled: True
  pre_capture: 5
  post_capture: 5
  objects:
    - person
    - cat
    - dog
cameras:
  back:
    #save_clips:
    #  enabled: true
    #  pre_capture: 10
    ffmpeg:
      inputs:
        - path: rtsp://viewer:<PASSWORD>@<CAMIP>:554/Streaming/Channels/1
          roles:
            - detect
            - rtmp
            - clips
            - record
    objects:
      track:
        - person
        - car
        - dog
        - cat
        - bird
    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      objects:
        - person
        - cat
        - dog
      retain:
        default: 10
        objects:
          person: 15
          dog: 15
          cat: 15
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: False
      crop: False
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15
          dog: 15
          cat: 15
          bird: 2
    width: 2688
    height: 1520
    fps: 20
   

Possible relevant debug?

[2021-07-12 11:46:15] frigate.events                 ERROR   : ffmpeg version 4.3.2 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x565091caa500] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x565091caa500] moov atom not found
[concat @ 0x565091ca8740] Impossible to open 'pipe:/tmp/cache/back-20210712114542.mp4'
pipe:: Invalid data found when processing input

[2021-07-12 11:46:15] peewee.sqliteq                 DEBUG   : received query INSERT INTO "event" ("id", "label", "camera", "start_time", "end_time", "top_score", "false_positive", "zones", "thumbnail", "has_clip", "has_snapshot") VALUES (?, ?, ?, ?, ?, ?, ?, json(?), ?, ?, ?)
[2021-07-12 11:46:15] peewee                         DEBUG   : ('INSERT INTO "event" ("id", "label", "camera", "start_time", "end_time", "top_score", "false_positive", "zones", "thumbnail", "has_clip", "has_snapshot") VALUES (?, ?, ?, ?, ?, ?, ?, json(?), ?, ?, ?)', ['1626086755.710832-p6iufx', 'person', 'back', 1626086755.710832, 1626086764.66152, 0.8046875, False, '[]', '****************** LOTS of DATA************=', False, True])
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Event received: end back 1626086755.930308-qzich6
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Event received: start back 1626086771.795203-qsbmar
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Cleaning up cached file back-20210712114422.mp4
[2021-07-12 11:46:15] frigate.events                 DEBUG   : Cleaning up cached file back-20210712114432.mp4
[2021-07-12 11:46:15] frigate.mqtt                   DEBUG   : Received mqtt message on frigate/back/person.
[2021-07-12 11:46:15] frigate.mqtt                   DEBUG   : Received mqtt message on frigate/back/person/snapshot.

Is there any way to configure frigate to use external USB mounted storage for recordings?

Hello together,
because of zmNinja - Support ends by end of 2021 i looked for alternatives…
Here i found Frigate.
I startet configuring frigate on my Unraidserver.
i’m using the frigate-nvidia docker container…
For some reasons i can’t get my Reolink RLC520 Camera’s to work properly…

Here is my Configuration:

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - cuda
    - -c:v 
    - h264_cuvid
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -c:v h264_cuvid
  output_args:
    detect: -f rawvideo -pix_fmt yuv420p -filter:v fps=fps=5
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    rtmp: -c copy -map 0 -f flv
      - -filter:v
      - fps=fps=5
mqtt:
  host: 10.0.0.65
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  stats_interval: 60
  
cameras:
  Front:
    ffmpeg:
      output_args:   
        rtmp: -c copy -map 0 -f flv
      inputs:
        - path: rtsp://admin:[email protected]/Streaming/Channels/1
          roles:
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1440
    fps: 5
    clips:
      enabled: false
      pre_capture: 5
      post_capture: 5
      objects:
        - person
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15
    motion:
      mask:
        - 0,0,1769,0,1885,0,1897,152,1669,165,1457,255,989,485,529,740,0,1083
    zones:  
      Vorgarten:
        coordinates: 0,1440,1737,1440,2560,1440,2560,0,1870,0,1878,163,1664,173,1555,219,1395,288,1156,411,788,609,704,650,63,1046,0,1079
  Garten:
    ffmpeg:
      global_args:
        - '-hide_banner'
        - '-loglevel'
        - info
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -rw_timeout
        - "5000000"
        - -use_wallclock_as_timestamps
        - "1"
      output_args:   
#        rtmp: -c:v libx264 -an -f flv -an
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy


      inputs:
        - path: rtmp://10.0.0.17/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=xxxx
          roles:
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1920
    fps: 5
    rtmp:
      enabled: True
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
      objects:
        - person
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15    
    motion:
      mask:
        - 1836,44,2441,346,2389,339,2378,404,2560,426,2560,0,0,0,0,347,113,368,79,293,563,54,1124,62,1598,73
    zones:  
      hinten_links:
        coordinates: 2560,1920,2560,418,2375,407,2378,347,2441,338,1824,49,1557,65,1125,52,1095,62,550,57,94,296,111,380,0,304,0,1920
  Gartenhaus:
    ffmpeg:
      global_args:
        - '-hide_banner'
        - '-loglevel'
        - info
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -rw_timeout
        - "5000000"
        - -use_wallclock_as_timestamps
        - "1"
      output_args:   
#        rtmp: -c:v libx264 -an -f flv -an
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy


      inputs:
        - path: rtmp://10.0.0.19/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=xxxx
          roles:
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1920
    fps: 5
    rtmp:
      enabled: True
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
      objects:
        - person
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15
    motion:
      mask:
        - 89,490,135,466,496,409,444,382,539,342,542,296,1446,32,1921,252,2132,409,2088,431,2086,623,2560,701,2560,0,0,0,0,509
    zones:  
      hinten_rechts:
        coordinates: 2560,1920,2560,1920,2560,705,2081,609,2096,415,2148,415,1918,249,1436,32,946,166,537,296,539,339,491,384,488,428,144,474,0,485,0,1920
    
objects:
  track:
    - person
    - cat
    - dog
    - car 
  filters:
    person:
      #min_area: 5000
      #max_area: 100000
      #min_score: 0.6
      threshold: 0.7
      
detectors:
  cpu1:
    type: cpu
  cpu2:
    type: cpu

First error ist the following…

frigate.video                  INFO    : Garten: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : Garten: ffmpeg process is not running. exiting capture thread...
ffmpeg.Garten.detect           ERROR   : Input #0, flv, from 'rtmp://10.0.0.17/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=27021986':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :   Duration: 00:00:00.00, start: 1626091836.851000, bitrate: N/A
ffmpeg.Garten.detect           ERROR   :     Stream #0:0: Data: none
ffmpeg.Garten.detect           ERROR   :     Stream #0:1: Video: h264, yuv420p(progressive), 2560x1920, 30 fps, 1k tbr, 1k tbn
ffmpeg.Garten.detect           ERROR   :     Stream #0:2: Audio: aac, 16000 Hz, mono, fltp
ffmpeg.Garten.detect           ERROR   : [segment @ 0x55d799e3f340] Opening '/tmp/cache/Garten-20210712141038.mp4' for writing
ffmpeg.Garten.detect           ERROR   : Output #0, segment, to '/tmp/cache/Garten-%Y%m%d%H%M%S.mp4':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :     encoder         : Lavf58.45.100
ffmpeg.Garten.detect           ERROR   :     Stream #0:0: Video: h264, yuv420p(progressive), 2560x1920, q=2-31, 30 fps, 1k tbr, 16k tbn, 1k tbc
ffmpeg.Garten.detect           ERROR   :     Stream #0:1: Audio: aac, 16000 Hz, mono, fltp
ffmpeg.Garten.detect           ERROR   : Output #1, flv, to 'rtmp://127.0.0.1/live/Garten':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :     encoder         : Lavf58.45.100
ffmpeg.Garten.detect           ERROR   :     Stream #1:0: Video: h264 ([7][0][0][0] / 0x0007), yuv420p(progressive), 2560x1920, q=2-31, 30 fps, 1k tbr, 1k tbn, 1k tbc
ffmpeg.Garten.detect           ERROR   :     Stream #1:1: Audio: aac ([10][0][0][0] / 0x000A), 16000 Hz, mono, fltp
ffmpeg.Garten.detect           ERROR   : Stream mapping:
ffmpeg.Garten.detect           ERROR   :   Stream #0:1 -> #0:0 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:2 -> #0:1 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:1 -> #1:0 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:2 -> #1:1 (copy)
ffmpeg.Garten.detect           ERROR   :   Stream #0:1 -> #2:0 (h264 (h264_cuvid) -> rawvideo (native))
ffmpeg.Garten.detect           ERROR   : Press [q] to stop, [?] for help
ffmpeg.Garten.detect           ERROR   : frame=    2 fps=0.0 q=-1.0 q=-1.0 q=0.0 size=N/A time=00:00:02.02 bitrate=N/A speed=3.57x
ffmpeg.Garten.detect           ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.Garten.detect           ERROR   : Finishing stream 2:0 without any data written to it.
ffmpeg.Garten.detect           ERROR   : Output #2, rawvideo, to 'pipe:':
ffmpeg.Garten.detect           ERROR   :   Metadata:
ffmpeg.Garten.detect           ERROR   :     displayWidth    : 2560
ffmpeg.Garten.detect           ERROR   :     displayHeight   : 1920
ffmpeg.Garten.detect           ERROR   :     encoder         : Lavf58.45.100
ffmpeg.Garten.detect           ERROR   :     Stream #2:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2560x1920 [SAR 1:1 DAR 4:3], q=2-31, 294912 kb/s, 5 fps, 5 tbn, 5 tbc
ffmpeg.Garten.detect           ERROR   :     Metadata:
ffmpeg.Garten.detect           ERROR   :       encoder         : Lavc58.91.100 rawvideo
ffmpeg.Garten.detect           ERROR   : [flv @ 0x55d799ed9700] Failed to update header with correct duration.
ffmpeg.Garten.detect           ERROR   : [flv @ 0x55d799ed9700] Failed to update header with correct filesize.
ffmpeg.Garten.detect           ERROR   : Error writing trailer of rtmp://127.0.0.1/live/Garten: Connection reset by peer
ffmpeg.Garten.detect           ERROR   : frame=    2 fps=0.0 q=-1.0 Lq=-1.0 q=0.0 size=N/A time=00:00:02.02 bitrate=N/A speed=3.39x
ffmpeg.Garten.detect           ERROR   : video:2409kB audio:4kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg.Garten.detect           ERROR   : Conversion failed!

But the Camera livestream is comming up correctly in frigate …
I have looked into the setting but the resolution is configured correctly…
Are there any errors in my config with RTMP?!

thanks in advance
chris

I am not sure what the difference between the Reolink RLC520 and my Reolink RLC-410-5MP is. However, I could only get it working reliably with RTMP not RTSP. Here is how I did my config if this helps:

  frontdriveway:
    ffmpeg:
      input_args: #Use for RTMP
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -rw_timeout
        - '5000000'
        - -use_wallclock_as_timestamps
        - '1'
      hwaccel_args:
        - -hwaccel
        - qsv
        - -qsv_device
        - /dev/dri/renderD128
      output_args:
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
      inputs:
        - path: rtmp://192.168.1.105/bcs/channel0_sub.bcs?channel=0&stream=1&user=USERNAME&password=PASSWORD
          roles:
            - detect
            - rtmp
            - record
        - path: rtmp://192.168.1.105/bcs/channel0_main.bcs?channel=0&stream=0&user=USERNAME&password=PASSWORD
          roles:
            - clips
    width: 640
    height: 480
    fps: 5
    record:
      enabled: true
      retain_days: 10
    rtmp:
      enabled: true
    detect:
      enabled: true
      max_disappeared: 60
    clips:
      enabled: true
      pre_capture: 10
      post_capture: 10
      objects:
        - person
      required_zones: []
      retain:
        default: 7
        objects:
          person: 30
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      track:
        - person

Nice i just played a bit with the resolution and set it to 2048x1536 and now it working properly…

Thanks

1 Like

Hi,
Yes i made a tutorial:
https://community.home-assistant.io/t/solved-hassos-mount-nas-network-share/

1 Like

I hate to see yellow font logs :slight_smile:

any fix/patch coming soon to take care of this?

Loving this integration… thank you!

Is there a way to enable/disable recording on a per-camera basis from Homeassistant? I see the switches for detect, clips and snapshots, but nothing similar for recording.

1 Like

As per docs

    # Optional: save clips configuration
    clips:

      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      #required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 14
        # Optional: Per object retention days
        #objects:
          #person: 15

    # Optional: 24/7 recording configuration
    record:
      # Optional: Enable recording (default: global setting)
      enabled: False
      # Optional: Number of days to retain (default: global setting)
      retain_days: 30

I am a longtime zoneminder/zmeventnotification user and I am really impressed with Frigate, really easy to setup!

You can only set width & height once per camera, and not per input, right? When using a lower resolution stream (640x360) for object detection, and a higher resolution (1920x1080) for recording/clips/rtmp, what width & height should be used in that case?
If using two inputs for a camera, can you select which of them is displayed in the web ui? Currently my low resolution stream (640x360) is the primary in the webui.