Local realtime person detection for RTSP cameras

Share your Projects!
2284

With version 0.7.3 my cameras in nightvison mode would not start ffmpeg anymore:

No frames received from garage in 5 seconds. Exiting ffmpeg...
Waiting for ffmpeg to exit gracefully...
garage: ffmpeg sent a broken frame. something is wrong.
garage: ffmpeg sent a broken frame. something is wrong.
garage: ffmpeg process is not running. exiting capture thread...

With version 0.7.1, ffmpeg starting fine in nightvision mode.

In /debug/stats I can see that in nightvison mode the cameras sending 2.5 frames instead 5 fps.
All cameras configured to send a stream with 5 fps.

global ffmpeg settings:

  input_args:
    - -avoid_negative_ts
    - make_zero
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags
    - +genpts+discardcorrupt
    - -r
    - '5'
    - -rtsp_transport
    - tcp
    - -stimeout
    - '5000000'
    - -use_wallclock_as_timestamps
    - '1'

camera settings:

  garage:
    ffmpeg:
      input: rtsp://user:[email protected]:554/11
    mask: poly,0,0,0,35,1920,35,1920,0
    take_frame: 1
    fps: 5  # tried 5 fps, 2.5 fps and complete disabled this option

Workaround would be to set all camera to 10 fps. But this would also increase unnecessary the cpu usage and load of the system.

Edit: Set all cameras to 7 fps and it works again. Now in nightvision mode camera is sending 5 fps.

2285

Can you enable ffmpeg logging for one of your cameras?

2286

@blakeblackshear

Input #0, rtsp, from 'rtsp://user:[email protected]/11':
  Metadata:
    title           : 10
  Duration: N/A, start: 1603800749.929589, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080, 2.50 tbr, 90k tbn, 180k tbc


[segment @ 0x55764e398b40] Opening '/cache/garage-20201027131238.mp4' for writing
Output #0, segment, to '/cache/garage-%Y%m%d%H%M%S.mp4':
  Metadata:
    title           : 10
    encoder         : Lavf58.45.100
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080, q=2-31, 2.50 tbr, 90k tbn, 90k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help


No frames received from garage in 5 seconds. Exiting ffmpeg...
Waiting for ffmpeg to exit gracefully...

Finishing stream 1:0 without any data written to it.
[swscaler @ 0x55764e835200] deprecated pixel format used, make sure you did set range correctly
Output #1, rawvideo, to 'pipe:':
  Metadata:
    title           : 10
    encoder         : Lavf58.45.100
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080, q=2-31, 124416 kb/s, 5 fps, 5 tbn, 5 tbc
    Metadata:
      encoder         : Lavc58.91.100 rawvideo
frame=    0 fps=0.0 q=-1.0 Lq=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=   0x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 15.
garage: ffmpeg sent a broken frame. something is wrong.
garage: ffmpeg process is not running. exiting capture thread...
2287

@NewToThis That looks suitable to me.

I should get my Dual TPU on Fri @rpress so I will keep you informed if I get get both TPUā€™s working in the Key-E Slot.

1 Like
2288

I donā€™t see anything obvious.

2289

hi,

i have masked out the area outside the box shown on below, but the white car far back still being detected. any idea? thanks

image
image1298Ɨ729 628 KB

  frontdoor_01:
    ffmpeg:
      input: rtsp://adm********
    fps: 6
    take_frame: 3
    mask: 'poly,874,1294,9,1294,3,0,2303,0,2303,1294,1957,1294,2059,686,2224,122,1829,59,1391,55,1102,79,325,806'
2290

The mask only applies to motion. The white car is being picked up because it is in the background of the detection region for the black car.

2291

so you mean the white car is in the black carā€™s green box? this is why the white car is detected?

2292

also, i noticed that the timestamp is not showing anymore in /latest.jpg and /best.jpg even i have enable the option in snapshots. Possible to reenable this function? thanks.

2293

Thanks, ordered

2294

Yes. Now that frigate has seen it, it will continue to be tracked until it goes away regardless of the mask. Also, the timestamps will be returning in a future version.

2295

got it! thanks!

2296

I turned on INFO and this seems to be an entire block of the relevant message

Creating ffmpeg process... size= 1831612kB time=00:02:00.60 bitrate=124416.0kbits/s dup=29 drop=472 speed=1.01x
ffmpeg -hide_banner -loglevel info -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.6:7447/lIRVfkMALJNwaNXv -r 5 -f rawvideo -pix_fmt yuv420p pipe:
[rtsp @ 0xaaab0ae2e6e0] DTS discontinuity in stream 0: packet 3 with DTS 76981890887679, packet 4 with DTS 76981890952006
[rtsp @ 0xaaab0ae2e6e0] DTS discontinuity in stream 0: packet 7 with DTS 76981890952023, packet 8 with DTS 76981890964857
Input #0, rtsp, from 'rtsp://192.168.1.6:7447/lIRVfkMALJNwaNXv':ate=373248.0kbits/s dup=979 drop=0 speed=1.01x
  Metadata:
    title           : B4FBE405C158_0
  Duration: N/A, start: 1603789393.493000, bitrate: N/A
    Stream #0:0: Audio: aac, 48000 Hz, mono, fltp
    Stream #0:1: Video: h264, yuvj420p(pc, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 24 fps, 14.92 tbr, 90k tbn, 48 tbc
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
More than 1000 frames duplicated 5701388kB time=00:02:05.13 bitrate=373248.0kbits/s dup=999 drop=0 speed=1.01x
No frames received from right in 5 seconds. Exiting ffmpeg...itrate=373248.0kbits/s dup=1038 drop=0 speed=1.01x
Waiting for ffmpeg to exit gracefully...
Finishing stream 0:0 without any data written to it.
[swscaler @ 0xaaab0d3c4f50] deprecated pixel format used, make sure you did set range correctly
Output #0, rawvideo, to 'pipe:':
  Metadata:
    title           : B4FBE405C158_0
    encoder         : Lavf58.45.100
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 497664 kb/s, 5 fps, 5 tbn, 5 tbc
    Metadata:
      encoder         : Lavc58.91.100 rawvideo
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.00 bitrate=N/A speed=   0x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Exiting normally, received signal 15.
right: ffmpeg sent a broken frame. something is wrong.
right: ffmpeg process is not running. exiting capture thread...
[h264 @ 0xaaab128eb8b0] error while decoding MB 81 33, bytestream -1424416.0kbits/s dup=27 drop=577 speed=1.01x
rtsp://192.168.1.6:7447/nMYzCJLFZdjQT6dH: corrupt decoded frame in stream 1
[rtsp @ 0xaaaae93606e0] CSeq 9 expected, 8 received.2:18.00 bitrate=124416.0kbits/s dup=27 drop=585 speed=1.01x
[h264 @ 0xaaaae9381900] error while decoding MB 114 49, bytestream -103248.0kbits/s dup=1095 drop=0 speed=1.01x
rtsp://192.168.1.6:7447/88Re44KwNH1yBMQz: corrupt decoded frame in stream 1
[rtsp @ 0xaaaad96b96e0] CSeq 9 expected, 8 received.2:20.80 bitrate=124416.0kbits/s dup=33 drop=549 speed=1.01x
Creating ffmpeg process... size= 2138400kB time=00:02:20.80 bitrate=124416.0kbits/s dup=27 drop=599 speed=1.01x

Iā€™m not sure which bit of that is a helpful error. I managed to establish that the swscaler message is irrelevant.

2297

It seems like ffmpeg is struggling to get valid packets from your camera. Is the network reliable between the devices? How many open connections do you have to the camera?

2298

The camera(s) are Unifi cameras. I only have one in this config for the purposes of testing this. I also have an i5 NUC with a Coral USB accelerator and Frigate works fine on that with all the cameras. Iā€™ve shut down Frigate on the i5 down so the only other connection to this camera is (should be) the Unifi NVR. The network is Unifi kit as well and although the camera and the Coral Dev Board are on different switches (1) theyā€™re connected with 10Gb fibre and (2) I get the same problem if I use the ā€˜porchā€™ camera which is on the same switch.

It sounds like the non-TPU bits of the board canā€™t cope with the workload.

2299

Hi Blake

Loving the latest features and the new documentation is superb - thanks.

Iā€™ve been running the frigate:0.7.1-armv7 image just fine. However, with 0.7.2-armv7 & 0.7.3 I get the following error. Tried removing hwaccel_args override, no difference.

2020-10-28T13:29:08.259087117Z Fontconfig error: Cannot load default config file
2020-10-28T13:29:09.477380596Z On connect called
2020-10-28T13:29:09.516742666Z Starting detection process: 15
2020-10-28T13:29:09.516825886Z Attempting to load TPU as usb
2020-10-28T13:29:09.530470931Z Camera capture process started for front: 16
2020-10-28T13:29:09.532425728Z Camera process started for front: 18
2020-10-28T13:29:09.556657281Z Creating ffmpeg process...
2020-10-28T13:29:09.556736798Z ffmpeg -hide_banner -loglevel info -c:v h264_mmal -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://xxx:1935/bcs/channel0_sub.bcs?channel=0&stream=0&user=xxx&password=xxx -f rawvideo -pix_fmt yuv420p pipe:
2020-10-28T13:29:09.599715727Z  * Serving Flask app "detect_objects" (lazy loading)
2020-10-28T13:29:09.600476472Z  * Environment: development
2020-10-28T13:29:09.600544137Z  * Debug mode: off
2020-10-28T13:29:12.291597904Z TPU found
2020-10-28T13:29:19.532699511Z Input #0, flv, from 'rtmp://xxx:1935/bcs/channel0_sub.bcs?channel=0&stream=0&user=xx&password=xxx':
2020-10-28T13:29:19.532841268Z   Metadata:
2020-10-28T13:29:19.532900359Z     displayWidth    : 640
2020-10-28T13:29:19.532956636Z     displayHeight   : 480
2020-10-28T13:29:19.533010894Z   Duration: 00:00:00.00, start: 1603891753.604000, bitrate: N/A
2020-10-28T13:29:19.533069986Z     Stream #0:0: Data: none
2020-10-28T13:29:19.534759936Z     Stream #0:1: Audio: aac, 16000 Hz, mono, fltp
2020-10-28T13:29:19.534882952Z     Stream #0:2: Video: h264, yuv420p(progressive), 640x480, 6 fps, 1k tbr, 1k tbn
2020-10-28T13:29:19.604051861Z Stream mapping:
2020-10-28T13:29:19.604511371Z   Stream #0:2 -> #0:0 (h264 (h264_mmal) -> rawvideo (native))
2020-10-28T13:29:19.604850162Z Press [q] to stop, [?] for help
2020-10-28T13:29:19.612088658Z No frames received from front in 5 seconds. Exiting ffmpeg...
2020-10-28T13:29:19.612421207Z Waiting for ffmpeg to exit gracefully...
2020-10-28T13:29:19.776976264Z [h264_mmal @ 0x23dfc60] Did not get output frame from MMAL.
2020-10-28T13:29:19.777127650Z Error while decoding stream #0:2: Unknown error occurred
2020-10-28T13:29:19.777192316Z Finishing stream 0:0 without any data written to it.
2020-10-28T13:29:19.780331387Z Output #0, rawvideo, to 'pipe:':
2020-10-28T13:29:19.780457292Z   Metadata:
2020-10-28T13:29:19.780892377Z     displayWidth    : 640
2020-10-28T13:29:19.781422164Z     displayHeight   : 480
2020-10-28T13:29:19.781933062Z     encoder         : Lavf58.45.100
2020-10-28T13:29:19.782553476Z     Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480, q=2-31, 22118 kb/s, 6 fps, 6 tbn, 6 tbc
2020-10-28T13:29:19.783017819Z     Metadata:
2020-10-28T13:29:19.783155632Z       encoder         : Lavc58.91.100 rawvideo
2020-10-28T13:29:19.783491922Z frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
2020-10-28T13:29:19.783684085Z video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
2020-10-28T13:29:19.783874915Z Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
2020-10-28T13:29:19.790963988Z Exiting normally, received signal 15.
2020-10-28T13:29:19.798857897Z front: ffmpeg sent a broken frame. something is wrong.
2020-10-28T13:29:19.799729455Z front: ffmpeg process is not running. exiting capture thread...

web_port: 5000

detectors:
  coral:
    type: edgetpu
    device: usb

mqtt:
  host: xxx
  port: 1883
  topic_prefix: frigate
  client_id: frigate

save_clips:
  max_seconds: 300
  clips_dir: /clips
  cache_dir: /cache

ffmpeg:
  global_args:
    - -hide_banner
    - -loglevel
    - info
  hwaccel_args:
    - -c:v
    - h264_mmal
  input_args:
    - -avoid_negative_ts
    - make_zero
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags
    - +genpts+discardcorrupt
    - -use_wallclock_as_timestamps
    - '1'

objects:
  track:
    - person
  filters:
    person:
      min_area: 5000
      max_area: 100000
      min_score: 0.5
      threshold: 0.85

cameras:
  front:
    ffmpeg:
      input: rtmp://xxx:1935/bcs/channel0_sub.bcs?channel=0&stream=0&user=xxx&password={FRIGATE_RTSP_PASSWORD}
    height: 480
    width: 640
    best_image_timeout: 60

    mqtt:
      crop_to_region: True
      snapshot_height: 300

    save_clips:
      enabled: False
      pre_capture: 30
      objects:
        - person      

    snapshots:
      show_timestamp: True
      draw_zones: False
      draw_bounding_boxes: True

    objects:
      track:
        - person
      filters:
        person:
          min_area: 5000
          max_area: 100000
          min_score: 0.5
          threshold: 0.85
2300

It seems like the changes after 0.7.1 introduced several unexpected issues. I think I am going to revert some changes and release 0.7.4.

1 Like
2301

I just started using this today. I have a stream and see files in the cache. But seems it doesnā€™t detect persons or record clips.

Should i wait for 0.7.4 and try again? Or what other steps can i take to find the reason?

Did run this with debug enabled and got this:

ffmpeg -hide_banner -loglevel info -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://xxx:[email protected]:554/h264 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an -map 0 /config/cache/voordeur-%Y%m%d%H%M%S.mp4 -r 2.5 -f rawvideo -pix_fmt yuv420p pipe:
* Serving Flask app "detect_objects" (lazy loading)
* Environment: development
* Debug mode: off
Guessed Channel Layout for Input Stream #0.1 : mono
Input #0, rtsp, from 'rtsp://xxx:[email protected]:554/h264':
Metadata:
title : RTSP/RTP stream from anjvision ipcamera
Duration: N/A, start: 1603900855.406625, bitrate: N/A
Stream #0:0: Video: h264, yuvj420p(pc, progressive), 2592x1944, 90k tbr, 90k tbn, 180k tbc
Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s
[segment @ 0x563007af0cc0] Opening '/config/cache/voordeur-20201028170056.mp4' for writing
Output #0, segment, to '/config/cache/voordeur-%Y%m%d%H%M%S.mp4':
Metadata:
title : RTSP/RTP stream from anjvision ipcamera
encoder : Lavf58.45.100
Stream #0:0: Video: h264, yuvj420p(pc, progressive), 2592x1944, q=2-31, 90k tbr, 90k tbn, 90k tbc
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #1, rawvideo, to 'pipe:':
Metadata:
title : RTSP/RTP stream from anjvision ipcamera
encoder : Lavf58.45.100
Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2592x1944, q=2-31, 151165 kb/s, 2.50 fps, 2.50 tbn, 2.50 tbc
Metadata:
encoder : Lavc58.91.100 rawvideo
2302

I would suggest trying 0.7.1 to avoid any issues possibly introduced in 0.7.2 and 0.7.3.

2303

same issue so it seemsā€¦ i see items in cache but seems not to detect persons