Local realtime person detection for RTSP cameras

Hi Blake, yes sorry I meant HassOS. No problem, I can’t figure it out yet but will keep working on it in the background.

I currently use MotionEye in addition to Frigate to record clips. MotionEye is triggered manually to avoid duplicating any detection processing but I find MotionEye still adds a (hopefully) unnecessary overhead of CPU use. I’d like to move to using Frigate to record clips of all events. I’m happy to manually manage archiving/removal of these for now, but am very excited about you future development in this area!

Unfortunately after some effort today I can’t seem to get any clips to record. Frigate is doing everything else correctly, and I think I have all the required config settings enabled, but no clips appear in the output directory. Could you check if I have done anything wrong? Config and log output below:

web_port: 5000
detectors:
  coral:
    type: edgetpu
    device: 'usb:0'
save_clips:
  clips_dir: /media/frigate
  cache_dir: /cache
  max_seconds: 300
mqtt:
  host: addon_core_mosquitto
  topic_prefix: frigate
  user: *******
  password: ******
ffmpeg: {}
cameras:
  garage_side:
    ffmpeg:
      input: 'rtsp://*********@192.168.0.133:554/12'
      take_frame: 2
      snapshots:
        show_timestamp: false
        draw_bounding_boxes: true
      save_clips:
        enabled: true
        pre_capture: 30
        objects:
          - person
      objects:
        track:
          - person
        filters:
          person:
            min_area: 10
            max_area: 100000
            min_score: 0.5
            threshold: 0.8
  garage_internal:
    ffmpeg:
      input: 'rtsp://********@192.168.0.89:554/12'
      take_frame: 2
      snapshots:
        show_timestamp: false
        draw_bounding_boxes: true
      save_clips:
        enabled: true
        pre_capture: 30
        objects:
          - person
      objects:
        track:
          - person
        filters:
          person:
            min_area: 10
            max_area: 100000
            min_score: 0.5
            threshold: 0.8
objects:
  track:
    - person
  filters:
    person:
      min_area: 10
      max_area: 100000
      min_score: 0.5
      threshold: 0.8
Fontconfig error: Cannot load default config file
ffprobe -v panic -show_error -show_streams -of json "rtsp://********@192.168.0.133:554/12"
Starting detection process: 16
Attempting to load TPU as usb:0
No EdgeTPU detected. Falling back to CPU.
On connect called
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'unknown', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 640, 'height': 352, 'coded_width': 640, 'coded_height': 352, 'closed_captions': 0, 'has_b_frames': 0, 'level': -99, 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'pcm_alaw', 'codec_long_name': 'unknown', 'codec_type': 'audio', 'codec_time_base': '1/8000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 's16', 'sample_rate': '8000', 'channels': 1, 'bits_per_sample': 8, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/8000', 'bit_rate': '64000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://********@192.168.0.133:554/12 -f rawvideo -pix_fmt yuv420p pipe:
ffprobe -v panic -show_error -show_streams -of json "rtsp://********@192.168.0.89:554/12"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'unknown', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 640, 'height': 352, 'coded_width': 640, 'coded_height': 352, 'closed_captions': 0, 'has_b_frames': 0, 'level': -99, 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'pcm_alaw', 'codec_long_name': 'unknown', 'codec_type': 'audio', 'codec_time_base': '1/8000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 's16', 'sample_rate': '8000', 'channels': 1, 'bits_per_sample': 8, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/8000', 'bit_rate': '64000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://********@192.168.0.89:554/12 -f rawvideo -pix_fmt yuv420p pipe:
Camera_process started for garage_side: 36
Starting process for garage_side: 36Camera_process started for garage_internal: 37
Starting process for garage_internal: 37
 * Serving Flask app "detect_objects" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off

I see some of your config is indented incorrectly. I really need to get schema validation implemented.

Try updating to this:

web_port: 5000
detectors:
  coral:
    type: edgetpu
    device: 'usb:0'
save_clips:
  clips_dir: /media/frigate
  cache_dir: /cache
  max_seconds: 300
mqtt:
  host: addon_core_mosquitto
  topic_prefix: frigate
  user: *******
  password: ******
ffmpeg: {}
cameras:
  garage_side:
    ffmpeg:
      input: 'rtsp://*********@192.168.0.133:554/12'
    take_frame: 2
    snapshots:
      show_timestamp: false
      draw_bounding_boxes: true
    save_clips:
      enabled: true
      pre_capture: 30
      objects:
        - person
    objects:
      track:
        - person
      filters:
        person:
          min_area: 10
          max_area: 100000
          min_score: 0.5
          threshold: 0.8
  garage_internal:
    ffmpeg:
      input: 'rtsp://********@192.168.0.89:554/12'
    take_frame: 2
    snapshots:
      show_timestamp: false
      draw_bounding_boxes: true
    save_clips:
      enabled: true
      pre_capture: 30
      objects:
        - person
    objects:
      track:
        - person
      filters:
        person:
          min_area: 10
          max_area: 100000
          min_score: 0.5
          threshold: 0.8
objects:
  track:
    - person
  filters:
    person:
      min_area: 10
      max_area: 100000
      min_score: 0.5
      threshold: 0.8

Great spot! That’s fixed it and I’m now getting clips recorded perfectly!

Thanks again for all the hard work!

Edit: Removing MotionEye brings my CPU use on the RPi4 from ~40% to 15% which is awesome! Keeps my Pi nice and cool!

Everything is working great except the clips are not being saved. Is this config correct for the hassio addon?

web_port: 5000
detectors:
  coral:
    type: edgetpu
    device: 'usb:0'
save_clips:
  enabled: true
  pre_capture: 30
  objects:
    - person
    - car
    - truck
    - bicycle
    - motorcycle
  clips_dir: /media/frigate/clips
  cache_dir: /media/frigate/cache
mqtt:
  host: 10.0.0.10
  topic_prefix: frigate
  user: mqtt
  password: 
ffmpeg: {}
cameras:
  drive:
    ffmpeg:
      input: >-
        rtmp://10.0.0.87/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=
      input_args:
        - '-avoid_negative_ts'
        - make_zero
        - '-fflags'
        - nobuffer
        - '-flags'
        - low_delay
        - '-strict'
        - experimental
        - '-fflags'
        - +genpts+discardcorrupt
        - '-vsync'
        - drop
        - '-use_wallclock_as_timestamps'
        - '1'
      take_frame: 2
      snapshots:
        show_timestamp: true
        draw_bounding_boxes: true
      save_clips:
        enabled: true
        pre_capture: 30
        objects:
          - person
          - car
          - truck
          - bicycle
          - motorcycle
        clips_dir: /media/frigate/clips
        cache_dir: /media/frigate/cache
objects:
  track:
    - person
    - car
    - bicycle
    - truck
    - motorcycle
  filters:
    person:
      min_area: 1
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    car:
      min_area: 110000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    bicycle:
      min_area: 20000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    truck:
      min_area: 20000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    motorcycle:
      min_area: 20000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7

Also if I black out part of the detection region if a person is half in the black and half in the white will they be detected?

Thank you!

@dave122, Try this:

There were indentation issues from take_frame to save_clips

web_port: 5000
detectors:
  coral:
    type: edgetpu
    device: 'usb:0'
save_clips:
  enabled: true
  pre_capture: 30
  objects:
    - person
    - car
    - truck
    - bicycle
    - motorcycle
  clips_dir: /media/frigate/clips
  cache_dir: /media/frigate/cache
mqtt:
  host: 10.0.0.10
  topic_prefix: frigate
  user: mqtt
  # password: 
ffmpeg: {}
cameras:
  drive:
    ffmpeg:
      input: >-
        rtmp://10.0.0.87/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=
      input_args:
        - '-avoid_negative_ts'
        - make_zero
        - '-fflags'
        - nobuffer
        - '-flags'
        - low_delay
        - '-strict'
        - experimental
        - '-fflags'
        - +genpts+discardcorrupt
        - '-vsync'
        - drop
        - '-use_wallclock_as_timestamps'
        - '1'
    take_frame: 2
    snapshots:
      show_timestamp: true
      draw_bounding_boxes: true
    save_clips:
      enabled: true
      pre_capture: 30
      objects:
        - person
        - car
        - truck
        - bicycle
        - motorcycle
objects:
  track:
    - person
    - car
    - bicycle
    - truck
    - motorcycle
  filters:
    person:
      min_area: 1
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    car:
      min_area: 110000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    bicycle:
      min_area: 20000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    truck:
      min_area: 20000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
    motorcycle:
      min_area: 20000
      max_area: 1000000
      min_score: 0.1
      threshold: 0.7
1 Like

Thanks! It accepts it but I get a error loop:

Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://10.0.0.87/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password= -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an -map 0 /media/frigate/cache/drive-%Y%m%d%H%M%S.mp4 -f rawvideo -pix_fmt yuv420p pipe:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5601cb34e140] moov atom not found
/media/frigate/cache/drive-20201021150011.mp4: Invalid data found when processing input
bad file: drive-20201021150011.mp4
drive: ffmpeg sent a broken frame. something is wrong.
drive: ffmpeg process is not running. exiting capture thread...

I don’t know what that is. I’ve seen it myself a few times. A restart of the addon “fixes” this in my case
I haven’t been able to reproduce this

1 Like

Take out below args from your input args:

 - -vsync
 - drop

You can’t have this option if you want to save clips

1 Like

Thanks.

If I remove those flags I still get the same error loop, however if I remove the save clips section from under the camera section and put it on its own then it gives the below error but seems to carry on. I’m not at home so Ill have to wait to see if it actually records.

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e14166e140] moov atom not found
/media/frigate/cache/drive-20201021151547.mp4: Invalid data found when processing input
bad file: drive-20201021151547.mp4

Edit: no luck, doesn’t record.

I don’t have any inputargs at all, completely removed the section. saving clips works like a charm.

  hwaccel_args:
    - '-hwaccel'
    - vaapi
    - '-hwaccel_device'
    - /dev/dri/renderD128
    - '-hwaccel_output_format'
    - yuv420p

I also noticed in your config that the following lines can be removed. They have no effect

save_clips:
  # enabled: true
  # pre_capture: 30
  # objects:
  #   - person
  #   - car
  #   - truck
  #   - bicycle
  #   - motorcycle
  clips_dir: /media/frigate/clips
  cache_dir: /media/frigate/cache
1 Like

If I remove the input_args it doesn’t run, I think they are needed for RTMP. I think I’ll use yaml to get motioneye to record if I can’t sus it out.

I have noticed the same, with rtmp you have to leave the
input_args

1 Like

Do clips save for you with the input args there?

I don’t use the save_clips function, I use HA’s camera.record service

1 Like

@dave122, I too have the same issue since upgrading to 0.7.1. (version pre this work fine).
I run 3 x Reolink cameras using RTMP and 1 x Hikvision.
The Hikvision works fine with save_clips, if I enable save_clips for the Reolinks the ffmpeg process keeps crashing.
If I comment out the save_clips for the Reolinks, Frigate works as expected (apart from not saving clips for the Reolinks of course).

1 Like

@dave122 since you are using RTMP, are you using Reolink Cameras?

@asve99 I just found out that reolink cameras stream is not compatible with Frigate ffmpeg save_clips arguments. It had to do with one of the ffmpeg output arguments, -map 0 to be specific. For reolink cameras, the argument for reolink video stream have to be -map 0:2.

I exec into frigate container to edit detect_objects.py file to change the -map argument from 0 to 0:2 and the recording works for all my reolink cameras.

For your case, its not so straight forward since you have hikvision and reolink cameras. You will have to add in some test condition to construct the correct ffmpeg command for different brands of cameras. Or wait for @blakeblackshear to add a feature to override save_clips output arguments.

Refer this link for more info

4 Likes

Correct, I am using a RLC-410W and a E1 Pro. The E1 pro annoyingly only has rtsp which I haven’t been able to get to work with frigate yet.

I read about e1 pro before and the specs are good. if I remember correctly, I didn’t buy it because it doesn’t have rtsp that’s why I bought c1 instead.

Since you’re using reolink u can either disable save clips first or hack frigate to work for reolink cams. Refer the link I shared for more info.

I have reduced my 3 streams to 720p@5fps CBR 1500kbps, and CPU is now on avg at 10% load on my LXC.

Hi,

Does Frigate supports this device ?
https://www.mouser.fr/ProductDetail/212-G650-06076-01

Kind regards,

1 Like