Local realtime person detection for RTSP cameras

thundergreen · April 5, 2021, 2:01pm

Can someone explain to me why my video is shown with 5 pics /s? this si my config:
running on intel NUC i5 with coral but the camera stream is NEVER fluent If I activate the stream compoennt i get the whole speed BUT delayed like hell.

detectors:
  coral:
    type: edgetpu
    device: pci
mqtt:
  host: 192.168.1.41
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: hass
  password: hass
ffmpeg:
  global_args:
  - "-hide_banner"
  - "-loglevel"
  - panic
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
  input_args:
    - -avoid_negative_ts
    - make_zero
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags
    - +genpts+discardcorrupt
    - -use_wallclock_as_timestamps
    - '1'
  output_args:
    # Optional: output args for detect streams (default: shown below)
    detect: -f rawvideo -pix_fmt yuv420p
    # Optional: output args for record streams (default: shown below)
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    # Optional: output args for clips streams (default: shown below)
    clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    # Optional: output args for rtmp streams (default: shown below)
    rtmp: -c copy -f flv
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
#    - cat
#    - dog
    - car
    - bicycle
#    - bird
cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://hass:[email protected]:88/videoMain
          roles:
            - detect
            - rtmp
            - clips
            - record
    width: 1280
    height: 720
    fps: 23
    record:
      enabled: true
      retain_days: 7
    # Optional: Configuration for the jpg snapshots written to the clips directory for each event
    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: true
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: true
      # Optional: crop the snapshot (default: shown below)
      crop: False
      # Optional: height to resize the snapshot to (default: original size)
#      height: 175
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7
    zones:
      parking:
        coordinates: '554,333,611,274,708,281,763,265,823,260,948,285,1021,341,1078,359,1102,387,1137,381,1154,417,1120,596,760,504,754,318'
      mailbox:
        coordinates: '416,454,434,401,387,346,336,395,337,446'
      main_door:
        coordinates: '1200,697,1159,699,1071,699,936,695,553,663,530,449,1280,420'
  porch:
    ffmpeg:
      inputs:
        - path: rtsp://hassio:[email protected]:554/stream1
          roles:
            - detect
            - rtmp
            - clips
            - record
    width: 1920
    height: 1080
    fps: 23
    record:
      enabled: true
      retain_days: 7
    # Optional: Configuration for the jpg snapshots written to the clips directory for each event
    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: true
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: true
      # Optional: crop the snapshot (default: shown below)
      crop: False
      # Optional: height to resize the snapshot to (default: original size)
#      height: 175
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7
    zones:
      door_porch_dining_room:
        coordinates: '1243,1080,1292,85,961,162,929,1080'


  garage:
    ffmpeg:
      inputs:
        - path: rtsp://40.40.40.220:554/media/stream1
          roles:
            - detect
            - rtmp
            - clips
            - record
    width: 1280
    height: 720
    fps: 23
    record:
      enabled: true
      retain_days: 7
    zones:
      garage_door:
        coordinates: 0,158,0,504,383,493,397,213
      refrigiator:
        coordinates: '1186,259,1159,665,1038,646,1070,259'
      table:
        coordinates: '269,486,0,486,0,720,142,720,592,720,256,482,145,720,376,720,588,720'
      window_front_house:
        coordinates: '599,265,605,419,507,418,514,250'
    # Optional: Configuration for the jpg snapshots written to the clips directory for each event
    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: true
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: true
      # Optional: crop the snapshot (default: shown below)
      crop: False
      # Optional: height to resize the snapshot to (default: original size)
#      height: 175
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7
  garden:
    ffmpeg:
      inputs:
        - path: rtsp://40.40.40.14:554/media/stream1
          roles:
            - detect
            - rtmp
            - clips
            - record
    width: 1280
    height: 720
    fps: 23
    record:
      enabled: true
      retain_days: 7
    # Optional: Configuration for the jpg snapshots written to the clips directory for each event
    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: true
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: true
      # Optional: crop the snapshot (default: shown below)
      crop: False
      # Optional: height to resize the snapshot to (default: original size)
#      height: 175
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: true
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 7
        # Optional: Per object retention days
        objects:
          person: 7

hawkeye217 · April 5, 2021, 2:15pm

The Frigate UI is for debugging, not for viewing full quality, max fps streams from your cameras - 5fps is hard-coded for the UI.

The fps: setting in your config file is applicable for the detect role only, and anything more than 5fps is only utilizing more system resources without increasing detection accuracy.

The stream component does introduce delays in the stream. There are a few things you can do to mitigate it, but the best way to look at your camera streams real-time at this point is still through directly connecting to the RTSP stream via VLC or another media player rather than a browser.

deputyd0ng · April 5, 2021, 2:15pm

Does someone have an example yaml of a working config w/ reolink? Im trying to follow the guid ebut ym syntax is wrong somewhere, keep getting broken frame, then the container exits. Running on unraid

mqtt:
  host: 192.168.86.****
  port: 1883
  user: mqtt1
  password: *********
cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtmp://192.168.xx.xx/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=PASSWORD
          roles:
            - detect
            - rtmp
    width: 2560
    height: 1920
    fps: 5

ffmpeg: 
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps 1
  hwaccel_args: 
    - "-hwaccel"
    - vaapi
    - "-hwaccel_device"
    - /dev/dri/renderD128
    - "-hwaccel_output_format"
    - yuv420p

hawkeye217 · April 5, 2021, 2:17pm

Have you checked the docs? It has a Reolink example.

thundergreen · April 5, 2021, 2:27pm

That’s a pity somehow. Would like to see my stream also in the front end without delay. I gues there’s no way to make it fluent without a streaming component right? Well , then I stick to the current 5 fps solution as it seems the most accurate with the ledd delay possible. Thanks to clarify that

hawkeye217 · April 5, 2021, 2:33pm

The limitation is really in Home Assistant and the way streaming is done, not directly with Frigate. Check out this thread for a deeper dive.

deputyd0ng · April 5, 2021, 2:34pm

yes, as you can see I have the arguments from that doc in my config, however, I am still getting broken frames. This is why I believe it is a syntax issue on my part.

I also have these cams in blue iris, is it possible to use the blue iris API url?

hawkeye217 · April 5, 2021, 2:38pm

You might want to start by removing the hwaccel args and seeing if your problem goes away. You can also enable ffmpeg debugging per this post.

mcpaha · April 5, 2021, 2:41pm

With docker Frigate & Frigate integration -> is it possible to get frigate link to the side panel (like when using addon)?

hawkeye217 · April 5, 2021, 2:48pm

You could try to add an iframe panel with your Frigate url (eg. http(s)://[your frigate ip]:5000)

birca1987 · April 5, 2021, 2:54pm

After you install Frigate Integration from HACS, restart homeassistant and go to Integrations in HA and click Add, search for Frigate, it should be in the list.

Sorb · April 5, 2021, 3:01pm

I’m still struggeling with getting the Reolink Lumus Cameras to work via Neolink “translator” for the RTSP protocol.
Can someone give me an example how to properly specify the output parameters for ffmpeg when using inside HA? Will those parameters show up in the debug info I can get from the UI? Tried to specify them in different ways, but they never show up in the debug view.
Is there an option to retrieve the full Frigate Logfile from the filesystem? Not just the part of the log that is displayed in the UI.

Ryan_Grandy · April 5, 2021, 3:49pm

Does anyone know if there is a program which will assemble all of the 1 minute recording videos from frigate into a long stream that I can fast forward through? Blue Iris 24/7 recording does this well but I don’t have a solution to do this as well in frigate, so many little clips is very tedious to navigate through to try to find a short event when you don’t know when it happened.

deputyd0ng · April 5, 2021, 4:17pm

OK, I got the RTMP stream working for my sub stream, but do I have to add the main stream as a different camera? They obviously have different heights and widths and FPS, as well as different roles

Do I need to do something like front_door_sub and fill that out and another one named front_door_main?

hawkeye217 · April 5, 2021, 4:33pm

Have a look at the example. You can add multiple camera streams as inputs and specify the role you want for each input.

deputyd0ng · April 5, 2021, 5:12pm

yes, thats where I am basing all of my work on, trust me, Id much rather have this working on my own instead of asking for help on the forum.

however, my substream has a different height/width/fps from my main stream, when inputting two paths like the example, and input a height/width/fps for either one of my streams, I get a broken frame error.

If I remove one input for example the main stream, and give it the correct width/height/fps it works, however, I want to set it up as in the example with each cam having a sub/main stream.

hawkeye217 · April 5, 2021, 5:31pm

The fps and resolution only applies to the detect input. Everything else is pass-through from the camera. Check the comments in the full example config.

Also, if you’re getting a broken frame error, you can enable debugging to see exactly what’s going wrong.

Minglarn · April 5, 2021, 6:44pm

Well, i’ve been using it on my ZoneMinder setup with various results.
I finally made shut down on that system and handed over the surveillance work to Frigate!
With that said… i now have a free Coral laying around.

Does a second Coral have any benefit to Frigate?

darknebula · April 5, 2021, 7:14pm

I’ve just had a look through the documentation again and couldn’t find it. Is it possible to change the date format to day/month/year ? Also how do i show the media browser in the sidebar as shown in some of the examples in the documentation?

blakeblackshear · April 5, 2021, 7:24pm

Absolutely. You can add a second coral in the detector config. Both will process the detection queue in parallel, so your max fps will be over 200fps if you are getting <10ms inference times.