Local realtime person detection for RTSP cameras

Thanks for the reply! So here’s what I have for the settings:

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m

and

  detectors:
    coral:
      type: edgetpu
      device: usb

VLC, while not perfectly smooth at 15fps, is noticeably better. They are all Ethernet, no wifi. Interesting, I’ll keep digging a bit.

Yea get rid of hwaccel, doesn’t help at all.

1 Like

So just remove the entire ffmpeg section?

Yep. Or comment it out and give it a test.

1 Like

Got it, thanks!

No change, still choppy while VLC and my i3 based DVR is not. So something isn’t right somewhere in my setup.

Using Frigate in Home Assistant (Hassio). Author - thanks for your creation!

Can somebody explain where the actual clips/snapshots storage is?
Media Library in HA shows all stored contents, but in actual paths I find in Portainer (Docker container of Frigate) there’s nothing in fact.
I changed “media” path from default “data/supervisor/homeassistant”
to


But nothing was found in a new path…

How can I manage storage?
Thank you in advance!

Storage is in /mnt/data/supervisor/homeassistant/media/frigate

Does anyone successfully run neolink with a Reolink Lumus Cam and frigate? Seems like any time frigate will access the exposed stream neolink crashes… but it works fine with VLC… does anyone know how to make it play nice together? Thanks! :wink:

My “/mnt” is empty. Does Frigate its mount folders onto Supervisor system, but not HA which is accessible for the user?

Struggeling with the same issue. Somehow it seems to be related to FFMPEG settings.
Running it on my local machine works fine with the following command:

./ffmpeg -rtsp_transport tcp -i rtsp://<ip>:8554/cam1 -c:v copy -f segment -segment_wrap 24 -strftime 1 -segment_time 30 -segment_format mp4 "./%Y-%m-%d_%H-%M-%S.mp4"

tried to replicate the same in frigate settings, but it keeps crashing all the time:

fmpeg:
  global_args: 
    -hide_banner 
    -loglevel info
  input_args:
    - -rtsp_transport 
    - tcp
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p

cameras:
  camera1:
    ffmpeg:
      inputs:
        - path: 'rtsp://<ip>:8554/cam1'
          roles:
            - detect
            - rtmp
      output_args:
        rtmp: -c copy -an -f flv
    width: 1920
    height: 1080
    fps: 15

It just generates the following errors:

 * Starting nginx nginx
   ...done.
frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.app                    INFO    : Camera processor started for camera1: 39
frigate.app                    INFO    : Capture process started for camera1: 42
frigate.mqtt                   INFO    : MQTT connected
frigate.video                  INFO    : camera1: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : camera1: ffmpeg process is not running. exiting capture thread...
detector.coral                 INFO    : Starting detection process: 36
frigate.edgetpu                INFO    : Attempting to load TPU as usb:0
frigate.edgetpu                INFO    : TPU found
ffmpeg.garage.detect           ERROR   : [h264 @ 0x55ca6eaf25c0] missing picture in access unit with size 22
ffmpeg.garage.detect           ERROR   : [h264 @ 0x55ca6eaf25c0] no frame!
ffmpeg.garage.detect           ERROR   : [rtsp @ 0x55ca6eaee940] decoding for stream 0 failed
ffmpeg.garage.detect           ERROR   : [rtsp @ 0x55ca6eaee940] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
ffmpeg.garage.detect           ERROR   : Consider increasing the value for the 'analyzeduration' and 'probesize' options
ffmpeg.garage.detect           ERROR   : Guessed Channel Layout for Input Stream #0.1 : stereo
ffmpeg.garage.detect           ERROR   : Input #0, rtsp, from 'rtsp://<ip>:8554/cam1':
ffmpeg.garage.detect           ERROR   :   Metadata:
ffmpeg.garage.detect           ERROR   :     title           : Session streamed with GStreamer
ffmpeg.garage.detect           ERROR   :     comment         : rtsp-server
ffmpeg.garage.detect           ERROR   :   Duration: N/A, bitrate: 512 kb/s
ffmpeg.garage.detect           ERROR   :     Stream #0:0: Video: h264, none, 90k tbr, 90k tbn, 180k tbc
ffmpeg.garage.detect           ERROR   :     Stream #0:1: Audio: pcm_s16be, 16000 Hz, stereo, s16, 512 kb/s
ffmpeg.garage.detect           ERROR   : [flv @ 0x55ca6eb0ef00] dimensions not set
ffmpeg.garage.detect           ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
ffmpeg.garage.detect           ERROR   : Stream mapping:
ffmpeg.garage.detect           ERROR   :   Stream #0:0 -> #0:0 (copy)
ffmpeg.garage.detect           ERROR   :   Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))
ffmpeg.garage.detect           ERROR   :     Last message repeated 1 times

I’m struggeling to understand what’s the different from my local ffmpeg command to what frigate is doing. This is also crashing the neolink server. No issues at all calling the same stream via VLC.

Look at the /api/config endpoint to see the exact ffmpeg command frigate runs.

Okay, now I have the command… but I can’t simulate it locally. Reason: Frigate kills the input RTSP stream and if I stop Frigate then I don’t have an endpoint for the output stream.

Funny part: now I get at least a distorted picture for some seconds, but the process crashes after some frames. Most of the times I just get errors, I have only seen it 1 or 2 times that I got an image.

/api/config/ content:

{
  "cameras": {
    "camera1": {
      "best_image_timeout": 60,
      "clips": {
        "enabled": false,
        "objects": null,
        "post_capture": 5,
        "pre_capture": 5,
        "required_zones": [],
        "retain": {
          "default": 10,
          "objects": {}
        }
      },
      "detect": {
        "enabled": true,
        "max_disappeared": 75
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel info -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -rtsp_transport tcp -i rtsp://<ip>:8554/cam1 -c copy -an -f flv rtmp://127.0.0.1/live/cam1 -r 15 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect",
            "rtmp"
          ]
        }
      ],
      "fps": 15,
      "frame_shape": [
        1080,
        1920
      ],
      "height": 1080,
      "motion": {
        "contour_area": 100,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 180,
        "mask": null,
        "threshold": 25
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "required_zones": [],
        "timestamp": true
      },
      "name": "camera1",
      "objects": {
        "filters": {
          "bird": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cat": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "dog": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "person": {
            "mask": [],
            "max_area": 100000,
            "min_area": 5000,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": null,
        "track": [
          "person",
          "cat",
          "dog",
          "bird"
        ]
      },
      "record": {
        "enabled": false,
        "retain_days": 30
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": false,
        "crop": false,
        "enabled": false,
        "height": null,
        "required_zones": [],
        "retain": {
          "default": 10,
          "objects": {}
        },
        "timestamp": false
      },
      "width": 1920,
      "zones": {}
    }
  },
  "clips": {
    "max_seconds": 300,
    "retain": {
      "default": 10,
      "objects": {}
    },
    "tmpfs_cache_size": ""
  },
  "database": {
    "path": "/media/frigate/clips/frigate.db"
  },
  "detectors": {
    "coral": {
      "device": "usb:0",
      "num_threads": 3,
      "type": "edgetpu"
    }
  },
  "environment_vars": {},
  "logger": {
    "default": "INFO",
    "logs": {}
  },
  "model": {
    "height": 320,
    "width": 320
  },
  "mqtt": {
    "client_id": "frigate",
    "host": "<ip>",
    "port": 1883,
    "stats_interval": 60,
    "topic_prefix": "frigate",
    "user": "frigate"
  },
  "snapshots": {
    "retain": {
      "default": 10,
      "objects": {}
    }
  }
}

Now in the frigate log I get the following entries over and over again:

frigate.video                  INFO    : camera1: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : camera1: ffmpeg process is not running. exiting capture thread...
ffmpeg.garage.detect           ERROR   : [h264 @ 0x564bb46bd5c0] missing picture in access unit with size 22
ffmpeg.garage.detect           ERROR   : [h264 @ 0x564bb46bd5c0] no frame!
ffmpeg.garage.detect           ERROR   : [rtsp @ 0x564bb46b9940] decoding for stream 0 failed
ffmpeg.garage.detect           ERROR   : [rtsp @ 0x564bb46b9940] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
ffmpeg.garage.detect           ERROR   : Consider increasing the value for the 'analyzeduration' and 'probesize' options
ffmpeg.garage.detect           ERROR   : Guessed Channel Layout for Input Stream #0.1 : stereo
ffmpeg.garage.detect           ERROR   : Input #0, rtsp, from 'rtsp://<ip>:8554/cam1':
ffmpeg.garage.detect           ERROR   :   Metadata:
ffmpeg.garage.detect           ERROR   :     title           : Session streamed with GStreamer
ffmpeg.garage.detect           ERROR   :     comment         : rtsp-server
ffmpeg.garage.detect           ERROR   :   Duration: N/A, start: 0.000000, bitrate: 512 kb/s
ffmpeg.garage.detect           ERROR   :     Stream #0:0: Video: h264, none, 90k tbr, 90k tbn, 180k tbc
ffmpeg.garage.detect           ERROR   :     Stream #0:1: Audio: pcm_s16be, 16000 Hz, stereo, s16, 512 kb/s
ffmpeg.garage.detect           ERROR   : [flv @ 0x564bb46dd500] dimensions not set
ffmpeg.garage.detect           ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
ffmpeg.garage.detect           ERROR   : Stream mapping:
ffmpeg.garage.detect           ERROR   :   Stream #0:0 -> #0:0 (copy)
ffmpeg.garage.detect           ERROR   :   Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))
ffmpeg.garage.detect           ERROR   :     Last message repeated 1 times

Just want to say thanks! Installed Frigate as an addon on my Ubuntu install and with two camera’s even my good old Celeron G640 handles with a little as 20% CPU!

How can I docker pull blakeblackshear/frigate:stable-amd64nvidia in Home assistant (using Portainer) to use Nvidia GPU acceleration in Frigate.
From author’s HowTo page I understood there should be docker-compose (not familiar with that) in order to use Nvidia NVDEC. Please provide simple way of installing Frigate in Home Assistant with Nvidia GPU decoding capability.
Thank you!

I’ve been using Frigate for a few weeks and I love it. But I have one problem, the camera shows up really small in HA, it’s only 492x277 and the rtmp stream from frigate is 1920x1080 (i’ve checked this with VLC). Any suggestions how to fix this?

The preview image will always be that size. You should see the full resolution stream when playing. What are you trying to do?

Your stream is h.265, but I don’t think frigate supports that. You need an h.264 stream.

When playing where? I’m trying to play it in lovelace.
I want to be able to see the live stream in a bigger resolution i lovelace. (Clips works as expected)

Maybe you don’t have the stream component enabled in homeassistant. It should play the rtmp feed if you do.