Local realtime person detection for RTSP cameras

rafagomes · October 24, 2020, 1:11pm

Would you mind sharing the link of the adapter you are using?

rpress · October 24, 2020, 1:34pm

https://www.mouser.com/ProductDetail/Coral/G650-06076-01?qs=W%2FMpXkg%2BdQ6LZJp2eyeh4w%3D%3D

blakeblackshear · October 24, 2020, 1:51pm

Note that your zone is being draw with an X. Your coordinates are not in the right order to make a rectangle. Try this:

    zones:
      ignore:
        coordinates:
          - '1,1'
          - '2560,1'
          - '2560,570'
          - '1,570'

blakeblackshear · October 24, 2020, 1:54pm

I took a look at several recent issues and other open source projects and revamped the docs. Feedback appreciated:

rafagomes · October 24, 2020, 1:57pm

Thank you @rpress, but I meant which M.2 adapter you are using for the coral dual edgetpu.

pollinolas · October 24, 2020, 2:02pm

once again, amazing work, thank you for your effort and dedication

rpress · October 24, 2020, 2:09pm

https://www.amazon.com/dp/B07SN3QYR9

MRobi · October 24, 2020, 7:22pm

They are reolink cams, but they’re the B800’s so they don’t have native RTSP/RTMP support. I’m using the RTSP stream that’s output from the NVR and it doesn’t have RTMP.

The camera’s work good now, and I am getting person detection just fine. Just getting the error on request in the logs now. I wonder if it has to do with the saving of the images? My clips and cache dir are both empty.

Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/serving.py", line 323, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/serving.py", line 314, in execute
    for data in application_iter:
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py", line 506, in __next__
    return self._next()
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
    for item in iterable:
  File "/opt/frigate/detect_objects.py", line 464, in imagestream
    frame = object_processor.get_current_frame(camera_name, draw=True)
  File "/opt/frigate/frigate/object_processing.py", line 357, in get_current_frame
    return self.camera_states[camera].get_current_frame(draw)
  File "/opt/frigate/frigate/object_processing.py", line 73, in get_current_frame
    frame_copy = cv2.cvtColor(frame_copy, cv2.COLOR_YUV2BGR_I420)
cv2.error: OpenCV(4.4.0) /tmp/pip-req-build-a98tlsvg/opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0>; cv::impl::{anonymous}::SizePolicy sizePolicy = cv::impl::<unnamed>::FROM_YUV; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
> Invalid number of channels in input image:
>     'VScn::contains(scn)'
> where
>     'scn' is 3
Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/serving.py", line 323, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/serving.py", line 315, in execute
    write(data)
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/serving.py", line 296, in write
    self.wfile.write(data)
  File "/usr/lib/python3.8/socketserver.py", line 799, in write
    self._sock.sendall(b)
OSError: [Errno 113] No route to host

blakeblackshear · October 24, 2020, 8:26pm

The key message in the logs is “Invalid number of channels in input image”. OpenCV is erroring when trying to convert the video frames from YUV to BGR so it can return the mjpeg feed. Someone else had the same issue on GH, but I’m not sure what is causing it.

scstraus · October 24, 2020, 8:27pm

Hey Blake,

One more quick question about zones. Is

frigate/<zone_name>/<object_name>/snapshot

an MQTT topic that we can subscribe to get snapshots for zones? If not, would you consider it as an FR?

blakeblackshear · October 24, 2020, 8:35pm

It isn’t currently, but yes. I want that feature too.

hawkeye217 · October 24, 2020, 9:30pm

@blakeblackshear The new docs are great!

sdc · October 25, 2020, 1:43pm

I’m having an issue getting hardware acceleration working on my Intel NUC 10i7FNH using docker. Edit: It looks like an i965 VAAPI library is missing - any advice on how to add this to the docker container? When i turn info logs on I see:

Frigate Log Files (INFO Log Detail)

Stream mapping:

  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))

Device setup failed for decoder on input stream #0:0 : Input/output error

garage: ffmpeg sent a broken frame. something is wrong.

garage: ffmpeg process is not running. exiting capture thread...

Input #0, rtsp, from 'rtsp://USER:PASS@IP:554/axis-media/media.amp':

  Metadata:

    title           : Session streamed with GStreamer

    comment         : rtsp-server

  Duration: N/A, start: 1603633834.541189, bitrate: N/A

    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 10 tbr, 90k tbn, 180k tbc

[AVHWDeviceContext @ 0x555a50b567e0] libva: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed

[AVHWDeviceContext @ 0x555a50b567e0] Failed to initialise VAAPI connection: -1 (unknown libva error).

Device creation failed: -5.

Config.yml Global Config

  hwaccel_args:
    - '-hwaccel'
    - vaapi
    - '-hwaccel_device'
    - /dev/dri/renderD128
    - '-hwaccel_output_format'
    - yuv420p

VAINFO Command on Host:

vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSliceLP
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD

VAINFO Command on Docker

libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva error: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed
libva info: va_openDriver() returns -1
vaInitialize failed with error code -1 (unknown libva error),exit

blakeblackshear · October 25, 2020, 2:41pm

Can you tell me more about your host machine?

sdc · October 25, 2020, 2:48pm

It’s the newer Intel NUC 10i7FNH - they are about a year old, but had driver issues (different GPU / driver then the older NUCs). Actually looks like Frigate Docker is built on 18.04 LTS - that might be the issue, I don’t think it has native driver support.

Do you know if there’s a way to tell the image to use a different driver/extension (my host uses : /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so)

blakeblackshear · October 25, 2020, 3:01pm

The latest version is built on 20.04. What version are using?

sdc · October 25, 2020, 3:27pm

Oops - I was running with the docker tag “stable” vs “stable-amd64”. I see it’s 20.04.1 now. Unfortunately I’m getting the same issue , but now I see the iHD driver listed when running vainfo in the container:

error: XDG_RUNTIME_DIR not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: va_openDriver() returns -1
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva error: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed
libva info: va_openDriver() returns -1
vaInitialize failed with error code -1 (unknown libva error),exit

Looks like these files are missing in the container.

blakeblackshear · October 25, 2020, 3:36pm

The i965 driver supports gen 5-9.5 intel processors. The iHD driver supports gen8+. You need the iHD driver installed too for 10th gen processors. I am trying to add it.

blakeblackshear · October 25, 2020, 5:20pm

Ok. Try running this image:
blakeblackshear/frigate:bus_error-22a1fe6-amd64

Your hwaccel args will need to be:

  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128

You will also need to set an environment variable on the container as LIBVA_DRIVER_NAME=iHD.

sdc · October 25, 2020, 6:42pm

Wow thank you, this works - the vainfo output from the container now matches my host. The CPU usage seems to match what it was without HW acceleration (portainer charts show 50% for the image). It works when i use the arguments you provided, when I also add these additional 2 arguments it crashes (not sure if they are needed?). Any idea how I can tell if i’m using CPU vs GPU?

    - '-hwaccel_output_format'
    - yuv420p

Logs:

Creating ffmpeg process...

ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://USER@IPADDR:554/cam/realmonitor?channel=1&subtype=2 -r 5 -f rawvideo -pix_fmt yuv420p pipe:

Camera process started for garage: 49