Local realtime person detection for RTSP cameras

blakeblackshear · June 28, 2019, 9:15pm

16ms should be in the 50fps ballpark, but other things could impact it as well. I won’t know for sure until I try it obviously. I can’t think of a reason frigate would choke on a portrait orientation. Anything in the logs?

scstraus · June 29, 2019, 12:36am

No, the log just stops showing any new events and I can’t open the cameras… Also getting cases where the camera is there but no bounding boxes are showing…

scstraus · June 29, 2019, 12:38am

Any hints for running on synology? Did you set it all up using the shell or do you use the synology UI? How much CPU is it using?

steve1 · June 29, 2019, 4:07am

i used the shell to install it, using this command.

docker run -d --name frigate --privileged --restart unless-stopped --device /dev/dri:/dev/dri -v /dev/bus/usb:/dev/bus/usb -v /volume1/apps/configs/frigate/config:/config:ro -v /volume1/motion/storage/frigate:/storage:rw -p 4444:4000 blakeblackshear/frigate

Running 5 cameras at the moment at about 8fps per camera on the low quality setting, averages about 20-23% cpu. I tried using a vaapi enabled ffmpeg build as well to offload the processing but never got it to work reliably.

scstraus · June 29, 2019, 12:01pm

First you pulled the repository any built the docker container, though. right?

steve1 · June 30, 2019, 1:08am

No, just pulled the pre built version. I did try building the container on the Nas but it failed and I moved on

I ended up forking the repo and doing my own build, which works well. You can try that if you like, replace the repo name with sneighbour/frigate

scstraus · June 30, 2019, 11:42am

Okay, as I have the identical model, I will try yours if it’s working well for you. Thanks.

steve1 · June 30, 2019, 10:25pm

Here is my Synology export, this might be easier for you? You can import this directly into the gui if you want, i’d change

“image” : “sneighbour/frigate:latest”,

to

“image” : “blakeblackshear/frigate:latest”,

though if i were you, to ensure you get updates. I’ve also changed port mappings and added a few extra folders so i could mess with the python scripts locally

{
   "cap_add" : null,
   "cap_drop" : null,
   "cmd" : "python3 -u detect_objects.py",
   "cpu_priority" : 50,
   "devices" : [
      {
         "CgroupPermissions" : "rwm",
         "PathInContainer" : "/dev/dri",
         "PathOnHost" : "/dev/dri"
      }
   ],
   "enable_publish_all_ports" : false,
   "enable_restart_policy" : true,
   "enabled" : true,
   "env_variables" : [
      {
         "key" : "TZ",
         "value" : "Australia/Brisbane"
      },
      {
         "key" : "PATH",
         "value" : "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      },
      {
         "key" : "PYTHONPATH",
         "value" : ":/usr/local/lib/python3.5/dist-packages/tensorflow/models/research:/usr/local/lib/python3.5/dist-packages/tensorflow/models/research/slim"
      }
   ],
   "exporting" : false,
   "id" : "74bb380ce3429d1460d16d4b45529b76e13123a4a3d0ea77684ee8e26339cc1e",
   "image" : "sneighbour/frigate:latest",
   "is_ddsm" : false,
   "is_package" : false,
   "links" : [],
   "memory_limit" : 0,
   "name" : "frigate-vaapi",
   "network" : [
      {
         "driver" : "bridge",
         "name" : "bridge"
      }
   ],
   "network_mode" : "default",
   "port_bindings" : [
      {
         "container_port" : 4000,
         "host_port" : 4444,
         "type" : "tcp"
      }
   ],
   "privileged" : true,
   "shortcut" : {
      "enable_shortcut" : false
   },
   "ulimits" : null,
   "use_host_network" : false,
   "volume_bindings" : [
      {
         "host_volume_file" : "/apps/configs/frigate/config",
         "mount_point" : "/config",
         "type" : "ro"
      },
      {
         "host_volume_file" : "/apps/configs/frigate",
         "mount_point" : "/opt/frigate",
         "type" : "rw"
      },
      {
         "host_volume_file" : "/motion/storage/frigate",
         "mount_point" : "/storage",
         "type" : "rw"
      },
      {
         "host_absolute_path" : "/dev/bus/usb",
         "mount_point" : "/dev/bus/usb",
         "type" : "rw"
      }
   ],
   "volumes_from" : null
}

Kyle · July 1, 2019, 11:47pm

Could you add the scripts folder (specifically) scripts/install_edgetpu_api.sh to your ffmpeg-subprocess branch? It’s called out in the Dockerfile. I’m trying to debug my size issue while decreasing the framerate from the ffmpeg subprocess.

Or any other changes that might help make it work. It’s hitting a lot of dead ends on the current branch state getting it set up, but I remember you mentioning you had some local stuff done that you hadn’t pushed a while back.

steve1 · July 2, 2019, 12:12am

happy to add it but i don’t see where i’ve included/referenced that file, am i missing it?

github.com

steveneighbour/frigate/blob/cc73de84b847465acc8419f46b7993118ebad27d/Dockerfile

#FROM ubuntu:18.04
FROM jrottenberg/ffmpeg:vaapi AS base

WORKDIR     /tmp/workdir

ARG python_version=3.6
ARG OPENCV_VERSION=4.0.1
ARG DEVICE

# Install packages for apt repo
RUN apt-get -qq update && apt-get -qq install --no-install-recommends -y \
    apt-transport-https \
    ca-certificates \
    curl \
    wget \
    gnupg-agent \
    dirmngr \
    software-properties-common \
    && rm -rf /var/lib/apt/lists/*

This file has been truncated. show original

It’s likely better to wait for a new official build, mine is working but hacked to be way different that the legit frigate

Kyle · July 2, 2019, 12:20am

Oops. Sorry. I should have been more specific. I was asking if @blakeblackshear could add the scripts folder on his partial branch.

github.com

blakeblackshear/frigate/blob/8218ea569974b83a470b40bf684319a9cac5b05f/Dockerfile#L83


&& cmake -D CMAKE_INSTALL_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local/ .. \
&& make -j4 \
&& make install \
&& ldconfig \
&& rm -rf /usr/local/src/opencv-4.0.1

# Download and install EdgeTPU libraries for Coral
RUN wget https://dl.google.com/coral/edgetpu_api/edgetpu_api_latest.tar.gz -O edgetpu_api.tar.gz --trust-server-names \
 && tar xzf edgetpu_api.tar.gz

COPY scripts/install_edgetpu_api.sh edgetpu_api/install.sh

RUN cd edgetpu_api \
 && /bin/bash install.sh

# Copy a python 3.6 version
RUN cd /usr/local/lib/python3.6/dist-packages/edgetpu/swig/ \
 && ln -s _edgetpu_cpp_wrapper.cpython-35m-arm-linux-gnueabihf.so _edgetpu_cpp_wrapper.cpython-36m-arm-linux-gnueabihf.so

# symlink the model and labels
RUN wget https://dl.google.com/coral/canned_models/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite -O mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite --trust-server-names

Unless he’s going a different route. But I was hoping to use that branch so I could limit the framerate with the ffmpeg -v command (since my cameras can’t adjust their framerate).

blakeblackshear · July 2, 2019, 12:35am

Not sure how I missed that. I can add it tonight.

steve1 · July 2, 2019, 12:44am

haha, that makes more sense. Sorry, i thought you were replying to my Synology specific comments!

Yeah, i noted that was missed and just ignored it - my hacked up branch just copies the correct files into place, without running the script. I believe the end result is the same.

blakeblackshear · July 2, 2019, 2:21am

I just added the scripts. Not sure if using ffmpeg to limit the frame rate will work like you want, so I went ahead and implemented a frame rate limiter on that branch too. I haven’t tested it much, but if you add a take_frame parameter to your config for a camera, it should process every nth frame. For example, if it is set to 2, it will process every 2nd frame, set to 3 will get every 3rd frame. See example here: https://github.com/blakeblackshear/frigate/blob/4ce6f657a1dca1280587e9df3d8b45782a8fe8a5/config/config.yml#L19

Kyle · July 2, 2019, 11:02am

Thanks for the commits! I’ll see if I can get a look at it today. Sorry about my wave of questions. I’m always trying to learn why decisions were made.

2 questions. Is the argument default value take_frame=1 necessary:

github.com

blakeblackshear/frigate/blob/b437d8494a3640a506b2f98cfbeaaa04e0f3d13c/frigate/video.py#L16


import ctypes
import multiprocessing as mp
import subprocess as sp
import numpy as np
from . util import tonumpyarray, draw_box_with_label
from . object_detection import FramePrepper
from . objects import ObjectCleaner, BestPersonFrame
from . mqtt import MqttObjectPublisher

# fetch the frames as fast a possible and store current frame in a shared memory array
def fetch_frames(shared_arr, shared_frame_time, frame_lock, frame_ready, frame_shape, rtsp_url, take_frame=1):
    # convert shared memory array into numpy and shape into image array
    arr = tonumpyarray(shared_arr).reshape(frame_shape)
    frame_size = frame_shape[0] * frame_shape[1] * frame_shape[2]

    ffmpeg_cmd = ['ffmpeg', 
        '-hide_banner', '-loglevel', 'panic',
        '-avoid_negative_ts', 'make_zero', 
        '-fflags', 'nobuffer',
        '-flags', 'low_delay',
        '-strict', 'experimental',

When the default value is already set here?:

github.com

blakeblackshear/frigate/blob/b437d8494a3640a506b2f98cfbeaaa04e0f3d13c/frigate/video.py#L132


                self.camera.start_or_restart_capture()
                time.sleep(5)

class Camera:
    def __init__(self, name, config, prepped_frame_queue, mqtt_client, mqtt_prefix):
        self.name = name
        self.config = config
        self.detected_objects = []
        self.recent_frames = {}
        self.rtsp_url = get_rtsp_url(self.config['rtsp'])
        self.take_frame = self.config.get('take_frame', 1)
        self.regions = self.config['regions']
        self.frame_shape = get_frame_shape(self.rtsp_url)
        self.mqtt_client = mqtt_client
        self.mqtt_topic_prefix = '{}/{}'.format(mqtt_prefix, self.name)

        # compute the flattened array length from the shape of the frame
        flat_array_length = self.frame_shape[0] * self.frame_shape[1] * self.frame_shape[2]
        # create shared array for storing the full frame image data
        self.shared_frame_array = mp.Array(ctypes.c_uint8, flat_array_length)
        # create shared value for storing the frame_time

Or is it guarding against other issues/readability.

What’s the difference between take_frame and adding a ffmpeg -r X option? Is that because pulling frames is considered a lossless conversion versus re-encoding the stream to a
lower framerate? Not to mention, it’s probably more performant?

My (probable lossy) method:

https://github.com/aav7fl/frigate/blob/d65e9321913fb8639f453a7b4a7f652100599778/frigate/video.py#L34

I’ll see if I can get a PR out there tonight to add in the label to that branch to resolve my own issue (https://github.com/blakeblackshear/frigate/issues/42).

I’m thinking of adding a new field to the obj here that contains the ['person_size'] so we don’t need to re-caculate it again when it gets passed into get_current_frame_with_objects(self).

Setting it here:

github.com

blakeblackshear/frigate/blob/4ce6f657a1dca1280587e9df3d8b45782a8fe8a5/frigate/video.py#L238



def get_capture_pid(self):
    return self.capture_process.pid

def add_objects(self, objects):
    if len(objects) == 0:
        return

    for obj in objects:
        if obj['name'] == 'person':
            person_area = (obj['xmax']-obj['xmin'])*(obj['ymax']-obj['ymin'])
            # find the matching region
            region = None
            for r in self.regions:
                if (
                        obj['xmin'] >= r['x_offset'] and
                        obj['ymin'] >= r['y_offset'] and
                        obj['xmax'] <= r['x_offset']+r['size'] and
                        obj['ymax'] <= r['y_offset']+r['size']
                    ): 
                    region = r

and using it here:

github.com

blakeblackshear/frigate/blob/4ce6f657a1dca1280587e9df3d8b45782a8fe8a5/frigate/video.py#L282



def get_current_frame_with_objects(self):
    # make a copy of the current detected objects
    detected_objects = self.detected_objects.copy()
    # lock and make a copy of the current frame
    with self.frame_lock:
        frame = self.shared_frame_np.copy()

    # draw the bounding boxes on the screen
    for obj in detected_objects:
        label = "{}: {}%".format(obj['name'],int(obj['score']*100))
        draw_box_with_label(frame, obj['xmin'], obj['ymin'], obj['xmax'], obj['ymax'], label)

    for region in self.regions:
        color = (255,255,255)
        cv2.rectangle(frame, (region['x_offset'], region['y_offset']), 
            (region['x_offset']+region['size'], region['y_offset']+region['size']), 
            color, 2)

    # convert to BGR
    frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)

Kyle · July 2, 2019, 10:27pm

I opened up a PR to add the area labels to the bounding boxes (from the ffmpeg branch).

https://github.com/blakeblackshear/frigate/pull/47

It’s now working great! Thanks to your latest changes with take_frame I have a lower framerate, more zones, and now my newly added area label.

Turns out I was wayyy underestimating my person area.

This also helped bring the load on my system from 5-6 to 1-2. Thanks!

blakeblackshear · July 3, 2019, 2:02am

To answer your previous questions:

Technically, the take_frame=1 default value isn’t necessary because I am never calling that function without passing a value. It just ensures the default is full speed if I use it differently in the future.
I am not sure exactly what the -r option is doing for ffmpeg, but I wouldn’t be surprised if it just limited the speed at which it processed frames. You really want it to skip/drop frames and move on as quickly as possible. There is probably some kind of optimized way to do that in ffmpeg, and it may be more efficient than what I am currently doing. The problem is always that you must decode every I-frame in an H264 stream, or you can’t calculate a full frame for the other frames.

tc23 · July 3, 2019, 8:22pm

Is it possible to get the person_score or confidence threshold passed through in the mqtt message as well? If so, I’m thinking I could use that score in a template_value in the binary sensor to create a dynamic/time of day min_threshold?

blakeblackshear · July 3, 2019, 10:36pm

That is actually how I initially implemented it. The problem is that the value changes on almost every frame when someone is detected. That resulted in frigate flooding MQTT. What if you could define your own score thresholds such as low, medium, high for each camera? Then you could dynamically choose the score threshold that would trigger an event however you want.

tc23 · July 3, 2019, 11:05pm

Is that low, medium, high function currently available? I know that I can set different thresholds for each camera, but I’m have an issue with my courtyard camera “seeing” a person at night when there is no one there. I’m thinking it’s a function of the IR. So I would like to set it low/medium during the day and high at night. Does that make sense?