Local realtime person detection for RTSP cameras

tc23 · June 10, 2019, 11:11am

I’m such a dunce. I’ll update the docker-compose and try again. Sorry for another stupid question, but what does the mask file do and how would I go about creating a personalized one vs the included file?

blakeblackshear · June 10, 2019, 11:37am

I haven’t added much info about the mask to the readme yet, but some info is available here in this comment: https://github.com/blakeblackshear/frigate/issues/30#issuecomment-490718844

Kyle · June 17, 2019, 6:06pm

This is a fantastic project. As commented on earlier in this thread, you really need a link to let us buy you a drink.

Is there anyway to limit the frame rate that python is analyzing? My WYZE cameras do not have a way to change the frame rate in the RSTP stream. This means frigate processes every single frame (multiplied by my 6 regions across 2 cameras) and destroys my CPU/maxes out the Coral buffer queue.

I noticed in the README you mention a tip about limiting the frame rate to lower the CPU usage, but I am unable to do that in my setup without pre-processing the video first.

blakeblackshear · June 17, 2019, 6:21pm

Can you open a github issue? I have implemented that already in a local version, and that will remind me to include it in the next release.

tc23 · June 17, 2019, 7:32pm

Are you on the dafang hack or their recently released version of RSTP? I only ask because I am having the same problem with my dafanged wyze (unifi works perfectly), and have tried adjusting the FPS, Resolution and Bitrate. But in the end I either get queue full. moving on or unable to grab frame. And then it terminates the capture process and restarts. Are these the same errors you are seeing as well?

Kyle · June 17, 2019, 7:50pm

I am not. I am using the Wyze beta2 build that gives it official RTSP support. But it seems to be streaming at 1080p with either 15 or 30 fps. (I think 15 is what I’ve heard?).

I did however get those messages because there were too many frames for the number of regions I was trying to process at the same time on the TPU. Limiting the framerate or number of regions will solve it in my case.

tc23 · June 17, 2019, 7:52pm

Interesting. Yeah, I dropped the frame rate down to 5fps and resolution at 1600X900 with a single region for my Wyze and I am still getting those errors

Kyle · June 17, 2019, 7:56pm

Issues submitted. Let me know if I can help with it.

Kyle · June 17, 2019, 7:57pm

That’s odd. Any other errors show in the logs?

tc23 · June 17, 2019, 8:02pm

I could never get debug to turn on, so not sure what the logs are showing. I have tried it at lower resolutions as well, but same errors occur. I tried 1280X720 and 960X540., both at 5 FPS. Not sure I can dip lower than 5 FPS

Kyle · June 17, 2019, 8:41pm

Is it plugged into a USB 3.X port?

tc23 · June 17, 2019, 8:57pm

Yep, and it works perfectly with my other Unifi cameras, across multiple zones.

Kyle · June 17, 2019, 9:00pm

Huh. I’m not really sure then. Maybe the dafanged implementation is buggy?

tc23 · June 17, 2019, 9:06pm

Yeah, I’ll have to check that GitHub and see if there are any discussions on the frame rate. Either way, hopefully your PR solves the problem

blakeblackshear · June 18, 2019, 1:31pm

I added a Ko-fi link for anyone who wants to contribute to the project because several of you asked for it. Thanks for your support.

@cjackson234 @scstraus @Kyle

Kyle · June 19, 2019, 4:58pm

I seem to be getting notifications for very small regions when my min_person_area is something absurd like 2500.

Is this conditional backwards? I’m not sure if I’m understanding it correctly but wouldn’t it want to be flipped to only exit the loop and continue if it’s not smaller?

github.com

blakeblackshear/frigate/blob/master/frigate/video.py#L255


    areas_of_interest.append((x_min, y_min, x_max, y_max))
unused_motion_boxes = set(range(0, len(motion_boxes))).difference(used_motion_boxes)

# compute motion regions
motion_regions = [calculate_region(frame_shape, motion_boxes[i][0], motion_boxes[i][1], motion_boxes[i][2], motion_boxes[i][3], 1.2)
    for i in unused_motion_boxes]

# compute tracked object regions
object_regions = [calculate_region(frame_shape, a[0], a[1], a[2], a[3], 1.2)
    for a in areas_of_interest]

# merge regions with high IOU
merged_regions = motion_regions+object_regions
while True:
    max_iou = 0.0
    max_indices = None
    region_indices = range(len(merged_regions))
    for a, b in itertools.combinations(region_indices, 2):
        iou = intersection_over_union(merged_regions[a], merged_regions[b])
        if iou > max_iou:
            max_iou = iou

Also, is there a chance that the area size can also be added to the label here to help with debugging person sizes?

github.com

blakeblackshear/frigate/blob/master/frigate/objects.py#L92


# compute centroids of new objects
for obj in group:
    centroid_x = int((obj['box'][0]+obj['box'][2]) / 2.0)
    centroid_y = int((obj['box'][1]+obj['box'][3]) / 2.0)
    obj['centroid'] = (centroid_x, centroid_y)

if len(current_objects) == 0:
    for index, obj in enumerate(group):
        self.register(index, obj)
    return

new_centroids = np.array([o['centroid'] for o in group])

# compute the distance between each pair of tracked
# centroids and new centroids, respectively -- our
# goal will be to match each new centroid to an existing
# object centroid
D = dist.cdist(current_centroids, new_centroids)

# in order to perform this matching we must (1) find the
# smallest value in each row and then (2) sort the row

blakeblackshear · June 19, 2019, 5:35pm

The conditional looks right. It should continue at that point to avoid report a person that is too small. Are you sure you have the minimum set on the correct region?

Adding the person area to the label should be simple enough. It will help me keep track of that request if you create a github issue for it.

Kyle · June 19, 2019, 8:47pm

Thanks! I’ve submitted 2 issues. One for the enhancement, and one for the issue I’m encountering. Sorry for adding so many lately.

It has my config listed, which I believe is correct? https://github.com/blakeblackshear/frigate/issues/43

tc23 · June 20, 2019, 2:58am

What is Unable to grab frame indicative of? I was able to solve my queue full problem. But now, after about 15-60 seconds, I get a blast of about 25 lines of Unable to grab frame? Is it a FPS problem or bitrate?

blakeblackshear · June 20, 2019, 3:34am

That means opencv returned an error code for a single frame. It could be anything, but as long as it clears up on its own, I would just ignore it. Probably just a few corrupt frames.