Local realtime person detection for RTSP cameras

I pushed up a new version of the beta image that might fix the issue @hawkeye217 and @billimek are running into. It also addresses the HW accell issues that @Kyle was having.

2 Likes

I’ll give it a go and let you know. Thanks @blakeblackshear!

Ah. I was under the impression that the MJPEG debugging stream would show all objects (even those not tracked) to aid with debugging. No worries if it isn’t.

What I’m seeing in my second example (posted again below) is that if I don’t include the global rule at the top of my config, the filters in the camera aren’t working. Specifically, I couldn’t get the min_area to work inside the camera config unless I declared it

After pulling your latest 0.4.0-beta image (sha256-a1219fac2320c10d58fec39723eed4d4da61302c6f1569161c1dad481f167f67) it looks like the filter issue is no longer a problem. :man_shrugging: It’s either fixed or I had a problem with my config this morning.

Thanks for updating the beta to fix the VAPI! HW Accel is working great again on the latest beta. :slight_smile:

Hi guys,
Was cpu (non coral) support included in the latest version?

Not yet. It will be available soon.

Sorry for the late reply!

Both frigate and motion are configured to receive the camera feed via rtsp.

In home assistant I have this binary_sensor for the mqtt topic that frigate publishes to

  - name: Person Front Yard
    platform: mqtt
    state_topic: "frigate/front/person"
    device_class: motion
    availability_topic: "frigate/available"

I then have two automations that trigger from the state changes of that binary sensor. These automations call a shell command action to toggle the motion of motioneye to on or off.

in my automations.yaml, configured through the ui editor

- id: '1576636282260'
  alias: Start motion capture
  description: Starts capture on motion eyebased on person sensor
  trigger:
  - entity_id: binary_sensor.person_front_yard
    from: 'off'
    platform: state
    to: 'on'
  condition: []
  action:
  - service: shell_command.motion_on
- id: '1576636379494'
  alias: Stop motion capture
  description: Stops capture on motion eyebased on person sensor
  trigger:
  - entity_id: binary_sensor.person_front_yard
    from: 'on'
    platform: state
    to: 'off'
  condition: []
  action:
  - service: shell_command.motion_off

from configuration.yaml

shell_command:
  motion_on: curl http://<motioneyeip>:7999/2/config/set?emulate_motion=on
  motion_off: curl http://<motioneyeip>:7999/2/config/set?emulate_motion=off

For motioneye I only have videos enabled (other than the required sections at the top). I’ve found that I can configure the motion detection section how I want it to behave, then I turn off the motion setting. I also mask out the entire screen in motion so that its motion detection does not fire on its own. The mask may not be necessary but I swear I was seeing it intervene. Even though motion is disabled, the shell command above will trigger ‘motion detected’ and it will record video with the settings (pre/post frames)

note, the 7999 is the motioneye api port, i had to enable remote control in the motion.conf file. See here

When you navigate the web ui you’ll see the structure, i.e the /2/ above is the camera id.

This has been pretty stable. I’m using a wyze cam w/ the dafang hacks and it’s a bit flakey. I have some todo’s like consolidate the automations to a single one, get more/better cameras, etc but otherwise I’m pretty happy with it.

I’m running the latest image you pushed and there was some activity on my camera, so I just went to check the frigate logs and saw about 15 of these:

RegionPrepper: frame_time not in frame_cache

No memory leak or restarting yet, though. And the debug output is clean with 0 frames in the queue, unlike earlier.

That message doesn’t necessarily indicate a problem. It means frigate was still trying to refine object tracking on that frame, but the image expired from the cache because it was too old. It is how I tell frigate to give up and move on because it should be looking at a more recent frame. As long as it isn’t causing your queues to fill up, it should be fine.

Yep, I looked at your code and noticed it was just an informational message. I’ll let you know if I run into any more issues. Thanks!

Pushed up a new version of the 0.4.0 beta that addresses the following:

  • Make ffmpeg watchdog timeout configurable per camera issue
  • Configurable timestamp in image snapshots issue
  • Use ffprobe to get frame shape for UDP camera support issue
3 Likes

@blakeblackshear I’m having trouble with this latest version of the 0.4.0-beta image. It starts-up with some new/different logging output, but never seems to start fully and eventually times-out the healthcheck,

On connect called
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@porch:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@front:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@driveway:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@pool:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@basement:554/h264Preview_01_sub"

Any ideas? Configuration used is here.

That’s strange. ffprobe is showing your width and height at 0. Not sure why it isn’t working for your camera. Can you try this command?

ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 <rtsp_url>

Sure thing,

root@frigate-577b57559f-jtgtl:/opt/frigate# ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 "rtsp://<redacted>@front:554/cam/realmonitor?channel=1&subtype=1"
[rtsp @ 0x55a201f2f6a0] method SETUP failed: 454 Session Not Found
0x0
root@frigate-577b57559f-jtgtl:/opt/frigate# ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 "rtsp://<redacted>@driveway:554/cam/realmonitor?channel=1&subtype=1"
[rtsp @ 0x5624fb41d6a0] method SETUP failed: 454 Session Not Found
0x0
root@frigate-577b57559f-jtgtl:/opt/frigate# time ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 "rtsp://<redacted>@pool:554/cam/realmonitor?channel=1&subtype=1"
[rtsp @ 0x55bdcc1ec6a0] method SETUP failed: 454 Session Not Found
0x0

real	0m10.433s
user	0m0.086s
sys	0m0.039s

Here is the command against three of the cameras - the last command included a time wrapper, it seems to ‘time out’ after 10s based on my observations.

As an aside, after auto-restarting about 7 times (due to the healthcheck probe failing), it eventually went live, but the camera pages are returning a 500 with the following stack trace in the logs,

10.42.3.41 - - [19/Jan/2020 01:52:04] "GET /porch HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 304, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 294, in execute
    for data in application_iter:
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/wsgi.py", line 506, in __next__
    return self._next()
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
    for item in iterable:
  File "/opt/frigate/detect_objects.py", line 146, in imagestream
    frame = cameras[camera_name].get_current_frame_with_objects()
  File "/opt/frigate/frigate/video.py", line 385, in get_current_frame_with_objects
    frame_bytes = self.frame_with_objects(frame_time)
  File "/opt/frigate/frigate/video.py", line 366, in frame_with_objects
    time_to_show = datetime.datetime.fromtimestamp(frame_time).strftime("%m/%d/%Y %H:%M:%S")
TypeError: an integer is required (got type NoneType)

… and the debug page suggests that it isn’t capturing anything from the cameras,

Ok. I pushed a new version. If ffprobe fails, it tries to use OpenCV. As a last resort, you can now specify the width and height in the config for each camera and it will skip the auto detection all together.

That did it, thank you!

By the way, I did need to bump the time the healthcheck probes wait to start checking, because the ffprobe thing seems to take ~10s per camera to time out. So in my case with 5 cameras, it’s trying for about 50 seconds before it starts serving and processing.

If you specify the resolution in your config for the cameras, it will skip that step altogether.

I’m on the second 4.0 beta from the 17th

Since 4.0 beta w/ this config I am getting person matches smaller than my min size.

objects:
  track:
    - person
    - car
    - truck
  filters:
    person:
      min_area: 3500
      max_area: 100000
      threshold: 0.5

cameras:
  front:
    ffmpeg:
    take_frame: 1
    regions:
      - size: 460
        x_offset: 60
        y_offset: 220

Most recently I had person w/ a 1827 size.

Did you see the person in the mjpeg stream or in the snapshot?

It was in the Snapshot

I haven’t been able to reproduce that with a similar config, and I don’t see anything obvious. Does it always happen? Try increasing the min_area to a much higher value so you can test to see if it filters you out when walking in the frame. There could be an edge case somewhere that makes it possible for that to happen occasionally.