Local realtime person detection for RTSP cameras

No problem. It is exactly what I suspected. Those are great inference speeds. What hardware are you running on?

1 Like

For anyone using Hassio on amd64 platform (For example the Celeron NUC Iā€™m running on)
You can add the following URL as a repository https://github.com/mories76/ha-addon-frigate
It might work :wink: I am not a developer, no prior github experience.

@blakeblackshear a parameter CONFIG_FILE or something like that would work fine. Iā€™ve added the /dev/dri/renderD128

2 Likes

Iā€™ll try it when I can

The hardware is an older Dell Optiplex i5 at 3.4ghz. Works great for HA and pretty much anything else Iā€™ve thrown at it.

The blakeblackshear/frigate:dev image works great with HW Accel now!

I do notice that my min_area isnā€™t being filtered though for a person.

EDIT: It looks like it is filteringā€¦ But itā€™s also sending notifications for ones that are below the threshold?

I think this is happening.

When I choose not to fill out the global config, the debug screen shows it tagging all objects.

Since I have a notification set up to detect a person, it pushes that notification and is ignoring the camera specific filtering in the yaml.

So this config is missing the optional global objects, so it tags all objects and sends notifications for people? I think thatā€™s why this is happening to me.

cameras:
  front_window:
    ffmpeg:
      input: REDACTED
    take_frame: 2
    objects:
      track:
        - person
      filters:
        person:         
          min_area: 235000
          max_area: 3686400
          threshold: 0.8          
    regions:
      - size: 1920
        x_offset: 0
        y_offset: 0

I an effort to stop editing my post, Iā€™ll summarize.

This does not correctly filter:

cameras:
  front_window:
    ffmpeg:
      input: REDACTED
    take_frame: 2
    objects:
      track:
        - person
      filters:
        person:         
          min_area: 235000
          max_area: 3686400
          threshold: 0.8          
    regions:
      - size: 1920
        x_offset: 0
        y_offset: 0

This does correctly filter (but at the cost of removing all non-persons from the debug view:

objects:
  track:
    - person

cameras:
  front_window:
    ffmpeg:
      input: REDACTED
    take_frame: 2
    objects:
      track:
        - person
      filters:
        person:         
          min_area: 235000
          max_area: 3686400
          threshold: 0.8          
    regions:
      - size: 1920
        x_offset: 0
        y_offset: 0

Im not sure I understand what you mean by ā€œbut at the cost of removing all non-persons from the debug viewā€. Only tracked objects are shown in the debug view now. It doesnā€™t waste time tracking objects that arenā€™t listed.

In the first example you posted, I would expect it to track person, car, and truck objects and apply the filters for size and threshold to the person object on the front_window camera. Specifying person under track on the camera would have no effect, because the global default already includes person.

In the second example, I would expect frigate to only look for person objects across all cameras. Again, specifying person under track on the camera would have no effect, because the global default already includes person.

I am starting to think that the way I am merging tracked objects is too confusing.

I pushed up a new version of the beta image that might fix the issue @hawkeye217 and @billimek are running into. It also addresses the HW accell issues that @Kyle was having.

2 Likes

Iā€™ll give it a go and let you know. Thanks @blakeblackshear!

Ah. I was under the impression that the MJPEG debugging stream would show all objects (even those not tracked) to aid with debugging. No worries if it isnā€™t.

What Iā€™m seeing in my second example (posted again below) is that if I donā€™t include the global rule at the top of my config, the filters in the camera arenā€™t working. Specifically, I couldnā€™t get the min_area to work inside the camera config unless I declared it

After pulling your latest 0.4.0-beta image (sha256-a1219fac2320c10d58fec39723eed4d4da61302c6f1569161c1dad481f167f67) it looks like the filter issue is no longer a problem. :man_shrugging: Itā€™s either fixed or I had a problem with my config this morning.

Thanks for updating the beta to fix the VAPI! HW Accel is working great again on the latest beta. :slight_smile:

Hi guys,
Was cpu (non coral) support included in the latest version?

Not yet. It will be available soon.

Sorry for the late reply!

Both frigate and motion are configured to receive the camera feed via rtsp.

In home assistant I have this binary_sensor for the mqtt topic that frigate publishes to

  - name: Person Front Yard
    platform: mqtt
    state_topic: "frigate/front/person"
    device_class: motion
    availability_topic: "frigate/available"

I then have two automations that trigger from the state changes of that binary sensor. These automations call a shell command action to toggle the motion of motioneye to on or off.

in my automations.yaml, configured through the ui editor

- id: '1576636282260'
  alias: Start motion capture
  description: Starts capture on motion eyebased on person sensor
  trigger:
  - entity_id: binary_sensor.person_front_yard
    from: 'off'
    platform: state
    to: 'on'
  condition: []
  action:
  - service: shell_command.motion_on
- id: '1576636379494'
  alias: Stop motion capture
  description: Stops capture on motion eyebased on person sensor
  trigger:
  - entity_id: binary_sensor.person_front_yard
    from: 'on'
    platform: state
    to: 'off'
  condition: []
  action:
  - service: shell_command.motion_off

from configuration.yaml

shell_command:
  motion_on: curl http://<motioneyeip>:7999/2/config/set?emulate_motion=on
  motion_off: curl http://<motioneyeip>:7999/2/config/set?emulate_motion=off

For motioneye I only have videos enabled (other than the required sections at the top). Iā€™ve found that I can configure the motion detection section how I want it to behave, then I turn off the motion setting. I also mask out the entire screen in motion so that its motion detection does not fire on its own. The mask may not be necessary but I swear I was seeing it intervene. Even though motion is disabled, the shell command above will trigger ā€˜motion detectedā€™ and it will record video with the settings (pre/post frames)

note, the 7999 is the motioneye api port, i had to enable remote control in the motion.conf file. See here

When you navigate the web ui youā€™ll see the structure, i.e the /2/ above is the camera id.

This has been pretty stable. Iā€™m using a wyze cam w/ the dafang hacks and itā€™s a bit flakey. I have some todoā€™s like consolidate the automations to a single one, get more/better cameras, etc but otherwise Iā€™m pretty happy with it.

Iā€™m running the latest image you pushed and there was some activity on my camera, so I just went to check the frigate logs and saw about 15 of these:

RegionPrepper: frame_time not in frame_cache

No memory leak or restarting yet, though. And the debug output is clean with 0 frames in the queue, unlike earlier.

That message doesnā€™t necessarily indicate a problem. It means frigate was still trying to refine object tracking on that frame, but the image expired from the cache because it was too old. It is how I tell frigate to give up and move on because it should be looking at a more recent frame. As long as it isnā€™t causing your queues to fill up, it should be fine.

Yep, I looked at your code and noticed it was just an informational message. Iā€™ll let you know if I run into any more issues. Thanks!

Pushed up a new version of the 0.4.0 beta that addresses the following:

  • Make ffmpeg watchdog timeout configurable per camera issue
  • Configurable timestamp in image snapshots issue
  • Use ffprobe to get frame shape for UDP camera support issue
3 Likes

@blakeblackshear Iā€™m having trouble with this latest version of the 0.4.0-beta image. It starts-up with some new/different logging output, but never seems to start fully and eventually times-out the healthcheck,

On connect called
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@porch:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@front:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@driveway:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@pool:554/cam/realmonitor?channel=1&subtype=1"
{'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 0, 'height': 0, 'coded_width': 0, 'coded_height': 0, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:0', 'level': -99, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '90000/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>@basement:554/h264Preview_01_sub"

Any ideas? Configuration used is here.

Thatā€™s strange. ffprobe is showing your width and height at 0. Not sure why it isnā€™t working for your camera. Can you try this command?

ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 <rtsp_url>

Sure thing,

root@frigate-577b57559f-jtgtl:/opt/frigate# ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 "rtsp://<redacted>@front:554/cam/realmonitor?channel=1&subtype=1"
[rtsp @ 0x55a201f2f6a0] method SETUP failed: 454 Session Not Found
0x0
root@frigate-577b57559f-jtgtl:/opt/frigate# ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 "rtsp://<redacted>@driveway:554/cam/realmonitor?channel=1&subtype=1"
[rtsp @ 0x5624fb41d6a0] method SETUP failed: 454 Session Not Found
0x0
root@frigate-577b57559f-jtgtl:/opt/frigate# time ffprobe -v error -select_streams v:0 -show_entries stream=height,width -of csv=s=x:p=0 "rtsp://<redacted>@pool:554/cam/realmonitor?channel=1&subtype=1"
[rtsp @ 0x55bdcc1ec6a0] method SETUP failed: 454 Session Not Found
0x0

real	0m10.433s
user	0m0.086s
sys	0m0.039s

Here is the command against three of the cameras - the last command included a time wrapper, it seems to ā€˜time outā€™ after 10s based on my observations.

As an aside, after auto-restarting about 7 times (due to the healthcheck probe failing), it eventually went live, but the camera pages are returning a 500 with the following stack trace in the logs,

10.42.3.41 - - [19/Jan/2020 01:52:04] "GET /porch HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 304, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 294, in execute
    for data in application_iter:
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/wsgi.py", line 506, in __next__
    return self._next()
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
    for item in iterable:
  File "/opt/frigate/detect_objects.py", line 146, in imagestream
    frame = cameras[camera_name].get_current_frame_with_objects()
  File "/opt/frigate/frigate/video.py", line 385, in get_current_frame_with_objects
    frame_bytes = self.frame_with_objects(frame_time)
  File "/opt/frigate/frigate/video.py", line 366, in frame_with_objects
    time_to_show = datetime.datetime.fromtimestamp(frame_time).strftime("%m/%d/%Y %H:%M:%S")
TypeError: an integer is required (got type NoneType)

ā€¦ and the debug page suggests that it isnā€™t capturing anything from the cameras,