Local realtime person detection for RTSP cameras

First time trying firgate in container using Portainer. Hoping someone can help me with the beta version issue.

“blakeblackshear/frigate:latest” worked without any issue.

“blakeblackshear/frigate:dev” just shows following 2 lines
ffprobe -v panic -show_error -show_streams -of json “rtsp://xxx:xxx@ip:554//h264Preview_01_sub”,
On connect called,

"blakeblackshear/frigate:0.4.0-beta" shows following

ffprobe -v panic -show_error -show_streams -of json “rtsp://xxx:xxx@ip:554//h264Preview_01_sub”

{‘streams’: [{‘index’: 0, ‘codec_name’: ‘h264’, ‘codec_long_name’: ‘H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10’, ‘profile’: ‘High’, ‘codec_type’: ‘video’, ‘codec_time_base’: ‘0/2’, ‘codec_tag_string’: ‘[0][0][0][0]’, ‘codec_tag’: ‘0x0000’, ‘width’: 640, ‘height’: 480, ‘coded_width’: 640, ‘coded_height’: 480, ‘has_b_frames’: 0, ‘sample_aspect_ratio’: ‘0:1’, ‘display_aspect_ratio’: ‘0:1’, ‘pix_fmt’: ‘yuv420p’, ‘level’: 51, ‘chroma_location’: ‘left’, ‘field_order’: ‘progressive’, ‘refs’: 1, ‘is_avc’: ‘false’, ‘nal_length_size’: ‘0’, ‘r_frame_rate’: ‘4/1’, ‘avg_frame_rate’: ‘0/0’, ‘time_base’: ‘1/90000’, ‘start_pts’: 20970, ‘start_time’: ‘0.233000’, ‘bits_per_raw_sample’: ‘8’, ‘disposition’: {‘default’: 0, ‘dub’: 0, ‘original’: 0, ‘comment’: 0, ‘lyrics’: 0, ‘karaoke’: 0, ‘forced’: 0, ‘hearing_impaired’: 0, ‘visual_impaired’: 0, ‘clean_effects’: 0, ‘attached_pic’: 0, ‘timed_thumbnails’: 0}}, {‘index’: 1, ‘codec_name’: ‘aac’, ‘codec_long_name’: ‘AAC (Advanced Audio Coding)’, ‘profile’: ‘LC’, ‘codec_type’: ‘audio’, ‘codec_time_base’: ‘1/16000’, ‘codec_tag_string’: ‘[0][0][0][0]’, ‘codec_tag’: ‘0x0000’, ‘sample_fmt’: ‘fltp’, ‘sample_rate’: ‘16000’, ‘channels’: 1, ‘channel_layout’: ‘mono’, ‘bits_per_sample’: 0, ‘r_frame_rate’: ‘0/0’, ‘avg_frame_rate’: ‘0/0’, ‘time_base’: ‘1/16000’, ‘start_pts’: 0, ‘start_time’: ‘0.000000’, ‘disposition’: {‘default’: 0, ‘dub’: 0, ‘original’: 0, ‘comment’: 0, ‘lyrics’: 0, ‘karaoke’: 0, ‘forced’: 0, ‘hearing_impaired’: 0, ‘visual_impaired’: 0, ‘clean_effects’: 0, ‘attached_pic’: 0, ‘timed_thumbnails’: 0}}]}

Traceback (most recent call last):

File “detect_objects.py”, line 155, in

main()

File “detect_objects.py”, line 87, in main

fps_tracker

File “/opt/frigate/frigate/object_detection.py”, line 20, in init

self.engine = DetectionEngine(PATH_TO_CKPT)

File “/usr/lib/python3/dist-packages/edgetpu/detection/engine.py”, line 73, in init

super().__init__(model_path)

File “/usr/lib/python3/dist-packages/edgetpu/basic/basic_engine.py”, line 92, in init

self._engine = BasicEnginePythonWrapper.CreateFromFile(model_path)

RuntimeError: No Edge TPU device detected!

On connect called

ffprobe -v panic -show_error -show_streams -of json “rtsp://xxx:xxx@ip:554//h264Preview_01_sub”

I am using the example config from here and only replaced the input stream url

Are you sure that you don’t have multiple frigate containers running?

No, I made sure of that. I did find a work around for the dev image - I have to access the container’s console and run “python3 -u detect_objects.py” then it works. If I make config changes and reboot, it again gets stuck at “On connect called.” but running the object.py from console resolves the issue.

The beta version still shows “RuntimeError: No Edge TPU device detected!” though.

Which coral device do you have?

I have the USB Accelerator. After rebooting host laptop and creating new container seems to fix the issue including the dev image but still having problem after restarting container. There must be something specific to my environment I need to figure out.

The object tracking seems to work really well - Thanks @blakeblackshear for all your work!! I am noticing that my HA’s binary sensor staying in “Detected” state long after there is no car or person in camera view.

It should take about 10 seconds to clear. If someone steps out of the frame and back in, they will keep the same object id.

@blakeblackshear I have 2 cameras, HA is getting the expected mqtt info from first camera “back” and triggering the notification but the 2nd camera “garage” doesn’t seems to be publishing mqtt “off?”. May be i don’t have my configuration properly configured?

web_port: 5000

mqtt:
  host: mqtt_ip
  topic_prefix: frigate
#  client_id: frigate # Optional -- set to override default client id of 'frigate' if running multiple instances
  user: user # Optional -- Uncomment for use
  password: pass # Optional -- Uncomment for use

objects:
  track:
    - person
    - car
    - truck
  filters:
    person:
      min_area: 5000
      max_area: 100000
      threshold: 0.5

cameras:
  back:
    ffmpeg:
      input: rtsp://user:[email protected]:554//h264Preview_01_sub 

    take_frame: 1

    objects:
      track:
        - person
      filters:
        person:
          min_area: 5000
          max_area: 100000
          threshold: 0.5

    regions:
      - size: 300
        x_offset: 340
        y_offset: 0
      - size: 320
        x_offset: 320
        y_offset: 180
      - size: 320
        x_offset: 0
        y_offset: 180        

  garage:
    ffmpeg:
      input: rtsp://user:[email protected]:554//h264Preview_01_sub  
    
    take_frame: 1

    objects:
      track:
        - person
      filters:
        person:
          min_area: 5000
          max_area: 100000
          threshold: 0.5  

    regions:
      - size: 350
        x_offset: 0
        y_offset: 0
      - size: 320
        x_offset: 250
        y_offset: 100
      - size: 300
        x_offset: 0
        y_offset: 180        
binary_sensor:
# frigate binary sensors:
  - name: Camera Person
    platform: mqtt
    state_topic: "frigate/back/person"
    device_class: motion
    availability_topic: "frigate/available"

  - name: Camera Car garage
    platform: mqtt
    state_topic: "frigate/garage/car"
    device_class: motion
    availability_topic: "frigate/available" 
    
  - name: Camera Person garage
    platform: mqtt
    state_topic: "frigate/garage/person"
    device_class: motion
    availability_topic: "frigate/available"   

Which version are you running?

I am running the beta version pulled from docker hub - blakeblackshear/frigate:0.4.0-beta

I have seen that happen on older versions, but not lately. The next version will probably fix it. I should be releasing a new beta in the next few weeks.

Hi Blake

I’ve encountered an error which I have not seen reported before (unless it’s related to the memory leak reported by hawkeye). Thought you might be interested.

v0.4.0-beta has been running perfectly on my Pi4 for a while. I recently added an extra camera running yi-hack-v4. It works fine and detects objects etc but after a number of hours frigate dies. I’ve removed it again now and frigate is once again stable. Please let me know if you want any more info. The new camera is the only RTSP one in the attached log.

2020-02-09T20:58:18.875611799Z  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
2020-02-09T22:14:57.347330541Z Exception in thread Thread-24:
2020-02-09T22:14:57.347408095Z Traceback (most recent call last):
2020-02-09T22:14:57.347433594Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:14:57.347456983Z     self.run()
2020-02-09T22:14:57.347478612Z   File "/opt/frigate/frigate/objects.py", line 96, in run
2020-02-09T22:14:57.347500593Z     self.camera.regions_in_process[frame['frame_time']] -= 1
2020-02-09T22:14:57.347522463Z KeyError: 1581286497.228188
2020-02-09T22:14:57.347543203Z 
2020-02-09T22:53:10.628104105Z Exception in thread Thread-47:
2020-02-09T22:53:10.628213400Z Traceback (most recent call last):
2020-02-09T22:53:10.628240473Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:53:10.628264195Z     self.run()
2020-02-09T22:53:10.628286325Z   File "/opt/frigate/frigate/video.py", line 87, in run
2020-02-09T22:53:10.628308546Z     raw_image = self.camera.ffmpeg_process.stdout.read(self.camera.frame_size)
2020-02-09T22:53:10.628330102Z MemoryError
2020-02-09T22:53:10.628350898Z 
2020-02-09T22:53:10.669584711Z Exception in thread Thread-48:
2020-02-09T22:53:10.669709895Z Traceback (most recent call last):
2020-02-09T22:53:10.669740172Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:53:10.669765653Z     self.run()
2020-02-09T22:53:10.669791171Z   File "/opt/frigate/frigate/video.py", line 87, in run
2020-02-09T22:53:10.669814133Z     raw_image = self.camera.ffmpeg_process.stdout.read(self.camera.frame_size)
2020-02-09T22:53:10.669836670Z MemoryError
2020-02-09T22:53:10.669862910Z 
2020-02-09T22:53:11.210122623Z Exception in thread Thread-50:
2020-02-09T22:53:11.210248380Z Traceback (most recent call last):
2020-02-09T22:53:11.210370359Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:53:11.210429840Z     self.run()
2020-02-09T22:53:11.210480876Z   File "/opt/frigate/frigate/video.py", line 87, in run
2020-02-09T22:53:11.210532282Z     raw_image = self.camera.ffmpeg_process.stdout.read(self.camera.frame_size)
2020-02-09T22:53:11.210582133Z MemoryError
2020-02-09T22:53:11.210631040Z 
2020-02-09T22:53:19.799636241Z Exception in thread Thread-49:
2020-02-09T22:53:19.799784220Z Traceback (most recent call last):
2020-02-09T22:53:19.799813053Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:53:19.799834071Z     self.run()
2020-02-09T22:53:19.799853608Z   File "/opt/frigate/frigate/video.py", line 87, in run
2020-02-09T22:53:19.799872904Z     raw_image = self.camera.ffmpeg_process.stdout.read(self.camera.frame_size)
2020-02-09T22:53:19.799892033Z MemoryError
2020-02-09T22:53:19.800159306Z 
2020-02-09T22:58:15.604241208Z side: last frame is more than 5 minutes old, restarting camera capture...
2020-02-09T22:58:15.604396539Z Terminating the existing ffmpeg process...
2020-02-09T22:58:15.604674960Z Waiting for ffmpeg to exit gracefully...
2020-02-09T22:58:15.604837050Z back: last frame is more than 5 minutes old, restarting camera capture...
2020-02-09T22:58:15.605214543Z Terminating the existing ffmpeg process...
2020-02-09T22:58:15.605296782Z Waiting for ffmpeg to exit gracefully...
2020-02-09T22:58:15.821632256Z front: last frame is more than 5 minutes old, restarting camera capture...
2020-02-09T22:58:15.821739106Z Terminating the existing ffmpeg process...
2020-02-09T22:58:15.821794402Z Waiting for ffmpeg to exit gracefully...
2020-02-09T22:58:25.562221403Z frontdoor: last frame is more than 5 minutes old, restarting camera capture...
2020-02-09T22:58:25.562350919Z Terminating the existing ffmpeg process...
2020-02-09T22:58:25.562471047Z Waiting for ffmpeg to exit gracefully...
2020-02-09T22:58:45.605057620Z FFmpeg didnt exit. Force killing...
2020-02-09T22:58:45.605778107Z FFmpeg didnt exit. Force killing...
2020-02-09T22:58:45.619697825Z Waiting for the capture thread to exit...
2020-02-09T22:58:45.620511477Z Waiting for the capture thread to exit...
2020-02-09T22:58:45.621344166Z Creating a new ffmpeg process...
2020-02-09T22:58:45.621633235Z ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -r 6 -use_wallclock_as_timestamps 1 -i rtmp://...82:1935/bcs/channel0_sub.bcs?channel=0&stream=0&user=...&password=... -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-02-09T22:58:45.622489683Z Creating a new ffmpeg process...
2020-02-09T22:58:45.622630477Z ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -r 6 -use_wallclock_as_timestamps 1 -i rtmp://...81:1935/bcs/channel0_sub.bcs?channel=0&stream=0&user=...&password=... -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-02-09T22:58:45.645460982Z Exception in thread Thread-23:
2020-02-09T22:58:45.645575425Z Traceback (most recent call last):
2020-02-09T22:58:45.645630016Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:58:45.645682775Z     self.run()
2020-02-09T22:58:45.645733681Z   File "/opt/frigate/frigate/video.py", line 70, in run
2020-02-09T22:58:45.645784458Z     self.camera.start_or_restart_capture()
2020-02-09T22:58:45.645834809Z   File "/opt/frigate/frigate/video.py", line 271, in start_or_restart_capture
2020-02-09T22:58:45.645886067Z     self.start_ffmpeg()
2020-02-09T22:58:45.645935437Z   File "/opt/frigate/frigate/video.py", line 291, in start_ffmpeg
2020-02-09T22:58:45.645985325Z     self.ffmpeg_process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, bufsize=self.frame_size)
2020-02-09T22:58:45.646035139Z   File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
2020-02-09T22:58:45.646084953Z     restore_signals, start_new_session)
2020-02-09T22:58:45.646133767Z   File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child
2020-02-09T22:58:45.646183321Z     restore_signals, start_new_session, preexec_fn)
2020-02-09T22:58:45.646231524Z OSError: [Errno 12] Cannot allocate memory
2020-02-09T22:58:45.646833051Z Exception in thread Thread-12:
2020-02-09T22:58:45.646922068Z Traceback (most recent call last):
2020-02-09T22:58:45.646977511Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:58:45.647030917Z     self.run()
2020-02-09T22:58:45.647081157Z   File "/opt/frigate/frigate/video.py", line 70, in run
2020-02-09T22:58:45.647131675Z     self.camera.start_or_restart_capture()
2020-02-09T22:58:45.647180470Z   File "/opt/frigate/frigate/video.py", line 271, in start_or_restart_capture
2020-02-09T22:58:45.647230340Z     self.start_ffmpeg()
2020-02-09T22:58:45.647278598Z   File "/opt/frigate/frigate/video.py", line 291, in start_ffmpeg
2020-02-09T22:58:45.647328097Z     self.ffmpeg_process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, bufsize=self.frame_size)
2020-02-09T22:58:45.647377134Z   File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
2020-02-09T22:58:45.647426114Z     restore_signals, start_new_session)
2020-02-09T22:58:45.647474317Z   File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child
2020-02-09T22:58:45.647523705Z     restore_signals, start_new_session, preexec_fn)
2020-02-09T22:58:45.647572038Z OSError: [Errno 12] Cannot allocate memory
2020-02-09T22:58:45.647677869Z 
2020-02-09T22:58:45.647998697Z 
2020-02-09T22:58:45.821947336Z FFmpeg didnt exit. Force killing...
2020-02-09T22:58:45.836512431Z Waiting for the capture thread to exit...
2020-02-09T22:58:45.837751224Z Creating a new ffmpeg process...
2020-02-09T22:58:45.838158532Z ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -r 6 -use_wallclock_as_timestamps 1 -i rtmp://...83:1935/bcs/channel0_sub.bcs?channel=0&stream=0&user=...&password=... -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-02-09T22:58:45.844479754Z Exception in thread Thread-45:
2020-02-09T22:58:45.844581326Z Traceback (most recent call last):
2020-02-09T22:58:45.844635436Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:58:45.844687417Z     self.run()
2020-02-09T22:58:45.844735935Z   File "/opt/frigate/frigate/video.py", line 70, in run
2020-02-09T22:58:45.844785545Z     self.camera.start_or_restart_capture()
2020-02-09T22:58:45.844833951Z   File "/opt/frigate/frigate/video.py", line 271, in start_or_restart_capture
2020-02-09T22:58:45.844883395Z     self.start_ffmpeg()
2020-02-09T22:58:45.845013152Z   File "/opt/frigate/frigate/video.py", line 291, in start_ffmpeg
2020-02-09T22:58:45.845078003Z     self.ffmpeg_process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, bufsize=self.frame_size)
2020-02-09T22:58:45.845128798Z   File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
2020-02-09T22:58:45.845178871Z     restore_signals, start_new_session)
2020-02-09T22:58:45.845239241Z   File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child
2020-02-09T22:58:45.845512217Z     restore_signals, start_new_session, preexec_fn)
2020-02-09T22:58:45.845572087Z OSError: [Errno 12] Cannot allocate memory
2020-02-09T22:58:45.845632104Z 
2020-02-09T22:58:55.562779422Z FFmpeg didnt exit. Force killing...
2020-02-09T22:58:55.576171390Z Waiting for the capture thread to exit...
2020-02-09T22:58:55.577284130Z Creating a new ffmpeg process...
2020-02-09T22:58:55.577427035Z ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -r 6 -use_wallclock_as_timestamps 1 -i rtsp://...84/ch0_1.h264 -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-02-09T22:58:55.583571630Z Exception in thread Thread-34:
2020-02-09T22:58:55.583681776Z Traceback (most recent call last):
2020-02-09T22:58:55.583740294Z   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
2020-02-09T22:58:55.583797126Z     self.run()
2020-02-09T22:58:55.583850162Z   File "/opt/frigate/frigate/video.py", line 70, in run
2020-02-09T22:58:55.583903439Z     self.camera.start_or_restart_capture()
2020-02-09T22:58:55.584016122Z   File "/opt/frigate/frigate/video.py", line 271, in start_or_restart_capture
2020-02-09T22:58:55.584068380Z     self.start_ffmpeg()
2020-02-09T22:58:55.584117472Z   File "/opt/frigate/frigate/video.py", line 291, in start_ffmpeg
2020-02-09T22:58:55.584166564Z     self.ffmpeg_process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, bufsize=self.frame_size)
2020-02-09T22:58:55.584217174Z   File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
2020-02-09T22:58:55.584265877Z     restore_signals, start_new_session)
2020-02-09T22:58:55.584314135Z   File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child
2020-02-09T22:58:55.584362857Z     restore_signals, start_new_session, preexec_fn)
2020-02-09T22:58:55.584411300Z OSError: [Errno 12] Cannot allocate memory
2020-02-09T22:58:55.584458948Z 

The changes I am making for the next release will likely address these types of issues.

1 Like

It there a switch that allows this to run without the Coral to try out before buying one?

There will be in the next version

2 Likes

I’m getting weird issues with the labels/ tags on the images served to the camera endpoint - the text disappears fully or partially:

Screenshot 2020-02-20 at 16.21.28

Any ideas?

I have seen that too. I am almost finished with the next version which should resolve this.

1 Like

Out of curiousity, what are the highlights of the next version? Sounds like a whopper!

It is a major overhaul. I added motion detection back in and eliminated the need for defined regions completely. The motion detection is capable of processing 100fps at 1080p per camera on my laptop, so the overhead is low. This also means that the Coral isn’t wasting cycles checking for objects when there is nothing happening. The Coral now runs in a dedicated process which means the overhead of the motion detection and object tracking does not impact Coral speeds. On 0.4.0, my system was maxing out at ~60fps on the coral with 7 cameras due to all the overhead of object tracking. Now I am seeing over 100fps when maxed out. I also simplified large portions of the codebase which have eliminated many of the little edge case bugs.

16 Likes

Looking forward to this update.

Yes! This is the one I’ve been waiting for! :smiley: