Local realtime person detection for RTSP cameras

ringmod · June 24, 2019, 8:34am

ok, so now the Raspberry Pi 4 is unveiled with USB3, I would expect it to be the new base level for Frigate builds. What are the memory demands on running Frigate? Would there be real benefits to go for 2 GB or even 4 GB? Do you have a good view of where the bottlenecks are in these kinds of cheap platforms?

This seems very promising:

blakeblackshear · June 24, 2019, 11:19am

I agree that looks promising. I have been testing the Odroid-XU4, but the USB3 bus speeds seem to be limiting me to about 20ms for inference times. Next on my list is the Atomic Pi. Memory demands are low at the moment because I am only storing 1 or 2 frames from each camera at a time. They could go up when I start outputting video clips. I am specifically trying to optimize for these kinds of low power SBCs.

blakeblackshear · June 24, 2019, 11:25am

I just ordered a 4GB model for testing.

scstraus · June 24, 2019, 8:34pm

Finally trying to get the Coral version up and running and getting this, any ideas what it means? I’d assume the 401 is authentication error logging into the cameras but I’m using same URL and uname/pw which worked on the old frigate version and in VLC… Unless I somehow defined it wrong with the new config format.

Starting frigate ... done
Attaching to frigate
frigate    | On connect called
frigate    | [rtsp @ 0x3446b60] method DESCRIBE failed: 401 Unauthorized
frigate    | Traceback (most recent call last):
frigate    |   File "detect_objects.py", line 99, in <module>
frigate    |     main()
frigate    |   File "detect_objects.py", line 53, in main
frigate    |     cameras[name] = Camera(name, config, prepped_frame_queue, client, MQTT_TOPIC_PREFIX)
frigate    |   File "/opt/frigate/frigate/video.py", line 137, in __init__
frigate    |     self.frame_shape = get_frame_shape(self.rtsp_url)
frigate    |   File "/opt/frigate/frigate/video.py", line 102, in get_frame_shape
frigate    |     frame_shape = frame.shape
frigate    | AttributeError: 'NoneType' object has no attribute 'shape'

uid0 · June 24, 2019, 9:52pm

Average inference time: 16.277397933006288

Not as good as yours, but not horrible - 8ms vs my 16ms means I can only run half the number of images…

ivelin · June 24, 2019, 10:05pm

Thank you for the prompt response.

Streaming media ingestion is a topic I am currently researching. There are research models that are trained directly on video, which I’m looking at. The popular ones who are trained on images might be also possible to reuse with transfer learning, but that’s just theory at this point. I don’t have answers yet, just wanted to ask the question before looking further.

As you pointed out, under 1 second detection is quite good. Have you had a chance to push further and benchmark performance for face detection (visitor identification)?

You mention that h264 decoding is hardware accelerated. Could you please share which Raspberry Pi add on you used for that? Sorry if I missed this info in your earlier notes.

Thanks again for sharing your great work on Frigate.

blakeblackshear · June 25, 2019, 11:14am

Which version are you running?

blakeblackshear · June 25, 2019, 11:26am

I haven’t looked into face detection yet. The Coral can only load a single model at a time, so it is best to have a second Coral if you are going to have multiple models. I would assume adding face recognition would take an additional 10-15ms, and I would only do it for frames where a person was detected.

For hardware accelerated h264 decoding, I am just using ffmpeg directly within frigate. The flags you pass to ffmpeg vary depending on your hardware, but -c:v h264_mmal should work on an RPi. I am not running frigate on an Raspberry Pi because you really need USB 3.0 speeds for the Coral. I have a RPi 4 on order, so we will see if that is feasible.

scstraus · June 25, 2019, 6:41pm

The latest, I pulled it and built it 4 days ago on the 21st.

Okay, now I have it doing something, but whenever I try to open the camera URL at http://frigate:5000/cameraname I get this:

frigate    | 10.0.0.4 - - [25/Jun/2019 20:58:40] "GET /back HTTP/1.1" 500 -
frigate    | Error on request:
frigate    | Traceback (most recent call last):
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 303, in run_wsgi
frigate    |     execute(self.server.app)
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 293, in execute
frigate    |     for data in application_iter:
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/wsgi.py", line 507, in __next__
frigate    |     return self._next()
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
frigate    |     for item in iterable:
frigate    |   File "/opt/frigate/detect_objects.py", line 88, in imagestream
frigate    |     frame = cameras[camera_name].get_current_frame_with_objects()
frigate    |   File "/opt/frigate/frigate/video.py", line 299, in get_current_frame_with_objects
frigate    |     (region['x_offset']+region['size'], region['y_offset']+region['size']),
frigate    | TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
frigate    | 10.0.0.4 - - [25/Jun/2019 20:58:40] "GET /favicon.ico HTTP/1.1" 500 -
frigate    | Error on request:
frigate    | Traceback (most recent call last):
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 303, in run_wsgi
frigate    |     execute(self.server.app)
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 293, in execute
frigate    |     for data in application_iter:
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/wsgi.py", line 507, in __next__
frigate    |     return self._next()
frigate    |   File "/usr/local/lib/python3.5/dist-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
frigate    |     for item in iterable:
frigate    |   File "/opt/frigate/detect_objects.py", line 88, in imagestream
frigate    |     frame = cameras[camera_name].get_current_frame_with_objects()
frigate    | KeyError: 'favicon.ico'

uid0 · June 25, 2019, 9:47pm

is your $RTSP_PASSWORD environment variable getting set correctly?

blakeblackshear · June 25, 2019, 10:28pm

Both of those errors look like you requested a path that isn’t linked to a camera. I see /favicon.ico and /back. Make sure you have x_offset, y_offset, and size defined correctly for each region. It looks like something is missing or has a typo.

scstraus · June 26, 2019, 9:13pm

The URL I’m trying to access is http://frigate_ip:5000/back

The browser gives me back

# Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Here’s my config (I actually have 4 cameras, but I’ve commented out 3 of them for testing). As you can see, the camera is called back same as in the failing URL. I can access this stream with VLC in the format

rtsp://USERNAME:PASSWORD@MY SYNOLOGY IP:554/Sms=4.unicast

web_port: 5000

mqtt:
  host: MQTT_HOST
  topic_prefix: frigate
  user: MQTT_USER
  password: MQTT_PASS

cameras:
  back:
    rtsp:
      user: USERNAME
      host: MY SYNOLOGY IP
      port: 554
      password: $RTSP_PASSWORD_BACK
      path: /Sms=4.unicast
    mask: 1080Back.bmp
    regions:
      - size: 1080
        x_offset: 0
        y_offset: 0
        min_person_area: 9000
        threshold: 0.2
      - size:
        x_offset: 839
        y_offset: 0
        min_person_area: 9000
        threshold: 0.2

And my docker-compose

version: "3"
services:

  frigate:
    container_name: frigate
    restart: unless-stopped
    privileged: true
    image: frigate:latest
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /root/frigate/config:/config
    ports:
      - 5000:5000
    environment:
      - RTSP_PASSWORD=PASSWORD
      - RTSP_PASSWORD_RIGHT=PASSWORD
      - RTSP_PASSWORD_LEFT=PASSWORD
      - RTSP_PASSWORD_FRONT=PASSWORD
      - RTSP_PASSWORD_BACK=PASSWORD

I’ve tried lots of password formats, but the above is the only one that doesn’t give me a 401.

blakeblackshear · June 26, 2019, 9:32pm

The second region in that config snippet does not have a value for size.

scstraus · June 26, 2019, 9:45pm

Argh, I’m an idiot. Sorry for wasting your time. It works now.

scstraus · June 27, 2019, 12:26am

So, it’s working okay for 1 or maximum 2 cameras, but when I turn on more than about that (36-48fps at 1080p), I get something like the below. What do you think, it just can’t pull it off? It’s an i7 with usb 3.0 and on the old version it was able to run the same 8 regions without problem off the cpu…

Attaching to frigate
frigate    | On connect called
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Creating a new capture process...
frigate    | Starting a new capture process...
frigate    | W third_party/darwinn/driver/package_registry.cc:65] Minimum runtime version required by package (5) is lower than expected (10).
frigate    | Capture process for back: 56
frigate    | Creating a new capture process...
frigate    | Starting a new capture process...
frigate    | Capture process for front: 60
frigate    | Creating a new capture process...
frigate    | Starting a new capture process...
frigate    | Capture process for right: 64
frigate    | Creating a new capture process...
frigate    | Starting a new capture process...
frigate    | Capture process for left: 68
frigate    |  * Serving Flask app "detect_objects" (lazy loading)
frigate    |  * Environment: production
frigate    |    WARNING: This is a development server. Do not use it in a production deployment.
frigate    |    Use a production WSGI server instead.
frigate    |  * Debug mode: off
frigate    |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Opening the RTSP Url...
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Opening the RTSP Url...
frigate    | Invalid UE golomb code
frigate    | Opening the RTSP Url...
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Opening the RTSP Url...
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | [h264 @ 0x30c7740] left block unavailable for requested intra4x4 mode -1 at 0 40
frigate    | [h264 @ 0x30c7740] error while decoding MB 0 40, bytestream 39810
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | [h264 @ 0x2ce2ae0] cabac decode of qscale diff failed at 118 15
frigate    | [h264 @ 0x2ce2ae0] error while decoding MB 118 15, bytestream 97830
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | queue full. moving on
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | [h264 @ 0x2ecc8c0] error while decoding MB 37 76, bytestream -30
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | Unable to grab a frame
frigate    | [h264 @ 0x2d1a2c0] error while decoding MB 96 54, bytestream -10
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | [h264 @ 0x2d1e520] left block unavailable for requested intra mode at 0 17
frigate    | [h264 @ 0x2d1e520] error while decoding MB 0 17, bytestream 858
frigate    | Invalid UE golomb code
frigate    | Invalid UE golomb code
frigate    | [h264 @ 0x2ce2ae0] error while decoding MB 86 44, bytestream -12

blakeblackshear · June 27, 2019, 12:54am

How many regions per camera? It should be able to look at about 100 regions per second. Also, the CPU version just dropped frames silently if it wasn’t able to process them in time. I would suggest lowering the frame rate of the feed Frigate is looking at. 5-10 fps is plenty from my testing. I don’t think anyone is going to get by in 0.2 seconds.

scstraus · June 27, 2019, 10:11am

It’s 6fps per camera, 2 zones per camera, 4 cameras… So 48 regions per second… I thought it would make short work of it. I’m starting to fear that the USB pass-through on VMware isn’t up to the task. I could theoretically move it to my synology but I’m not sure I want to use that much of the bus bandwidth on this…

So I tried bumping them down to 1fps (so 8fps with all 8 zones total), and was able to get it to show the camera feeds (slowly), but no bounding boxes… Log is still full of “Unable to grab a frame”. Interesting, because I was successfully running 12FPS on a single camera using 2 zones of 6fps each.

I will wait for your results on the Pi4… I’m cautiously optimistic that that will be the final solution.

blakeblackshear · June 28, 2019, 2:51am

Finally got some time to run a benchmark on the Atomic Pi, and it is coming in at about 16ms. Bummer. Really hoping the RPi4 us under 10ms, but I am not holding my breath.

Here are the devices I have tested so far:

Raspberry Pi 3b+ - 50ms (10-20fps)
Odroid-XU4 - 20ms (25-50fps)
Atomic Pi - 16ms
NUC/Laptop - 5ms (100-200fps)

steve1 · June 28, 2019, 3:46am

i’m using this within docker on a Synology NAS DS918+ , with 40-50 fps i get;

root@74bb380ce342:/opt/frigate# python3 benchmark.py 
Average inference time: 6.506533992767334

Incidentally, running the script crashes the container with nothing shown in the docker logs (ie it seems to crash/kill the container immediately )

scstraus · June 28, 2019, 8:48pm

Does 16ms on the Atomic equate to 40-50fps? If so, I’m guessing a Pi4 will be adequate for my needs and I will just order it.

On a side note, is there any reason why frigate might choke on a camera feed that’s in portrait rather than landscape orientation (ie 1080x1920 rather than 1920x1080)? I tried starting it on one of my portrait orientation cameras and it is just freezing…