Object detection for video surveillance

asmirnou · June 21, 2020, 11:04am

Watsor

Watsor detects objects in video stream using deep learning-based approach. Intended primarily for surveillance it works in sheer real-time analysing the most recent frame to deliver fastest reaction against a detected threat.

What it does

Performs smart detection based on artificial neural networks significantly reducing false positives in video surveillance.
Capable to limit detection zones using mask image with alpha channel.
Supports multiple hardware accelerators such as The Coral USB Accelerator and Nvidia CUDA GPUs to speed up detection algorithms.
Reports the detected objects via MQTT protocol primarily for integration with HomeAssistant.
Allows to control video decoder using the commands published over MQTT.
Broadcasts video stream with rendered object detections in MPEG-TS and Motion JPEG formats over HTTP.
Captures video from any source and encodes video with rendered object detections in any format supported by FFmpeg.

Being applicable in CCTV, Watsor also suits other areas, where object detection in video stream is required.

sender · June 21, 2020, 11:17am

Cool! Will this become an “addon”?

asmirnou · June 21, 2020, 11:39am

Good point. It probably will.

Airbag888 · June 21, 2020, 6:21pm

What kind of training model is it using? Would you be able to add more of your own down the line such as your family’s faces, your pets and car plates. Also would be nifty to flag moments it sees people walking near your property (either when no one is home or between 10pm-4am) and also if it can spot someone climbing your wall

Would love to see it monitor the 2nd channel of my ip cctvs and send texts when someone I know is around for example

Anyways some thoughts

asmirnou · June 21, 2020, 8:40pm

Right, all of that would be nice to have, hopefully we’ll get to that point.
It uses Single-Shot Detector (SSD) detector trained to recognize 90 classes of object from COCO dataset. The model can be trained further to recognize custom objects and probably can be smart enough to distinguish the owner from the strangers. The faces and car plates require another models, however.

Airbag888 · June 21, 2020, 8:47pm

It’s pretty exciting and if the framework is there to grow upon then it’s even better
I’m thinking it should be rather easy to obtain the requisite dataset of familiar faces pictures to train and also make it able to read plates… of course easy is ‘relative’

chairstacker · June 22, 2020, 2:01am

Hi @asmirnou - is there a docker version that will run on a Pi4?

katsaplias · June 22, 2020, 9:11pm

seems very interesting, but for my setup I need the addon and support for Intel Neural Compute Stick 2

Eeeeeediot · June 22, 2020, 10:12pm

I for one would love if someone could write a guide on how to install this on a Pi4 with Hassio!

asmirnou · June 25, 2020, 5:06pm

Hi @chairstacker.
Do you mind to test the image built for Raspberry Pi 4 64-bit version?
Bear in mind that object detection as such requires much computation. To get decent performance on a device such as Raspberry Pi you need the Coral USB accelerator. The image supports USB devices with the help of UDEV flag (by adding ENV UDEV=1).

Unfortunately, I haven’t got a chance to run the image on a real device, but emulated and tested as much as I could. If your Raspberry Pi is based on ARM 32, you can tweak the sources and build your own image. The build process takes 6 hours though.

chairstacker · June 25, 2020, 7:13pm

I had set up Tensorflow on an RPi4 4GB according to these instructions and it worked great, no performance issues:

Currently my Pi is not set up with as 64-bit machine but I’d be happy to dig out an SD Card and give it a whirl. Which OS flavor would you recommend to use for this, do you maybe have a link?

Re. building my own container for the 32-bit, I just don;t have the skills do do that, sorry.

asmirnou · June 25, 2020, 7:25pm

I hesitate to recommend as I haven’t tried it myself, but may be this one: https://www.balena.io/os/docs/raspberrypi4-64/getting-started/

asmirnou · June 25, 2020, 7:30pm

Hi @katsaplias,

Thanks for feedback. I created a feature request to add the support for Intel Neural Compute Stick 2, but honestly there are no plans to work on it at the moment unless someone decides to contribute to the project. People can vote on this issue to raise the interest to it. I’ll consider adding the support if it gets 100 positive reactions.

chairstacker · June 25, 2020, 7:42pm

Thanks!
I’ll give it a try later today.

chairstacker · June 26, 2020, 12:49am

Hi @asmirnou,
tried balena, didn’t work; the image I downloaded and installed didn’t open any port for SSH access and the display only showed the balena logo, nothing else.

Using the Raspbian 64bit version now; installed Docker and Docker Compose but I’m always getting the same error message when trying to start the container:

MainThread root ERROR : [Errno 2] No such file or directory: '/etc/watsor/config.yaml'
watsor exited with code 1

When I first went to /etc the directory watsor did not exist, so I created it and added a config.yaml file with some of your dummy-entries from github.

This is what my docker-compose.yaml looks like:

version: '3'
services:
  watsor:
    container_name: watsor
    image: smirnou/watsor.pi4:latest
    environment:
      - LOG_LEVEL=info
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ../config:/etc/watsor:ro
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri:/dev/dri
    ports:
      - 8080:8080
    shm_size: 512m

I’m still getting the message: No such file or directory

asmirnou · June 26, 2020, 4:33am

Right, the ../config folder was supposed to be relative to the location of your docker-compose.yaml file. That directory, located at the Raspberry file system, is mounted in Docker container file system at /etc/watsor. As far as your put the config in /etc/watsor of Raspberry file system, you now can mount is as follows:

volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/watsor:/etc/watsor:ro

Configuration file describes the cameras you’ve got and few other options such as types of detection and zones. Refer to the following guide.

chairstacker · June 26, 2020, 7:03am

Okay, thanks - made a little progress.

Latest error message is now:
ERROR : expected '<document start>', but found '<block mapping start>' in "/etc/watsor/config.yaml", line 18, column 1

This is the line where the ffmpeg definition starts.

Any other hints?

asmirnou · June 26, 2020, 7:59am

There is YAML syntax error somewhere above line 18, probably missing : or redundant #. Post your config (sanitized if some sensitive data is there).

chairstacker · June 26, 2020, 5:00pm

Okay - figured out how to configure my MQTT user with no password

Looks like the container is up & running now.

Now I ‘just’ need to figure out how to access the live stream from my Foscam FI9828P.
As I mentioned, I had it running for TensorFlow once before but it looks like I overwrote the configuration there

Simply adjusting your sample with my info doesn’t seem to do the trick - yet:

input: !ENV "rtsp://${RTSP_USERNAME}:${RTSP_PASSWORD}@192.168.0.10:554/cam/realmonitor?channel=1&subtype=2"

EDIT:
The URL rtsp://user:[email protected]:88/videoMain works fine in VLC bust throws an error in when I use it in the config like this:
input: !ENV "rtsp://user:[email protected]:88/videoMain"

These are the lines I see in the log:

watsor    | MainThread       werkzeug                 INFO    : Listening on ('0.0.0.0', 8080)
watsor    | MainThread       root                     INFO    : Starting Watsor on 3f38f402e268 with PID 13
watsor    | outdoorcam       FFmpegDecoder            INFO    : [AVHWDeviceContext @ 0x5572de8730] No VA display found for device: /dev/dri/renderD128.
watsor    | outdoorcam       FFmpegDecoder            INFO    : Device creation failed: -22.
watsor    | outdoorcam       FFmpegDecoder            INFO    : Device setup failed for decoder on input stream #0:0 : Invalid argument
watsor    | watchdog         WatchDog                 WARNING : Thread outdoorcam (FFmpegDecoder) is not alive, restarting...
watsor    | outdoorcam       FFmpegDecoder            INFO    : [AVHWDeviceContext @ 0x55aa605730] No VA display found for device: /dev/dri/renderD128.
watsor    | outdoorcam       FFmpegDecoder            INFO    : Device creation failed: -22.
watsor    | outdoorcam       FFmpegDecoder            INFO    : Device setup failed for decoder on input stream #0:0 : Invalid argument
watsor    | watchdog         WatchDog                 WARNING : Thread outdoorcam (FFmpegDecoder) is not alive, restarting...

asmirnou · June 26, 2020, 8:44pm

Remove the following hardware acceleration options for FFmpeg from config:

    - -hwaccel
    -  vaapi
    - -hwaccel_device
    -  /dev/dri/renderD128
    - -hwaccel_output_format
    -  yuv420p

They are not suitable for Raspberry.