I still get many false detection with tensorflow type, can we use other libraries, like these: https://github.com/tensorflow/models/blob/e54fcee236a1258302342bd703ee27cbba0c12e3/research/object_detection/g3doc/tf2_detection_zoo.md ? I had with tensorflow faster_rcnn_inception_v2_coco model, but had better detection.
But it is a very good component for HA, easy to use!
Great question. I just tried this yesterday and couldn’t get it to work. I probably missed something simple though.
I’ll have to sit down and try again.
I’m using the V2 edgetpu coco model with a coral on a pi4. Detection times are in the 0.01 to 0.02 sec range, but i can’t say it is any more accurate. It is convinced my weber grill and my neighbor’s super beetle are people.
Nonetheless, i love this thing. What a cool integration. I have it setup run through like five detections when my hikvision cams trigger on motion. I put a cool down in there to keep it from running away from me, but all in all, the pi handles it like a champ.
Edit: per how i got it working, i simply downloaded the model and labels files, then made sure to bind them to the container:
One thing i’ve noticed since switching to the coral: when i restart the pi, the doods container bugs out and responds to each service call with a 404. From the logs it looks like it is because it can’t find the coral. A restart of the container and it is back up and running.
Anyone else see this?
Hi,
@snowzach: i really like doods & the way you packaged!
- works out of the box
- REST API + pydoods allows testing, e.g. model performance
- (somewhat) good documented samples how to get it working
I have it running, meanwhile with edgeTPU (dump idea) - still some questions:
- for permanent monitoring a scan_interval =1 or smaller will require HW acceleration (or a weak model)
- Coral Edge TPU doesn’t seem worth its money for this. Its very quick, but snowzach is right - its just to unreliable with the models supported
- Trigger doods based on cam motion alert - start detection / recording - I don’t see how that would work, as its involving doods ‘after the fact’ only?
- I’m imagining a 24/7 stream detection as alternative (also would make a nice showcase) - anybody went down that path? I know its going to lean heavy on the HW.
- Nvidia GPU acceleration feels like best model support. Anybody has that running with a ‘budget’ card?
- faster_rcnn_inception_v2_coco seems to have better detection - but is in the 15s+ on my j4105 CPU
- Doods doesn’t take rtsp streams - only still images, right?
Would be very cool to move the ‘draw rectangle’ function into the container. That would allow easy testing of models via RESTApi. Alternativly a pydoods sample on how to test & draw the resulting rectangles in python afterwards might be nice.
Do you accept contributions?
Where I’m coming from:
- Couple of foscam & new reolink around the house
- Motion detection of cameras to FTP
- Postprocessing with OpenCv: compare n-1 + n image, convert both to b&w, highlight brightness differences, define min / max rectange size, mask areas = filter down pictures by ~75%)
- Tried to train custom model, liked Google Collabs, but the effort (per cam?) kills my available time. Models / code still seem to change
ItsMee
Posting some files as I always enjoy others showing stuff that works well for them.
Supersimple test - takes filename, returns result. Missing rectangle drawing:
cat /root/doods/test.py
#! /usr/bin/python3from pydoods import PyDOODS
import sys
import jsontry:
pd = PyDOODS(“http://192.168.177.75:8080”,
auth_key=“abc1234”, timeout=90)
except OSError as err:
print(“OS error: {0}”.format(err))
exit
#image = open(“grace_hopper.png”, “rb”).read()
image = open(sys.argv[1], “rb”).read()dconfig={}
dconfig[“car”] = 20
dconfig[“person”] = 10detector_name=“default”
result=pd.detect(image, dconfig, detector_name)
print(json.dumps(result, indent=2, sort_keys=True))
Dockerfile + docker-compose.yml
cat Dockerfile
FROM snowzach/doods:latestRUN apt-get update && apt-get -y install vim
RUN wget https://raw.githubusercontent.com/snowzach/doods/master/fetch_models.sh
RUN . ./fetch_models.sh
docker-compose.yml
cat docker-compose.yml
version: ‘2.4’services:
iot-homeassistant-doods:
container_name: iot-home-assistant-doods
image: iot-home-assistant-doods
build: ./volumes: - ./config/opt/doods/config.yaml:/opt/doods/config.yaml devices: - /dev/bus/usb:/dev/bus/usb privileged: true ports: - 8080:8080 cpus: 2
@snowzach
I’ve got a silly question. What should I do to pass Coral accelerator to the DOODS docker (hass.io add-on) when I’m running it on HassOS?
I’m getting following error in DOODS add-on logs:
2020-08-08T16:11:38.348Z ERROR detector/detector.go:73 Could not initialize detector mobilenet: could not initialize edgetpu /sys/bus/usb/devices/2-4.1 {“package”: “detector”}
I’m running HassOS with DOODS add-on under PROXMOX VE with USB passthrough enabled.
EDIT:
Ok, I managed to set it up by passing not USB port but specific device to my VM. Sorry, for the hassle.
Anyways, I’m not sure how to make Hass.io add-on restart itself if it’s being stopped by detector timeout. I’m not sure why that happens to me almost instantly after it detects an object.
I’m using only ‘person’ label, one tflite EdgeTPU detector and one camera/image_processing integration.
Sorry for potentially being a bit stupid, but I can’t for the life of me figure out how to get a snapshot taken, and getting DOODS to process that file on disk?
I’m using the latest docker image, with the below as my config.
I do have another automation that runs on motion (PIR) and saves 7 snapshots to disk (1 sec apart), and that gives me what I want to process, but how do I point DOODS to these snapshots on disk?
image_processing:
- platform: doods
scan_interval: 10000
url: "http://192.168.111.249:8124"
timeout: 60
detector: tensorflow #default
source:
- entity_id: camera.entrance_camera
file_out:
- "/share/motioneye/Camera2/{{ camera_entity.split('.')[1] }}_latest.jpg"
- "/share/motioneye/Camera2/{{ camera_entity.split('.')[1] }}_{{ now().strftime('%Y%m%d_%H%M%S') }}.jpg"
confidence: 50
# This global detection area is required for all labels
# area:
# # Exclude top 10% of image
# top: 0.1
# # Exclude right 5% of image
# right: 0.95
# # The entire detection must be inside this box
# covers: true
labels:
- name: person
confidence: 70
area:
# Exclude right 15% of image
right: 0.85
# Exclude left 25% of image
left: 0.25
#Exclude bottom 20% of image
bottom: 0.8
# Any part of the detection inside this area will trigger
covers: false
- name: dog
area:
# Exclude top 50% of image
top: 0.5
# Exclude right 15% of image
right: 0.85
# Exclude left 25% of image
left: 0.25
# All part of the detection inside this area will trigger
covers: true
- bird
I’ve done it by setting up a snapshot “camera” (like this), and using the same path in a snapshot command.
Doods is set up with that as one option, and I call that for processing.
I’ve just set up DOODS myself and those file_out files are only output once DOODS has processed the images and actually spotted something. By that I mean, if DOODS has detected (in your case) a person, dog or bird then it will output a file with the camera entity name appended with either “latest.jpg” or the time and date stamp.
You have your output files pointing at what looks like a specific camera folder, whereas the documentation here suggests outputting those files to the /tmp directory. Obviously this is a personal choice but is worth noting.
So, the configuration that you have provided is for image processing to utilise once DOODS has spotted something that matches your parameters (ie. the object is in the right area and DOODS is confident enough).
What I would suggest is, rather than having 7 snapshots sent to the disk, you have an automation that triggers DOODS to carry out image processing once motion is detected. This is what I’m currently using to do just that:
- id: '1596819411241'
alias: Driveway Person Detection
description: Run image processing when movement is seen on the driveway
trigger:
- entity_id: binary_sensor.motion_driveway
from: 'off'
platform: state
to: 'on'
condition:
- condition: template
value_template: '{{ ( as_timestamp(now()) - as_timestamp(state_attr(''automation.driveway_person_detection'',
''last_triggered'')) |int(0) ) > 300 }}'
action:
- data: {}
entity_id: image_processing.doods_driveway
service: python_script.driveway_doods
mode: single
So when my driveway motion sensor changes state from off to on (ie. movement is detected) and the automation hasn’t run in the last 5 minutes (personal preference) it then carries out the action (in my case, I have a Python script).
For your use case you could simply have multiple actions to trigger image processing 7 times with a one second delay between each action. For example:
EDIT: I’ve just spotted Tinkerer’s method above regarding snapshots which I somehow completely missed(!) so please feel free to ignore me! 
Hopefully an easy question from me - Is there a way for DOODS to output an image that is just the object it has detected (ie. the box it has drawn around a person)? I’ve got Home Assistant to send me the “xxx_latest.jpg” image as an iOS notification but obviously the image is of the whole camera view with the boxes drawn around them, which isn’t super helpful for the little thumbnail in the iOS notification!
Thanks in advance 
That’s pretty cool, thanks for sharing 
So to take a snapshop, you just call that snapshot command. If you want 10 snapshots, you just call that command 10 times which would fire the snapshot camera 10 times, and process it using DOODS 10 times? That’s cool.
So the next question would be if I may:
Is there a way to queue these up? I want to run this multiple times, 1 second apart. I don’t want to wait for the previous snapshot to be processed before taking the next snapshot. I basically want to take my 7 snapshots 1 second apart, and process them in parallel as soon as each of those snapshots are ready to be processed. DOODS can then come back to me once it’s done. At least I won’t miss a frame/snapshot as such.
I feel that if I run it as per your example, I would call the snapshot, it would say take 3 seconds round trip (which includes DOODS processing), then take another snapshot, and I’d have missed things happening in between.
We almost need a ‘watch’ folder for DOODS, and we can just dump files in there to be processed, and DOODS updates after each file update.
Thanks for your time!
Hey, thanks for answering.
I have it set up with the automation, to do as you suggested currently, but it can only take and process a snapshot, then take the next.
Using the automation, it will take longer than a second before it can take the next snapshot, so I will have a snapshot say every 3 seconds, which means I have missed things happening in between.
Heh the reason I’m going to a camera folder is, I have jerry rigged Motion Eye as a viewer of all these files. It handles the cleanup for me, and I can easily view all my snapshots from Home Assistant by day etc. Is there a better way to have all that handled that I may have missed?
Thanks for reading
Hi Guys,
Been tinkering a bit more recently, and found that my objects are all being mislabeled. I tried switching the labels files around, but they are identical, other than the one has a line number on the start of the file. I’m not sure why this is happening. Could it be when the models file was updated on Jun 24 (I think that was mentioned above as a comment in this thread, look for Jun 24 in here for further info). Maybe the model was updated, but we don’t have the latest labels for it?
Dog is now a person, a light is a fire hydrant, and my car is a motor cycle.
Any help would be appreciated.
logger:
level: debug
server:
port: 8124
doods:
detectors:
- name: tensorflow
type: tensorflow
modelFile: models/faster_rcnn_inception_v2_coco_2018_01_28.pb
labelFile: models/coco_labels0.txt
width: 224
height: 224
numThreads: 0
numConcurrent: 4
And my labels (both files identical other than the one has the index on the left, I tried both, one detects completely wrong, and the otherone detects person, but can’t detect anything else, gives me warning missing label.
0 person
1 bicycle
2 car
3 motorcycle
4 airplane
5 bus
6 train
7 truck
8 boat
9 traffic light
10 fire hydrant
12 stop sign
13 parking meter
14 bench
15 bird
16 cat
17 dog
18 horse
19 sheep
20 cow
21 elephant
22 bear
23 zebra
24 giraffe
26 backpack
27 umbrella
30 handbag
31 tie
32 suitcase
33 frisbee
34 skis
35 snowboard
36 sports ball
37 kite
38 baseball bat
39 baseball glove
40 skateboard
41 surfboard
42 tennis racket
43 bottle
45 wine glass
46 cup
47 fork
48 knife
49 spoon
50 bowl
51 banana
52 apple
53 sandwich
54 orange
55 broccoli
56 carrot
57 hot dog
58 pizza
59 donut
60 cake
61 chair
62 couch
63 potted plant
64 bed
66 dining table
69 toilet
71 tv
72 laptop
73 mouse
74 remote
75 keyboard
76 cell phone
77 microwave
78 oven
79 toaster
80 sink
81 refrigerator
83 book
84 clock
85 vase
86 scissors
87 teddy bear
88 hair drier
89 toothbrush
Have you tried downloading your own model files from google and just using those? It is a pretty painless process and would at least let you rule out the model being an issue. I assume you’ve also added the labels you’re most interested in to the yaml config of the image processor in home assistant configuration.yaml? I only get results on people because I only have the person label in my config:
- platform: doods
scan_interval: 10000
url: "http://192.168.1.205:8080"
timeout: 60
detector: edgetpu
source:
- entity_id: camera.camera_back_porch
name: DOODS - Back Porch
confidence: 55
file_out:
- "/config/www/doods_back_porch.jpg"
- "/config/www/doods_latest.jpg"
area:
left: 0.1
right: 0.9
labels:
- name: person
I could be completely wrong (so someone can feel free to correct me) but I was under the impression that you can send snapshots to DOODS in quick succession (ie. every second) and DOODS will process each image sequentially (ie. in the order that they arrived). The amount of time that DOODS takes to process those images may take a few seconds but the results would be the same I’d have thought? By that I mean that you send DOODS 3 snapshots in 3 seconds, it may take 3 seconds to process each snapshot but the DOODS entity in Home Assistant will change its state should any of those snapshots contain a recognised object?
Cleanup with Motion Eye probably makes sense, the only other way I could think of doing it would be via a cron on the host OS (depending on how you’re running HA). But I’d stick with what you’re doing if it works for you to be fair!
Heh, I tried to find models, but I found it more complicated than it needs to be. So, I’ve found a whole pile of model files on https://tfhub.dev/. Which ones should I use? Is there a guide on what to look for, or which types? So say I’ve now found a model file, where do I find the labels for that model file? I haven’t been able to find any labels on tfhub.dev at all?
I also checked the fetch_models.sh file to see where they get the labels, and the model comes from tfhub, but the labels come from some gist on the net. That’s why I’m having my doubts about the model and labels ‘ligning up’ correctly…
I’d be a lot more confident in this all if I could go to a page where you can fine the model, and the labels for that exact model file. Only then would it rule out that as the issue.
Heh, I’m new to TensorFlow, and have never played with AI, NNs, and such, so maybe I’m sounding very silly, but I’m pretty sure more people would have these questions?
As for the labels in HA, at first I entered person, dog, bird. Those led to no detections, so I thought ok, let’s remove those, so that it detects everything, and see what the labels that gets assigned are. Yup, it detected those things as completely different things.
Thanks guys!
Edit: Can you give me an idea of what to search for specifically? I want to use tensorflow, not tf lite, or anything else. I’m running this on a reasonably grunty workstation pc.
I can’t offer much help here. I’m using a coral, and google makes the models and label files easy to find and compare. I assumed there’d be something similar for other tensorflow models.
I really have no idea what i’m doing. I managed to figure out how to get docker up and running on a pi4, then poked around for a few days until i figured out one can map directories to containers. Then i reread the doods documentation like 50 times until it clicked that I can download something called a model file and something called a label file, put it in a folder, and point the container at it. Last i figured out that a simple yaml file in that directory then tells the container about the new model files.
So i think the hard part for you is figuring out what makes for a good model, get one that either has a label file or doesn’t need one (i think some dont, like the MobileNet SSD v2 model). After that, pointing the container to it is easy.
DOODS absolutely looks really interesting, THANKS, and I’m looking forward to replacing parts of my current BASH/Python surveillance camera workflow with it.
Before doing so - any of you experienced, or have deeper insight in why the Coral USBs can befome unresponsive/crash and the LED on them flash either rapidly or slowly ? Giving errors such as
RuntimeError: Error in device opening (/sys/bus/usb/devices/3-1.4)
when doing object detection? (need unplugging to wake up again)
(I’m running on a i3 “NUC-alike” PC USB 3 port (MSI Cubi), expect enough power on the port, which could be the reason for it to happen. Also been doing what I can to secure mutex on the Coral). The EdgeTPU driver reports:
Edge TPU Driver version: BuildLabel(COMPILER=5.4.0 20160609,DATE=redacted,TIME=redacted,CL_NUMBER=317268237), RuntimeVersion(13).
Will I see the same trouble with the Coral USB when using DOODS ?
Thanks!
i have the same problem. Using a Raspi 4 4GB with hassio. The default models are a bit to inacurate and the inception models won’t work.
have you found a better solution beside the default models?
I’ve started going down the TensorFlow2 avenue.
I liked DOODS because I could run it on another of my servers, but it’s making things hard to get going, so now I’m trying out the TensorFlow integration to see how that’s going. It has also been a little hit and miss, heh, so we’ll see in the next couple of weeks what the outcome of this experiment is.