New Custom Compontent - Image Processing - Object Detection - DOODS

snowzach · August 29, 2019, 1:11am

    file_out:
      - "/tmp/{% raw %}{{ camera_entity.split('.')[1] }}{% endraw %}_latest.jpg"
      - "/tmp/{% raw %}{{ camera_entity.split('.')[1] }}_{{ now().strftime('%Y%m%d_%H%M%S') }}{% endraw %}.jpg"

andreasfelder · September 1, 2019, 12:10am

Sorry for the delayed response. Was traveling.
I tried to set the confidence to 10 in the config as you can see below. I also looked at the detection with the debug and it is showing multiple objects detected.

  - platform: doods
    scan_interval: 10000
    url: "http://192.168.1.15:8080"
    detector: default
    file_out:
      - "/home/homeassistant/.homeassistant/www/tmp/{{ camera_entity.split('.')[1] }}_latest.jpg"
    source:
      - entity_id: camera.driveway
      - entity_id: camera.frontdoor
    confidence: 10
    labels:
      - name: person
        confidence: 10
      - car

Log:

2019-08-31T23:59:34.760Z        DEBUG   detector/detector.go:58 Configuring detector    {"package": "detector", "config": {"name":"default","type":"tflite","model_file":"models/coco_ssd_mobilenet_v1_1.0_quant.tflite","label_file":"models/coco_labels0.txt","num_threads":4,"num_concurrent":4,"hw_accel":false}}
INFO: Initialized TensorFlow Lite runtime.
2019-08-31T23:59:34.898Z        INFO    detector/detector.go:76 Configured Detector     {"package": "detector", "name": "default", "type": "tflite", "model": "models/coco_ssd_mobilenet_v1_1.0_quant.tflite", "labels": 80, "width": 300, "height": 300}
2019-08-31T23:59:34.908Z        INFO    server/server.go:222    API Listening   {"package": "server", "address": ":8080", "tls": false}
2019-09-01T00:02:42.919Z        DEBUG   tflite/detector.go:165  Got Image       {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 1066, "height": 600}
2019-09-01T00:02:42.920Z        DEBUG   tflite/detector.go:167  Resizing Image  {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 300, "height": 300}
2019-09-01T00:02:44.207Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 66.796875, "location": "%!d(float32=0.055098265),%!d(float32=0.43829048),%!d(float32=0.48801824),%!d(float32=0.7392647)"}
2019-09-01T00:02:44.207Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 53.90625, "location": "%!d(float32=0),%!d(float32=0.3829388),%!d(float32=0.14547582),%!d(float32=0.48769468)"}
2019-09-01T00:02:44.207Z        INFO    tflite/detector.go:266  Detection Complete      {"package": "detector.tflite", "id": "", "duration": 1.255325322, "detections": 2}
2019-09-01T00:02:44.223Z        INFO    server/server.go:88     API Request     {"status": 200, "took": 1.363177218, "remote": "192.168.1.183:40404", "request": "/detect", "method": "POST", "package": "server.request", "request-id": "f749d7dafdd4/TmowE9Ti4r-000001"}
2019-09-01T00:05:02.430Z        INFO    server/server.go:88     API Request     {"status": 200, "took": 0.016561892, "remote": "192.168.1.183:40872", "request": "/detectors", "method": "GET", "package": "server.request", "request-id": "f749d7dafdd4/TmowE9Ti4r-000002"}
2019-09-01T00:05:25.202Z        DEBUG   tflite/detector.go:165  Got Image       {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 1066, "height": 600}
2019-09-01T00:05:25.202Z        DEBUG   tflite/detector.go:167  Resizing Image  {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 300, "height": 300}
2019-09-01T00:05:26.424Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 65.625, "location": "%!d(float32=0.055098265),%!d(float32=0.44297314),%!d(float32=0.48801824),%!d(float32=0.73458207)"}
2019-09-01T00:05:26.426Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 53.90625, "location": "%!d(float32=0),%!d(float32=0.38767767),%!d(float32=0.14507787),%!d(float32=0.48601562)"}
2019-09-01T00:05:26.429Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 50, "location": "%!d(float32=0),%!d(float32=0.22003245),%!d(float32=0.15798804),%!d(float32=0.43706542)"}
2019-09-01T00:05:26.431Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 39.0625, "location": "%!d(float32=0.052513093),%!d(float32=0.097026706),%!d(float32=0.98623765),%!d(float32=1)"}
2019-09-01T00:05:26.431Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 37.890625, "location": "%!d(float32=0.007793609),%!d(float32=0.47803453),%!d(float32=0.06524968),%!d(float32=0.5402155)"}
2019-09-01T00:05:26.431Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 34.375, "location": "%!d(float32=0.0070268214),%!d(float32=0.37261128),%!d(float32=0.18374865),%!d(float32=0.4977162)"}
2019-09-01T00:05:26.431Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 34.375, "location": "%!d(float32=0.048252463),%!d(float32=0.43728563),%!d(float32=0.58133113),%!d(float32=0.8578448)"}
2019-09-01T00:05:26.431Z        DEBUG   tflite/detector.go:263  Detection       {"package": "detector.tflite", "id": "", "label": "car", "confidence": 33.203125, "location": "%!d(float32=0.013728801),%!d(float32=0.37236094),%!d(float32=0.12890534),%!d(float32=0.41491246)"}
2019-09-01T00:05:26.431Z        INFO    tflite/detector.go:266  Detection Complete      {"package": "detector.tflite", "id": "", "duration": 1.201710235, "detections": 8}
2019-09-01T00:05:26.433Z        INFO    server/server.go:88     API Request     {"status": 200, "took": 1.260348486, "remote": "192.168.1.183:40944", "request": "/detect", "method": "POST", "package": "server.request", "request-id": "f749d7dafdd4/TmowE9Ti4r-000003"}

Please let me know. Thanks!!!

snowzach · September 1, 2019, 1:52am

Hi @andreasfelder just pushed a bunch of changes and fixes to the repo. Trying pulling down the latest and see if it fixes your issues. They are reviewing the code now to include it into 0.99

andreasfelder1 · September 1, 2019, 2:18am

Awesome will pull down the new version now and report back. Thanks!

andreasfelder · September 1, 2019, 3:06am

After updating to the new version. It still didn’t work however this time i started looking into it a bit more and realized i do have an error in the home assistant log. It showed a failure due to incorrect path. I fixed my file_out path and everything is working now.

Sorry to waste your time. Awesome component!!!

Darbos · September 1, 2019, 5:09am

Hey @snowzach, in the documentation for Dockers I noticed this image…

x86avx - This is an x64_64 with AVX and SSE4.2 features enabled. Use this for newer, more powerful x86

Is this for Nvidia gpus or just a newer i7?

I also just noticed you added the coral…that’s awesome man that little thing is amazing. Thanks again!

runningman84 · September 1, 2019, 10:46am

Does this component also recognize different persons?

snowzach · September 1, 2019, 1:20pm

Hey Darbos, there is no docker image that supports GPUs (yet). It’s the avx image supports more modern CPUs. Probably most Core chips support AVX extensions. It’s the really old and simple chips that do not support AVX. I use an Atomic Pi which has a Celeron J processor I think that does not support AVX.

snowzach · September 1, 2019, 1:21pm

@runningman84 if you trained a model to detect different people, it could. It’s really just a mechanism for running tensorflow models. You can train the models to recognize all kinds of stuff. Maybe someday I can add the training function to it.

Darbos · September 1, 2019, 1:27pm

Oh cool, thanks for the details man I appreciate it.

runningman84 · September 1, 2019, 2:30pm

It would be great if you would provide some hints how to train a model to recognize persons and how to integrate that here.

Darbos · September 2, 2019, 8:04pm

if you have a coral the instructions are pretty straight forward

Juggler · September 6, 2019, 11:20pm

Can you post the link to your push to 0.99?

runningman84 · September 7, 2019, 6:03am

runningman84 · September 7, 2019, 8:28pm

I have created an appdaemon app which triggers the image scanner based on sensors (like motion sensors or something like that):

github.com

runningman84/home-assistant-apps/blob/965a0d950a45f284331f5a7484d777545b69e794/apps/camera.py

import appdaemon.plugins.hass.hassapi as hass
from datetime import datetime
import time
import os

#
# CameraImageScanner App
#
# Args:
#


class CameraImageScanner(hass.Hass):

    def initialize(self):
        self.log("Hello from CameraImageScanner")

        self._processing_handle = None
        self._processing_count = 0
        self._processing_max_count = 10000

This file has been truncated. show original

Maybe this is also useful for other users.

xdss · September 11, 2019, 1:09am

Looks very promising, great work getting it as a component.

I have set up robs version as a more accurate alarm system by listening to the save file event and passing back bounding box information. In my case tracking if the x_min value dropped below a certain amount an automation will trigger.

Would it be possible integrate something similar into DOODS to set up some events to pass back information such as object coordinates and the camera which triggered.

see:

snowzach · September 11, 2019, 8:42pm

Hey @robmarkcole if I can get them to accept my merge, I’ll start adding features like events. I don’t want to interrupt the merge and lose my progress.

Silicon_Avatar · September 12, 2019, 1:46am

I might have missed this reading through the thread, but since DOODS is already a docker container, any plans to make it available as a HassIO add-on?

snowzach · September 12, 2019, 12:27pm

Yeah, I am working on the build process right now. Building tensorflow is kind of a pain, and building it for arm and aarch64 is even more of a pain. Once I have that working I will work on the hass.io thing next.

johntdyer · September 16, 2019, 6:57pm

I am running the laltest container on 0.99.b1 and I can see in the doods logs that it is detecting an object

Sep 16 14:49:53 ubuntu.local docker[23084]: 2019-09-16T18:49:53.132Z        DEBUG        tflite/detector.go:165        Got Image        {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 1920, "height": 1080}
Sep 16 14:49:53 ubuntu.local docker[23084]: 2019-09-16T18:49:53.132Z        DEBUG        tflite/detector.go:167        Resizing Image        {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 300, "height": 300}
Sep 16 14:49:53 ubuntu.local docker[23084]: 2019-09-16T18:49:53.527Z        DEBUG        tflite/detector.go:263        Detection        {"package": "detector.tflite", "id": "", "label": "car", "confidence": 50, "location": "%!d(float32=0.07881215),%!d(float32=0.25541472),%!d(float32=0.99492276),%!d(float32=0.96682394)"}
Sep 16 14:49:53 ubuntu.local docker[23084]: 2019-09-16T18:49:53.527Z        INFO        tflite/detector.go:266        Detection Complete        {"package": "detector.tflite", "id": "", "duration": 0.362563593, "detections": 1}
Sep 16 14:49:53 ubuntu.local docker[23084]: 2019-09-16T18:49:53.528Z        INFO        server/server.go:88        API Request        {"status": 200, "took": 0.473126473, "remote": "192.168.100.13:54048", "request": "/detect", "method": "POST", "package": "server.request", "request-id": "a5c7effbe9bb/FkKqxbcDyU-000003"}
Sep 16 14:51:54 ubuntu.local docker[23084]: 2019-09-16T18:51:54.138Z        DEBUG        tflite/detector.go:165        Got Image        {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 1920, "height": 1080}
Sep 16 14:51:54 ubuntu.local docker[23084]: 2019-09-16T18:51:54.138Z        DEBUG        tflite/detector.go:167        Resizing Image        {"package": "detector.tflite", "id": "", "format": "jpeg", "width": 300, "height": 300}
Sep 16 14:51:54 ubuntu.local docker[23084]: 2019-09-16T18:51:54.564Z        DEBUG        tflite/detector.go:263        Detection        {"package": "detector.tflite", "id": "", "label": "car", "confidence": 51.171875, "location": "%!d(float32=0.09170675),%!d(float32=0.25541472),%!d(float32=0.9934515),%!d(float32=0.96682394)"}
Sep 16 14:51:54 ubuntu.local docker[23084]: 2019-09-16T18:51:54.564Z        INFO        tflite/detector.go:266        Detection Complete        {"package": "detector.tflite", "id": "", "duration": 0.393365321, "detections": 1}
Sep 16 14:51:54 ubuntu.local docker[23084]: 2019-09-16T18:51:54.565Z        INFO        server/server.go:88        API Request        {"status": 200, "took": 0.500263875, "remote": "192.168.100.13:55516", "request": "/detect", "method": "POST", "package": "server.request", "request-id": "a5c7effbe9bb/FkKqxbcDyU-000004"}

but there is nothing in entry changing

you can see I created this template

root@ubuntu:/usr/share/hassio# cat cat.tmp
{{states.image_processing.doods_unifi_garage }}

which I then eval and its empty

root@ubuntu:/usr/share/hassio# hass-cli template cat.tmp


<template state image_processing.doods_unifi_garage=0; matches=, summary=, total_matches=0, friendly_name=Doods unifi_garage @ 2019-09-16T14:05:31.700746-04:00>
root@ubuntu:/usr/share/hassio#

This seems to be the same behavior @andreasfelder reported 16 days ago