New Custom Compontent - Image Processing - Object Detection - DOODS

Step · January 29, 2021, 8:44pm

Hi, thanks for the awesome work @snowzach.
My first post, however been using Home Assistant since the first few days of January this year and I’m impressed (came from Domoticz). I’ve struggled with all kinds of issues and these forums helped me a lot (w/ trial and error) and allowed me to get very far.

My setup is an HA OVA w/ the hass.io addon added through supervisor.

My current issue is that I am trying to see if I can adjust the automation trigger to work with the motion detector AND (this a condition?) if it’s a person defined by doods. I get way too many motion detections (I’ve adjust the sensitvity in Hikvision to try and lower the amount to no avail).

configuration.yaml

image_processing:
  - platform: doods
    scan_interval: 10000
    url: "http://IP:8080"
    auth_key: !secret doodskey
    detector: tensorflow
    file_out:
      - "/config/www/pictures/{{ camera_entity.split('.')[1] }}_latest_doods.jpg"
    #  - "/config/www/pictures/{{ camera_entity.split('.')[1] }}_{{ now().strftime('%Y%m%d_%H%M%S') }}_doods.jpg"
    source:
      - entity_id: camera.ds_2cd2112_i_mainstream
    confidence: 40
    labels:
      - name: person
        confidence: 40
      - name: car
        confidence: 60
      - name: truck
        confidence: 40

automation.yaml

# DOODs w/ camera (front)
- alias: Camera - Person detected
  trigger:
    - platform: state
      entity_id:
        - binary_sensor.videosource_1_motion_alarm
      #  - binary_sensor.mymotiondetectorrule_cell_motion_detection disabled due to the above working better.
      to: "on"
  condition: []
  action:
    - service: image_processing.scan
      entity_id: image_processing.doods_ds_2cd2112_i_mainstream
    - service: notify.mobile_app_step
      data:
        title: Motion detected!
        message: Someone in front of your house. Prepare for battle!
        data:
          attachment:
            url: https://myIP:8123/local/pictures/ds_2cd2112_i_mainstream_latest_doods.jpg
  mode: single

The automation.yaml above spams me with messages (everything seems to works fine). However the following code added silences the message but doesn’t send me a message if I stand in front of the camera:

  condition:
    - condition: template
      value_template: "{{ 'person' in state_attr('image_processing.doods_ds_2cd2112_i_mainstream','summary') }}"

I’m trying to create an “if-then-else” situation where the trigger and the condition are checked (is motion detected and is it a person scanned by doods) before performing the action. However I think I’ll need to start this somewhere before the action-sequence otherwise doods will not have scanned a person:

    - service: image_processing.scan
      entity_id: image_processing.doods_ds_2cd2112_i_mainstream

On the 30th of January: apparently sometimes it seems to detect someone in front of my house (w/ the extra condition) however there was no person in sight (doods didn’t see a person either):

matches: 
car:
  - score: 93.24881
    box:
      - 0.14872093
      - 0.061141133
      - 0.2969224
      - 0.18527538
  - score: 89.866615
    box:
      - 0.21446973
      - 0.055247918
      - 0.95497346
      - 0.6572731
  - score: 81.25439
    box:
      - 0.17098683
      - 0.0012004471
      - 0.28035635
      - 0.056980252
truck:
  - score: 40.161026
    box:
      - 0.04752839
      - 0.30466023
      - 0.17807434
      - 0.49091586

summary: 
car: 3
truck: 1

total_matches: 4
process_time: 1.4891642459988361
friendly_name: Doods ds_2cd2112_i_mainstream

Can anyone point me in the right direction?

My second question is:
I know that I can add my back camera in the config.yaml by adding another entity_id to the source. However how can I remove the car label from my backyard (there will never be a car there unless it falls from the sky) while keeping the car label in the front camera.

Third question (while I am at it, this will be my last and least necessary):
Is it possible to rename labels in the txt file or does that mess up the system?

snowzach · January 30, 2021, 6:05pm

Honestly, I don’t do a ton of work with Nvidia cards. I have an GTX970 that I’ve tested it with. If it’s 0.2-0.3 seconds with the inception model I would say that’s typical if I recall correctly. I couldn’t tell you much about how much memory is required. Sorry, I don’t honestly know much about the CUDA/tensorflow stuff. I just figured out how to make it basically work.

snowzach · January 30, 2021, 6:14pm

I won’t honestly be much help with HASS automations. I have tried a few times and gotten frustrated with it and switched to using NodeRED and won’t go back at this point. At least for me it’s so much easier.

If you want to create 2 instances of DOODS, just create another one with all the same stuff… ie:

image_processing:
  - platform: doods
    scan_interval: 10000
    url: whatever
    blah blah

  - platform: doods
    url: whatever
    scan_interval: 10000
    blah blah

You can change the labels if you like. It’s just a file that’s mounted inside the docker image. You’ll need to mount your own labels file over top the one inside the doods image. For the tflite detector it’s /opt/doods/models/coco_labels0.txt for tensorflow it’s /opt/doods/models/coco_labels1.txt the labels are based on the line number. Update it to whatever you want. I’ll leave it to google/stack overflow on how to mount files inside of docker. They explain it better than I can.

miki_lbc · January 31, 2021, 10:13am

I spent some time to find how to work with parameters. This code count number of persons in image (it is template sensor). Hope that helps.

- platform: template
  sensors:
    doods_cam_stodola_person:
      friendly_name: Stodola cam person
      value_template: >
        {% set ns = namespace(ok = 0) %}
        {% for item in state_attr('image_processing.doods_cam_stodola', 'matches').person %}
          {% set score = (item.score | float) %}
          {% set y = (item.box[0] | float) * 100 %}
          {% set x = (item.box[1] | float) * 100 %}
          {% set h = (item.box[2] | float - item.box[0] | float) * 100 %}
          {% set w = (item.box[3] | float - item.box[1] | float) * 100 %}
          {% if score > 0 and x > 0 and y > 0 and w > 0 and h > 0 %}
            {% set ns.ok = (ns.ok + 1) %}
          {% else %}
          {% endif %}
        {% endfor %}
        {{ ns.ok | int}}

Replace camera name (doods_cam_stodola) and object (person)

Step · January 31, 2021, 3:15pm

Thank you for the useful information, I have NodeRED installed and working. Been following tutorials for NodeRED but still need a good use case and example (if you have any, i’ll be much obliged) and I will need to see how this works without HASS automation and still keep the overview.

Also a big thanks to @miki_lbc for showing me an decent example of a person automation!

snowzach · January 31, 2021, 3:57pm

Honestly this thread is the one that got me started with NodeRED Integrating Blue Iris, Pushover, Node-RED, and Amazon Rekognition - #167 by snowzach

After that I figured out how to use it. Now basically I have and entire alarm system state machine implemented with NodeRED. Any time someone pulls in my driveway or walks to the front door it plays a chime on my Google Homes. If I’m not home and it detects someone it sends notifications to my phone with Pushover. My outside lights are all controlled with it. Those are a handful of my automations.

quizzical · January 31, 2021, 10:32pm

Thanks a lot for a very cool integration!
In this cropped image, DOODS does not recognise me:

I assume I’m asking too much of DOODS right? It’s a cropped (only looking at the front door part of the image) night image, so I’m not giving DOODS much to work with…
For a human brain it’s blatantly obvious someone is behind the door, but I’m too cheap to pay a human to look at my camera images

Any idea’s for another approach I could take?

mickeynygaard · February 1, 2021, 7:48am

I’ll volunteer to monitoring your cameras!

Nah just kidding, I have similar issues…DOODS detects my dog as 93% human and sometimes 90+% for a chair…it seems to struggle most with night time images, similar to your image…maybe some tweaking is needed here?

snowzach · February 1, 2021, 1:45pm

Yeah, I am not sure what you can do there. I suspect the door frame is confusing the Tensorflow model. The only other thing might be to train a custom model here. I have never done it myself but in theory you could train the model to understand person with a door in the way there. I hope to get time someday to actually learn how to do this myself.

Step · February 1, 2021, 3:32pm

As to your suggestion I moved to Node-RED to see if I can make something awesome happen in it.

So far I’ve created a sequence and it seems to me the idea works but I’m trying to see if an attribute contains a value (attribute summary contains person) to continue with the sequence:

This is what I got from the debug log:

So far I’ve tried to fetch the attribute person with this line:
msg.data.attributes.summary person
Screenshot 2021-02-01 at 16.34.16

Though it keeps giving me this error:

“Error: Invalid property expression: unexpected ’ ’ at position 23”

Now before I try to debug something I’ve never done before, maybe anyone can point out what kind of node or parameters/properties I am missing or defined wrong?

`

quizzical · February 1, 2021, 5:51pm

My tests during the day are also failing, which fits well with your theory that the door frame is confusing the model.
I’m already proud that I managed to integrate DOODS and get it working (even if you seem to have managed to hide most of the complexity, thanks again!), I guess training the model will be out of my league
Perhaps in a year or two it will become easy enough that even I can do it.

gazzaman · February 2, 2021, 10:37am

Hi All , Love the project
2 question
1 is it possible to put a Coral Mini PCIe M.2 into a USB converter
2 has anyone done it

snowzach · February 2, 2021, 1:46pm

I would use the debug option a bunch to see what’s coming out of the previous block. Sometimes I’ve found it easier to use the code node to just process each message by hand. Sometimes the home assistant nodes are goofy. What is coming out of msg. might be a sub object of that message. It takes a little fiddling to get used to node red but once you do it’s magic.

snowzach · February 2, 2021, 1:49pm

I know that people have used the mini PCIe EdgeTPU with DOODS but I have not heard of anyone doing it with a USB converter. I am pretty skeptical that the EdgeTPU would work in one of those adapters. I think they are mostly designed for network cards that support some sort of USB mode. I could be wrong though.

gazzaman · February 2, 2021, 2:12pm

Thanks for the reply

terryhonn · February 2, 2021, 2:30pm

@snowzach, forgive me if I’m asking in the wrong place, as I’m not using the HA component right now, but rather another instance of DOODS in a Docker container on a different rPi. I’m using motion detection in motionEye to detect and save image to disk, and Node-Red to watch the folder and send found images to DOODS with the following JSON:

    "detector_name": "default",
    "data": msg.payload.toString('base64'),
    "detect": { "car": 60,
                "truck": 60
              }
}
msg.headers = {
    "Content-Type": "application/json"
}
return msg;

That’s working great, and if there are detections returned the file is saved off for review, sensors are triggered, automations run, etc.

What I’m missing is seeing DOODS ‘work’: the bounding boxes and labels and confidence.

In the HA component, there is the file_out: "/config/www/tmp/{{ camera_entity.split('.')[1] }}_latest.jpg" directive that gives me what I want.

From snooping at the msg that DOODS is returning in Node-Red, it looks like there is an image in there. Is it the equivalent of file_out:, and if so, how can I extract it?

Thanks for amazing work with DOODS… it’s my first foray into ML and AI stuff, and for an old guy, it’s truly incredible stuff.

Step · February 2, 2021, 2:30pm

I’m really starting the get the hang of it, the debug block tells me the information on the second picture from my previous post.

I do not know the appropriate syntax or current state block to see if “person” exists in msg.data.attributes.summary.

According to DOODs and the debug log it should tell me whether “car” or “person” or “truck” exists:
msg.data.attributes.summary.person gives me a wrong example because if person = 0, the attribute does not exist.

snowzach · February 2, 2021, 2:43pm

DOODS doesn’t return any image data. It just returns the coordinates and labels detected. Inside of the home assistant component is where draws the boxes and annotations. This is outside of the scope of what DOODS does (at least for now)

snowzach · February 2, 2021, 2:45pm

That’s why the code node can be easier. You can test to see if it exists first.

Step · February 2, 2021, 2:47pm

I’m kinda missing that kind of code, not sure what to put there (I’ve tried everything in a Current State Node).

All kinds of “is”, “is not”, and I even tried a Switch Node which has an option for “contains” person in the msg.data.attribute.summary however it’s not accepting it. It’s probably due to lack of understanding of code on my side…