TensorFlow False Detection

I setup TensorFlow on my HA instance and its working pretty well. I do get these a couple times a day:

I am using the “faster_rcnn_resnet101_coco_2018_01_28” model zoo. I am running on a decent laptop with a lot of ram. Can anyone recommend a more accurate library to use that would work well on a 2.7 ghz I7 4800MQ with 24gb of ram? It has an Nvidia Quadro K610M mobile video card.

I dont know if the pictures are showing up but it detected a tree as a person with 88% confidence.

I would just cut down the tree… :slight_smile:

I have played with all of them, you’ll just have to experiment. Some do have a serious CPU hit, but what helped me was to pair it with Blueiris’s motion detection. So TF only runs when Blueiris signals motion was detected as a prefilter. That allowed me to use the high CPU models.

I can change the detection area in my Amcrest. What does Blue Iris have that you like?

It was easy to get running for me. I don’t like that it was Windows based, meaning I needed a Windows license to run it. I didn’t have much success with the open source and other linux based software out there. It has all the features I need and an API that works with HA and guides on how to set it up

It is not without it’s drawbacks, besides windows I did have to dedicate a NUC to it, so I could reduce CPU. I have 3 8MP cameras running now, and it really works best with an Intel CPU with built in video processing. So I didn’t have any success virtualizing it. But now I have enough CPU to run way more cameras in the future.

I think the key with TF is to get it to run on a trigger if you can there is no need to have TF wasting CPU cycles on unchanged video.

I have it running off of a trigger from my Amcrest motion sensor. It triggers pretty often. I am starting to realize that this is going to take a lot of tweaking to fine tune it.

@wuench is correct. May be easier to pair with some other sensor. But I capture all detection above xx% as image and use combination TF(Doods) + motion to filter some video recording

Just as surveillance concept I’ve learned that a combination of TF/image detection + motion work well.

Unless 24 hour monitoring biggest challenge:

1 knowing when incident occur without needing to review all video ; like person come to door and look in home but not ring/knock

2 secondary case being when you know incident occur during general time frame and need only find video showing occurance; like Amazon package taken

Motion detect and tensor flow may have false detect but this only issue if you cannot review images recordings for incident due to large qty and no labels or receive some notification

I decide on motioneye for record, review and motion video capture. motion capture is fairly accurate and simple software. It also has API that allow notification and trigger as needed

I use Doods(currently) for TF image capture. I prefer image due to fast review time and TF mostly get good photo showing Face, plate, etc. I drop captured images into motioneye folder and can review there easily.

I trigger html5 notification with image from door camera on detect. Future I plane notify when TF detect Person + Motioneye detect motion from same camera in critical area or some other method that allow me detect incident that normally would be unnoticed (person looking into side window).

My method I capture few TF incorrect detect images but can ignore easily. Also photo is small and storage cheap so best have many then miss the important one. Pairing this with motion trigger or other trigger for video record allows reduce unneeded recording and I can use mask on this trigger to reduce that as well. Doods not have mask option but others may.

I’ve amended my detection zone for the motion part, and I’ve amended the area that TF monitors. I’ve got next to 0 false alerts now on my outside cameras (hikvision)
.
The only false alerts I still have are in on the camera in the garage because the camera is of low quality and does not have IR for night so whenever there’s noise on the feed (a lot during dark times), it triggers motion and TF returns false alerts for shadows that it thinks is a person. I’v further reduced the false notifications by only being alerted when the probability / confidence of a person is above 90%:

- alias: Tensorflow Notify Person Detected
  initial_state: true
  trigger:
    - platform: numeric_state
      entity_id:
        - image_processing.tensorflow_back
        - image_processing.tensorflow_front
        - image_processing.tensorflow_side
        - image_processing.tensorflow_garage
      above: 0
  condition:
    - condition: template
      value_template: >-
        {{ (trigger.entity_id == "image_processing.tensorflow_garage" and (state_attr("image_processing.tensorflow_garage", "matches").person[0].score|int) >= 90)
        or ( trigger.entity_id in ["image_processing.tensorflow_back","image_processing.tensorflow_front","image_processing.tensorflow_side"] ) }}
  action:
    - service: notify.gmaillolo
      data_template:
        title: "HA2 - Person Detected on CCTV"
        message: "{{state_attr(trigger.entity_id, 'summary').person}} persons detected on {{ trigger.entity_id.split('.')[1].split('_')[1]|capitalize }} CCTV !"
        data:
          images:
            - "/config/tensorflow/storagemotion/{{ trigger.entity_id.split('.')[1].split('_')[1] }}_latest.jpg"

you need to adjust the scan area for tensorflow. check the docs and you can omit that area.

For the record I do get false postives as well for cats but its not frequent. Only thing you can do is ignore the area but if you want that area detected you need to live with it.

Which model zoo are you using to successfully detect cats? I have not ever had one successful detection over the last 3 months with the model zoo I have been using.

my false positives for cats is that they occasionally get detected as person, i get maybe one or so a week and i am using faster_rcnn_inception_v2_coco model

in my setup I am only looking for people

Sure that is the same as I am using, I do have cats in my config, and don’t even get false positives!

its all about the angle and cats position…normally when they stop to clean themselves on my path they get detected as a person lol…hell i have seen shadows and crows detected as people…even chicken wire lmao

Is this showing up as an image? It detected a pile of cement pads as a person lol

yea until we can start training images you have to look and see why it looks like that lol…the bottom of the yellow border could look like a pair of legs :stuck_out_tongue:

its def annoying lol

I’d like to do this confidence check in Node-Red, any suggestions on how to check the state attribute Person-Matches-Score in Node-Red?

Here is what I have:

After triggering the TensorFlow scan use a current state node to pull in the results - here I have it check if the person detection is greater than 0 (I have it setup only to check for person - nothing else - so anything over 0 is a person).

After that I use a function node, but what you are looking for is msg.data.attributes.matches.person[0].score.
So you’ll want to use a function, or template, or some other node to compare that value.

Cheers!
DeadEnd

Would you be willing to paste in what’s in your function node?

My function node does the comparison and also builds a message to be sent to my phone - so much of it is not relevant. I can trim it down later when I am home and have time and reply here if someone doesn’t give a version before then.

DeadEnd

OKay, here is a slimmed down version of my function node.
It sets the variable “score” to the confidence of the tensorflow scan.
Then if this is above 90%, it generates a payload message.
The next step is to do a service call to my notification in HA.
Since I write this into the message payload, it is automatically used as the service call payload.

// set score to person score from sensor
var score = msg.data.attributes.matches.person[0].score;

if (score>90){
    
    // generate payload message
    msg.payload =
        {"data":{
            "data": {
                "tag": 3,
                "image": "{PATH TO IMAGE}.jpg",
            },
            "title": "Front Door Person",
            "message": "Someone is at the Front Door"
        }};
    return msg;

}

else return null;

I hope this helps!
DeadEnd

1 Like