Tensorflow + camera = door sensor!?!

I installed an ip camera in my garage and did not have a sensor to monitor if the garage door was open or not. After some experimenting an, I am now using tensorflow to determine the garage door status.

I used the doc from https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/#0 after getting the tutorial working. I created the following structure

  • tf_files
    • garagedoor_photos
      • dooropen
      • doorclosed

I then captured pictures and moved them into the respective folders to train the model.

(Captured like:
curl “https://XXXXXXXX/api/camera_proxy/camera.garage_camera?api_password=XXXX” > ~/tf_files/unsorted-$(date +"%Y%m%d%H%M%s%S").jpg

I used the following to train the model.


tensorboard --logdir tf_files/training_summaries &
python -m scripts.retrain --bottleneck_dir=tf_files/bottlenecks --how_many_training_steps=$trainingsteps --model_dir=tf_files/models/ --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" --output_graph=tf_files/retrained_door_graph.pb --output_labels=tf_files/retrained_door_labels.txt --architecture="${ARCHITECTURE}" --image_dir=tf_files/garagedoor_photos

I’m not thrilled with this yet, but I am using flask + bash to integrate with HA. (I’d like to not jump out to subprocess in the future.)

    import subprocess
    import sys
    import os

    from flask import Flask
    app = Flask(__name__)

    def hello_world():
        return 'Hello, World!'

    def show_user_profile(model,camera):
        # show the model
        conn = http.client.HTTPConnection("XXXXXXX")

        headers = {}

        url = "/api/camera_proxy/camera." + camera +"?api_password=XXXXXXXX"

        conn.request("GET", url, headers=headers)

        res = conn.getresponse()
        data = res.read()


        output = subprocess.check_output(["./webit.sh",model,camera])

        return '%s' % output.decode('ascii') 

the webit.sh

cd tensorflow-for-poets-2

curl -s "https://XXXXXXXX/api/camera_proxy/camera.$2?api_password=XXXXX" > tf_files/temp.jpg

source ../bin/activate

python -m scripts.label_image_json     --graph=tf_files/retrained_$1_graph.pb      --image=tf_files/temp.jpg --labels=tf_files/retrained_$1_labels.txt 2> /dev/null

then added the following to HA. My example need 90% confidence in one or the other

- platform: rest
    name: garage_door
    resource: http://localhost:5000/tf/door/garage_camera
    value_template: '{% if value_json.dooropen > 0.89 %} open {% elif value_json.doorclosed > 0.89 %} closed {% else %} unknown {% endif %}'

Thanks for posting, this is exactly what I was considering for a project.
Are you performing calculations on a pi3?

no … the model once generated might work okay on pi3

Tensorflow has pretty good bindings for Python, so you should be able to just import your tensorflow code into your flask app.

Also, have a look at the python requests library to replace your curl call.

1 Like

I was planning on that direction. :slight_smile:

Tensorflow has a PyPi package https://pypi.python.org/pypi/tensorflow

It will be amazing to have a Tensorflow image_processing component in HA. Looking forward to this.


Having worked through the tutorial, I believe image classification could be implemented as a command line sensor. I’ve found a release of [Tensorflow for the pi,](https://github.com/lhelontra/tensorflow-on-arm/releases/tag/v1.3.1 (python 3.5 & Stretch) and one on docker, and a ref on performance (it’s practical).

Progress on pi3:

  1. pip3 install tensorflow-1.3.1-cp35-none-linux_armv7l.whl Install took about 15 mins
  2. Verified tensorflow works with hello-world program

Yes, you could set it up that way.

I want to set it up as a reusable rest service so it doesn’t need to reload the model each time.

1 Like

Would love to know more about that. If we run TensorFlow in a docker and interact with it using a command line sensor, that should solve many issues.

1 Like

OK running tensorflow on my Macbook pro to classify an image from this tutorial, classification taking about 4 seconds. Note that this is using the Inception-v3 model which can classify over 1000 classes, so hopefully a simpler classifier would be much quicker. Compiling C++ code would also provide speed improvements I assume.

@texnofobix have you timed your classifier - and what hardware are you running on?

1 Like

This is an interesting article. I think it would be awesome to have a TensorFlow component and community contributed models that could be added e.g via Hassio addon…


time python -m scripts.label_image_json     --graph=tf_files/retrained_door_graph.pb      --image=tf_files/temp.jpg --labels=tf_files/retrained_door_labels.txt
{"doorclosed": 0.9996117949485779, "dooropen": 0.00038829262484796345}

real    0m0.777s
user    0m0.700s
sys     0m0.468s

I am running hass in docker, so I wanted to create some separation which is why I went towards RestAPI method.

CPU >> Intel® Core™ i7-6700 CPU @ 3.40GHz

1 Like

Will compare performance on a pi 3. However if a single pi 3 does not provide adequate grunt could consider a pi cluster :slight_smile:

Also to try, Caffe https://github.com/Pi-DeepLearning/RaspberryPi-FaceDetection-MTCNN-Caffe-With-Motion

And to query the online Caffe demo https://github.com/StevenHickson/RPi_CaffeQuery With this method, the image classification takes ~1.5s, which is usable for a system.

OK running the tutorial classification problem on a pi 3 classifying a single image takes approximately 11 seconds. I think this would be adequate for the use case of determining whether a door is open or not. I would setup an automation to run this script every 15 seconds.
Note that performance improvements could probably be improved by using a less accurate classifier. I plan a study to examine the tradeoff between accuracy and compute time in the context of classifying the state of objects (e.g. doors) around the house.


Sounds like tensorflow lite for the pi3 is in the pipeline:

1 Like

OK following the tutorial below it is very simple to setup a local flask server that can perform image classification. Images are submitted and the classification returned in approx 0.5 seconds (on my Mac), so decent enough for practical usage. My testing is on GitHub. Would be nice to deploy the classification server as a Hassio addon, will test performance on a pi.



Got a native component working in principle:


@texnofobix I expect my upcoming tagbox component will work for your garrage door open/closed example - are you either:

  1. Happy to share a couple of your images for me to try?
  2. Willing to try the component below?

Use this training script

@robmarkcole. Since you are the expert in Machinebox and experienced in Tensorflow, may I know how is Tensorflow compare to Machinebox? Can it replace Facebox and Classificationbox? I am just curious because Tensorflow is officially supported by HA now.

@masterkenobi The short answer is they are different:

  1. Tensorflow = object detection, cars, bikes et
  2. Facebox = faces and recognition
  3. Classificationbox = assign an image to a class

The long answer is interesting and perhaps worthy of a blog article, which I will write soon.