Awesome tool for HA. I’m currently using in Unraid with everything in separate dockers. With my Celeron G4900 w/16GB RAM it’s taking 3-4 secs to analyze pictures. I’m using a curl command to save pictures from my Hikvision camera feeds to a jpg file then moving jpg file to /config/www. I’m then using another curl command to send latest analyzed photo to me via telegram. I set save_timestamped_file: false so my docker won’t get filled photos. This is what my note red looks like. I’ve also included a picture of my partner trying to trick the AI. That’s not a cat… If you have any questions about my setup feel free to ask.
It does look like a convincing cat…!
I see a lot of people using node red with this integration, and I don’t have any node red experience myself. So could someone explain the benefits of node red for image processing vs using home assistant automations etc?
Hi Robin, some news about the RPI docker version? no pressure at all … just info. Thank you in advance
I am not keeping any secrets, I will know when you know
I find it easier to build my automations visually in nodered than using ymal. This was before HA had the automation UI builder.
1 Like
I created an app for exploring the config parameter for the deepstack object integration, created a thread here
4 Likes
Hey… could you please export me node red code please… I only do mine in mode red as well…
For starters; thanks a lot for this great integration, it’s fun and easy to use. You obviously spent a lot of time developing it - to share that effort for free is really kind of you 
I have a question: currently I’m using your tooling to detect whether neighborhood cats came into the house (we keep the door open, and they do that all the time to eat our cat’s food, bastards). When a cat is detected, I get a telegram notification.
For the next phase, I want to use the images of the various cats to create a new ml model, so I can recognize my cat from the others - that way I only get notifications for other cats.
Is there a way I can store the images in which deepstack detected a cat (or dog, as deepstack keeps calling my cat a dog) as-is? I’d like to keep the single ‘deepstack_object_local_file_latest.jpg’ setup so I can send that through telegram.
At any rate, thanks for your time, and best of luck
@SamKr you want save_timestamped_file: True
To recognise YOUR cat that is a classification problem, and you would want to deploy a custom model (possible in deepstack when they open source it)
Almost, I’d like those images, but without the red rectangle (because it’d mess with the model).
I’m planning to use Microsoft’s ML.NET model builder to get some experience with that as well
Hey @robmarkcole,
I’m still using your “tflite_server integration” as custom component and it is so far working fine, how ever I want to save all the detected images in my save_file_folder folder with timestamp, currently it is only saving the latest detected picture, could you please help me get this on this component.
My configuration.yaml is as below
image_processing:
- platform: tflite_server
ip_address: 192.168.1.120
port: 5000
scan_interval: 15
save_file_folder: /config/www/
source:
- entity_id: camera.esp_camera
name: AI_Cam
"""
Component that will perform object detection via tensorflow-lite-rest-server
"""
import datetime
import io
import json
import logging
import os
from datetime import timedelta
from typing import List, Tuple
import requests
from PIL import Image, ImageDraw
from homeassistant.util.pil import draw_box
import deepstack.core as ds
import homeassistant.helpers.config_validation as cv
import homeassistant.util.dt as dt_util
import voluptuous as vol
from homeassistant.components.image_processing import (
ATTR_CONFIDENCE,
CONF_ENTITY_ID,
CONF_NAME,
CONF_SOURCE,
DOMAIN,
PLATFORM_SCHEMA,
ImageProcessingEntity,
)
from homeassistant.const import (
ATTR_ENTITY_ID,
ATTR_NAME,
CONF_IP_ADDRESS,
CONF_PORT,
HTTP_BAD_REQUEST,
HTTP_OK,
HTTP_UNAUTHORIZED,
)
from homeassistant.core import split_entity_id
_LOGGER = logging.getLogger(__name__)
CONF_SAVE_FILE_FOLDER = "save_file_folder"
CONF_TARGET = "target"
DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S"
DEFAULT_PORT = 5000
DEFAULT_TARGET = "person"
RED = (255, 0, 0)
SCAN_INTERVAL = timedelta(days=365) # NEVER SCAN.
PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend(
{
vol.Required(CONF_IP_ADDRESS): cv.string,
vol.Optional(CONF_PORT, default=DEFAULT_PORT): cv.port,
vol.Optional(CONF_TARGET, default=DEFAULT_TARGET): cv.string,
vol.Optional(CONF_SAVE_FILE_FOLDER): cv.isdir,
}
)
def get_target(predictions: List, target: str):
"""
Return only the info for the targets.
"""
targets = []
for result in predictions:
if result["name"] == target:
targets.append(result)
return targets
def setup_platform(hass, config, add_devices, discovery_info=None):
"""Set up the classifier."""
save_file_folder = config.get(CONF_SAVE_FILE_FOLDER)
if save_file_folder:
save_file_folder = os.path.join(save_file_folder, "") # If no trailing / add it
entities = []
for camera in config[CONF_SOURCE]:
object_entity = ObjectDetectEntity(
config.get(CONF_IP_ADDRESS),
config.get(CONF_PORT),
config.get(CONF_TARGET),
config.get(ATTR_CONFIDENCE),
save_file_folder,
camera.get(CONF_ENTITY_ID),
camera.get(CONF_NAME),
)
entities.append(object_entity)
add_devices(entities)
class ObjectDetectEntity(ImageProcessingEntity):
"""Perform a face classification."""
def __init__(
self,
ip_address,
port,
target,
confidence,
save_file_folder,
camera_entity,
name=None,
):
"""Init with the API key and model id."""
super().__init__()
self._object_detection_url = f"http://{ip_address}:{port}/v1/object/detection"
self._target = target
self._confidence = confidence
self._camera = camera_entity
if name:
self._name = name
else:
camera_name = split_entity_id(camera_entity)[1]
self._name = "tflite_{}".format(camera_name)
self._state = None
self._targets = []
self._last_detection = None
if save_file_folder:
self._save_file_folder = save_file_folder
def process_image(self, image):
"""Process an image."""
self._image_width, self._image_height = Image.open(
io.BytesIO(bytearray(image))
).size
self._state = None
self._targets = []
payload = {"image": image}
response = requests.post(self._object_detection_url, files=payload)
if not response.status_code == HTTP_OK:
return
predictions = response.json()
self._targets = get_target(predictions["objects"], self._target)
self._state = len(self._targets)
if hasattr(self, "_save_file_folder") and self._state > 0:
self.save_image(image, self._targets, self._target, self._save_file_folder)
@property
def camera_entity(self):
"""Return camera entity id from process pictures."""
return self._camera
@property
def state(self):
"""Return the state of the entity."""
return self._state
@property
def name(self):
"""Return the name of the sensor."""
return self._name
@property
def unit_of_measurement(self):
"""Return the unit of measurement."""
target = self._target
if self._state != None and self._state > 1:
target += "s"
return target
@property
def device_state_attributes(self):
"""Return device specific state attributes."""
attr = {}
if self._targets:
attr["targets"] = [result["score"] for result in self._targets]
if self._last_detection:
attr["last_{}_detection".format(self._target)] = self._last_detection
return attr
def save_image(self, image, predictions, target, directory):
"""Save a timestamped image with bounding boxes around targets."""
img = Image.open(io.BytesIO(bytearray(image))).convert("RGB")
draw = ImageDraw.Draw(img)
for prediction in predictions:
prediction_confidence = ds.format_confidence(prediction["score"])
if (
prediction["name"] in target
and prediction_confidence >= self._confidence
):
draw_box(
draw,
prediction['box'],
self._image_width,
self._image_height,
text=str(prediction_confidence),
color=RED,
)
latest_save_path = directory + "{}_latest_{}.jpg".format(self._name, target[0])
img.save(latest_save_path)
@SamKr I released v3.4 just for you , let me know how you get on.
@Hitesh_Singh I think I archived that integration, please use deepstack-object
Hey @robmarkcole, thanks for all the hard work with these two great custom components. I managed to get both Object and Face detection running smoothly and absolutely love it. My question to you is, can we install/teach new models?
I specifically have an issue with snakes and would like to have my driveway CCTV cam alert me if any snakes come in from under the gate; is that something that is possible to integrate with the current Deepstack setup?
Looking forward to hearing from you and thanks once again for al the effort with this integration!
Working on it…
2 Likes
@robmarkcole I’m using a Raspberry Pi 4 4GB model, and deepstack-object will have performance issue in it, which is why was asking help on tflite_server integration. as it runs fine on a raspberry. 
@Hitesh_Singh use https://github.com/robmarkcole/tensorflow-lite-rest-server runs on rpi and I just added face detection to it too! works with deepstack integrations
2 Likes
Thanks for this, working great. Is it possible have an option to export another image of the frames only as a transparent PNG (same size as the frame size) so I can overlay it over an image ?
Can you give some example python code?
can you show me what you did to resolve this? I am having the same error.
Thanks