HOLODECK project - Projection Based Kitchen Assistant

No voice commands used:

https://imgur.com/gallery/jIgBmiQ

WARNING: Voice control with Google Assistant, better kill your Google assistant before watching this
Commands: prepare pizza | prepare quesadilla | test quesadilla | test homeassistant video

Voice commands are not fixed (set via Google Home App) - they are dynamic, accepting any query.
The vocal control with Google Assistant was a bit tricky since the only way to get an assistant query in HomeAssistant was by using IFTTT

Commands breakdown: prepare ā€˜foodā€™ will play videos from the same channel while test ā€˜somethingā€™ is not restricted, will broadcast the first YouTube video that matches the query; There is also a 3rd command - mix, that is also channel bound but related to mixology.

This project is not complete yet. Iā€™ve been planning this for over a year (is one of the reasons I installed rail lighting) and Iā€™ve been working on it (on and off) for the last 6 months to get it to a state where is stable and fun to use, but since it looks a lot better than I ever expectedā€¦here we goā€¦another big ass postā€¦
I still want to buy another projector (the 5th, just for this project), but this time a non-smart one, since I believe I can make it even better without Android annoying boot times and I will differently do a complete code overhaul once I decide what I want from it and how I will use it most oftenā€¦so a part II of this might happen.

Since I have this installed on the kitchen furniture, I wanted to give it the feel of a kitchen assistant . . . so alcohol-related stuff, maybe cooking and as soon as I integrate a food delivery system in Home Assistantā€¦ there will defiantly be a part II.

HARDWARE

The basic idea is very simple; I have rail lighting with Cinema Track Projectors with the E27 socket, bulb, and the rest of the electronics stripped out and everything inside was replaced with a DLP P09 projector on a stand and a DIY power supply-adaptor-thingie (official name).

The projector has a standard tripod screw thread which allows me to fit it on the stand and then, through extremely precise eyeballing measurements I managed to fit that stand perfectly (ish) horizontal inside the light projector. . . and any hypothetical misalignment is solved by software/keystone correction.

https://imgur.com/a/3ka9Owq

The power supply-adaptor-thingie (that is completely wrapped in electrical tape on the final build) is made out of a random modular power plug connected to a Shelly 1 relay following this connection scheme. I would recommend using the Shelly 1PM 220V relay instead but I only had a dry contact at hand.

The supply-adaptor-thingie is there to power the projector from its original charger, which I didnā€™t want to cannibalize just in case I decide to take it out at some point, and is electrical input comes from what was initially at the ent of the E27 light socket, getting power from the lights electrical rail.

The point of the relay is to regulate charging of the projector, since it is powered from the light network, I donā€™t want it charging every time I turn on the lights (even for only 5 minutes) to preserve the life of the battery.
The projector comes with an integrated battery so it will work even if the lights are off (since it is powered as a light bulb)ā€¦it will also work when the power is completely offā€¦but probably no netflix/internet/etc.

CONTROL

This is an Android projector. So you have WiFi and ADB to control it. The greatest challenge (check the end of the postā€¦) is turning it on; this particular model has an IR port on the front so Iā€™m using a Broadlink (in my kitchen) to turn it on. It also has a physical power button (most projectors this size come with a touch interface)ā€¦so worst-case scenario, you can connect a smart relay in parallel with the button and you have control.

After is turned on, there are no more problems. You can add some peripherals to it, Iā€™m only using a H18 keyboard.

ADB seems slow sometimes so I prefer using Fully Kiosk Browser API (if you never heard of it, donā€™t judge it by its looks/logo.etc, - itā€™s amazing)

I prefer turning it on and changing the screens simply using my light switches that can control anything in the home. . .if you ever have way too much time, you can check that integration as well

KITCHEN ASSISTANT SCENARIO

TL;DR LOGIC:

  • I ask Google Assistant for something
  • I get that certain something in Home Assistant using IFTTT
  • From HA itā€™s sent to a bash script that builds a Youtube URL around it and sends the URL back to HA
  • HA is casting the final URL to a device

The IFTTT integration is as follows:

  1. On HomeAssistant side you will need to set it up as an Integration and save the URL you will get.

  2. On the IFTTT Applet IF you will need to use Google Assistant - Say a phrase with a text ingredient

  3. On the IFTTT Applet THEN THAT you will select Webhooks - Make a web request

  • Under URL you need to use the URL that you got in HomeAssistant in the first step
  • Method: POST
  • Content Type: application/json
  • Body: { "action": "holodeck", "cast": "{{TextField}}" }
    {{TextField}} is going to be your Assistant query saved under ā€œ$ā€ in the IF phase

https://imgur.com/a/lbnatiy

So far IFTTT will send to your HA instance your voice command, but in order to use it you have to intercept it in HomeAssistant using an automation like this:

- id: ifttt_cast
  alias: ifttt_cast
  initial_state: on
  trigger:
  - platform: event
    event_type: ifttt_webhook_received
    event_data:
      action: 'holodeck'
  action:
  - service: input_text.set_value
    data_template:
      entity_id: input_text.holo_query
      value: '{{ trigger.event.data.cast }}'
  - service: shell_command.holodeck_x

This automation is triggered by an ifttt event, and will add the string received (your assistant query, from ā€œ$ā€ or ā€œ{{TextField}}ā€) to a input_text.

input_text:
  holo_query:
    name: Holo
    initial: ' '
  holo_video_url:
    name: Video URL
    initial: ' '

Everything up to the last service from the automation above, will only get a Google Assistant voice command string in Home Assistant; the last service is what you actually want to do with it.
In my case, this is the shell_command.holodeck_x:

holodeck_x: sudo sh -c 'cd /home/pi/.SCRIPTS/HOLO && sudo bash holodeck_x.sh {{states("input_text.holo_query") | replace(' ', '+') }}'

In the future, I will integrate this completely in HA, but for now, I have it as an independent shell script.

This script accepts an argument, and the command above will send your Google Assistant query as the argument to that script.

If you ask the recipe for a ā€œMargaritaā€ there will be no problems but if you ask for something like ā€œjack on the rocksā€ (just an exampleā€¦I donā€™t actually need a recipe for that) the spaces between the words will break the URL, so the | replace(' ', '+') part is to ensure that the URL stays Youtube friendly no matter what you say, by replacing spaces with ā€œ+ā€

The purpose of the script is to generate a video ID for a Youtube video corresponding to your query, build the corresponding URL (embedded and set to autoplay) and send the final URL back to HA, to use it as you want. (I donā€™t send it to the projector because I might want to use it on a smart mirror or on some other DIY display)

In order to get it working you will need to get a Google Project API key and a HomeAssistant Long-Lived Access Token and pass them to the script below.

  • Google API:
  1. Go to https://console.cloud.google.com/
  2. Creat a a project
  3. From the LEFT menu - select APIs & Services - Credentials
  4. From the top bar - ā€œ+ Create credentialsā€ - API key
  • Home Assistant Token
  1. Go to http://your.homeassitant.ip:8123/profile
  2. scroll down to the bottom - Long-Lived Access Tokens
  3. Click ā€œCREATE TOKENā€

holodeck_x.sh:

#!/bin/bash

# CREDENTIALS
API_KEY="YOUR_API_KEY"

# VIDEO DATA
QUERY=$@
VIDEO_ID=$(sudo curl "https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=1&order=relevance&publishedAfter=2018-01-01T00:00:00Z&q=${QUERY}&safeSearch=none&type=video&key=${API_KEY}" | grep "videoId" | awk '{print $2}' | tr -d \")

# VIDEO URL
URL="https://www.youtube.com/embed/${VIDEO_ID}?autoplay=1"

# # TESTING
# echo ${URL}

# HA INTEGRATION
curl -X POST -H "Authorization: Bearer your_homeassistant_authorization_bearer" \
  -H "Content-Type: application/json" \
  -d '{"state": "'"${URL}"'", "attributes": {"editable": "false", "min": "0", "max": "100", "pattern": "null", "mode": "text", "friendly_name": "Video URL"}}' \
  http://192.168.0.100:8123/api/states/input_text.holo_video_url


Finally, we have the final URL, Youtube friendly, and ready to be broadcasted to a device, in this case to a smart projector. (Planning to exploit this integration a lot more in the future)

The bash script will send the URL to another input_text, and from there will be intercepted by an automation.
If you manually write/paste an URL in that input_text, in HA UI, it will trigger the automation and the manually entered URL will also be broadcasted to your device.

The projector is powered by Android, so I could cast it directly to it via ADB and wait for the YT app to launch, but it takes way to long, so Iā€™m using Fully kiosk Browser - amazing app, very useful in this project, but not mandatory.

Automation:

- id: holodeck_broadcast  
  alias: "HOLODECK BROADCAST"
  initial_state: on
  trigger:
  - platform: state
    entity_id: input_text.holo_video_url
  action:
  - service: script.holodeck_cast_link
  - delay: 00:00:03
  - service: script.daydream_off

Script:

holodeck_cast_link:  
  sequence:
  - service: homeassistant.turn_off
    entity_id: automation.holodeck_random
  - service: shell_command.daydream_off
  - service: shell_command.holodeck_cast_link

For now you can ignore the first 2 services.

shell_command:

holodeck_cast_link: curl "http://your.projector.ip:2323/?password=YOUR_KIOSK_PASSWORD&type=json&cmd=loadUrl&url={{states.input_text.holo_video_url.state}}"

At this point, I have 3 IFTTT applets that eventually link to 3 different bash scripts. I will most probably unify everything in the future, or just scrap what Iā€™m not using. The case presented so far will create a YT link based on popularity alone, basically, it will be the first video that YT lists if you manually search for something.
The other 2 are channel-specific; one for food and one for mixology; the main difference is in the bash script under VIDEO DATA. There is a static CHANNEL_ID (use whatever you need) and then it is added to the URL template &channelId=${CHANNEL_ID}

# VIDEO DATA
QUERY=$@
CHANNEL_ID="UCRIZtPl9nb9RiXc9btSTQNw"
VIDEO_ID=$(sudo curl "https://www.googleapis.com/youtube/v3/search?part=snippet&channelId=${CHANNEL_ID}&maxResults=1&order=relevance&publishedAfter=2018-01-01T00:00:00Z&q=${QUERY}&safeSearch=none&type=video&key=${API_KEY}" | grep "videoId" | awk '{print $2}' | tr -d \")

HOMEASSISTANT UI SCENARIO

https://imgur.com/a/l4OEtpz

If you can cast any URL it only makes sense to have a full HomeAssistant interface on display. Not as practical as a wall-mounted tablet, but a lot more futuristic. For me, this is the default view of the projector.

Iā€™m using Kiosk Browser for this and I have an automation that randomizes the lovelace tabs to be casted.

- id: holodeck_random
  alias: "HOLODECK RANDOM"
  initial_state: off
  trigger:
  - platform: time_pattern
    minutes: "/10"
  action:
  - service: script.turn_on
    data_template:
      entity_id: '{{ ["script.poltergeist_uix", "script.poltergeist_climate", "script.poltergeist_electricity", "script.poltergeist_player"] | random }}'

script:

poltergeist_uix: 
  sequence:
  - service: shell_command.daydream_off
  - service: shell_command.poltergeist_uix

shell_command:

poltergeist_uix: curl "http://your_projector_ip:2323/?password=YOUR-KIOSK-PASSWORD&type=json&cmd=loadUrl&url=http://your_homeassistant_ip:8123/lovelace/UIX"

SMART MIRROR INTERFACE

https://imgur.com/a/pFgZVQU

Since Iā€™m using MagicMirror and Iā€™m casting stuffā€¦it makes sense to have it integrated as well.

Funny thing is that this is actually the only integration I could use considering that my smart mirror setup is really powerful, being fully integrated with any voice assistant and Home Assistant.
Full details about that in thisā€¦slightlyā€¦smaller post.

This part is mostly useful since I have my phone notifications, news and other stuff mirrored . . . on the mirror.

The Smart Mirror is casted just like Home Assistant UI presented above.

MOVIE STREAMING

https://imgur.com/a/KwYBHc9

All my drive for automation comes from movies so if a streaming platform has an Android app or a web app it will most certainly not be forgotten.
It is absolutely amazing that the lighting projector Iā€™m using can rotate 360 degrees horizontally on itā€™s rail and it is able of 360 degrees vertical adjustment as well . . . ish (until it hits the ceiling); that combined with auto keystone and manual keystone adjustment of the projector means that it can be rotated or moved and project any of this on any wall/area of the room is installed it.

ā€¦next project - I need to motorize itā€¦

FULLY FUNCTIONAL ANDROID DEVICE

https://imgur.com/a/txStUpb

At this point there is no surprise - there is a fully functional Android 9.0 device with all itā€™s perks and drawbacks (boot time is the main problem).

Add a wireless keyboard or any peripherals and you have access to any android app. Add a jack 3.5mm microphone and you can use the integrated Google Assistant / Alexa / whatever; same microphone + fully kiosk browser or tasker and you can have the projection change depending on noise levels, etc.

Iā€™m thinking about using tasker to automate some delivery services that donā€™t have APIs.
Maybe call for an uber and see it on the map on the projectionā€¦you get the pointā€¦itā€™s nice to have a functional OS as backup.

CHALLENGES (A.K.A. dumb shit)

The greatest challenge with this project was turning it onā€¦I shit you notā€¦it was exhausting. . . everything else is trivial, but turning on something like this (installed in this manner) almost drove me insane.

The Android is heavily modified and even if you have access to bootloader/fastboot it will not accept any commands; the idea was to modify oem off-mode-charge 0 to boot when charger is connected / relay powered.

Then I tried to re-mount the Android /system partition to be able to write on read-only system files in order to replace the charging icon with a boot/reboot scriptā€¦to start when the charger is connected / relay poweredā€¦the result was even less promisingā€¦I had to buy a new motherboardā€¦

On my third attempt, I got really closeā€¦if that would have worked I would actually be impressed with my ingenuity (spoiler: it didnā€™t).
The problem was that the IR port was on the back of the device, which is encased in aluminum (by the light projector) so no remote can reach that portā€¦so what is the next logical step? if something is not where you want it to be - you move itā€¦
Therefore, since I had scrap parts already, I desoldered a 38 kHz IR receiver from a previews candidate and soldered it parallel with itā€™s existing IR receiverā€¦but not directly on the motherboardā€¦I used some wires to have the receiver mobile and at a distance; sure, I had a little dick hanging out of the projectorā€¦but it was workingā€¦until I turned it offā€¦
That was a DLP X2 that comes with 2 remotes (this is not a typo, it has 2 remotes) and none of them can turn the damn thing on - Iā€™m not crazy, the seller confirmed, the X2 doesnā€™t start from remote, you need to use the touch interface, then you can use the remote . . . so I bought another one

On my fourth attempt, the results were overwhelming; I bought an L1 projector this time (simply because at this point, if it was really really quiet, I could hear my wallet screaming in agony); itā€™s greatest feature was itā€™s price - only 40 USD | no Android, no nothing. I paired it with a Google TV Chromecast and a MiBox S (for some smarts) and it was amazing - this is the reason I want to buy another non-smart projector (but DLP) for a part II.
Boot time was 10 seconds, boot on last used app/screen (HDMI) - perfect! . . . just one problem, itā€™s not a Texas Instruments DLP . . . so the image is absolute garbageā€¦i mean, itā€™s so shit that even the Chinese store that sells it, says that is a TOY and it is for children

By the time I got to my fifth attempt they launched the 2020 DLP P09 - IR port on the front (that actually works), Android 9.0, physical ON button (so if everything else fails, I can remove the button and add a relay to control it) - itā€™s like it was especially designed for this project - itā€™s the one Iā€™m currently using

OKā€¦i think itā€™s time to stopā€¦sorry, I talk a lotā€¦Have fun and thanks for the read!

24 Likes

Nice build, excellent markup, where is the audio coming from?

1 Like

Thanks!
The projector has an integrated speaker which is quite good on itā€™s own but the aluminum somewhat-cylindrical case it is installed in seems to also amplify the sound a bit.

Well done @smowk looks very nice and professional build.

Out of curiosity just wanted to know what are these:
image

i.e. what do they control? I thought I read in your post that you were using a projector, are those controls for controlling a screen?

1 Like

Thank you!

That is not part of the projector, itā€™s just my touch-controlled kitchen hood.

2 Likes

I do plan to integrate gesture recognition sensors in this project, and those will probably be powered from the electrical wiring of that hood

1 Like

Nice cant wait to see that. Good job

1 Like

:exploding_head: :+1:

1 Like

Holy moly this is cool.

btw, the URLs of your first 2 videoā€™s have expired and they no longer work.

1 Like

Thank you! Really appreciate it.

Also, thanks for letting me know about the URLs, to stay true to this project, everything that could go wrong went wrongā€¦including the presentation; now I changed the videos so there shouldnā€™t be any more problems.

1 Like

Wow, thatĀ“s totally cool. Really awesome! Good to see smart people building such great things :smiley: :+1:

1 Like

Thank you!

Maybe a simple question, but I couldnā€™t figure out by examining the photos or your postsā€¦ where is the projector? Is it in the cabinet behind the mirror? Or is it mounted to your ceiling somewhere in the kitchen?

Very cool setup!

1 Like

There is no mirror, everything is solid wood (MDF). The Projector is installed in a ā€œcinema light projectorā€ on the ceiling.

You can see the video-projector installed in the ā€˜cinema-light-projectorā€™ on the video posted under HARDWARE

Also, in the first 2 videos, at the beginning, you will see that 1 light projector is turned towards the kitchen - that is where the projector is installed

Thanks!

Ah! Yes now I see it. Super nice :slight_smile:

One of the most important things for me was to have everything completely hidden, both projector and wiringā€¦so Iā€™m kind of glad you missed the projector

wow! This is amazing.

Love to see the mount for the projector and how that looks, at first I thought it was a seemless LED screen integration and was amazing at how perfectly it aligned but the projector idea is a great one!

In the th video you wans see the projector inside the ā€œlight shadeā€, ie the cinema ā€œlightā€

@smowk that is a great job.

I have a question about the projector though - the advertising refers to 4k, but the specs say the resolution is ā€œWVGA 854x480ā€

I know you are not using it in a home theatre, but that seems a contradiction in the advertising.

It is advertised as a 4k projector because it can decode 4k video from a source and ā€˜projectā€™ said video at WVGA resolution.

Most Chinese products are advertised in such way, but the actual video quality is really good for itā€™s size and priceā€¦plus, Iā€™m not trying to get a 100" video projection out of it