ESPHome Ducky - MQTT keyboard/mouse control

jwm · January 9, 2022, 4:32pm

What is it?
Control devices with a virtual usb keyboard/mouse.

How does it work?

The arduino is emulating HIDs (Human Interface Devices) such as keyboard/mouse via USB
The ESPHome board is receiving commands via wifi/mqtt and writes them to the serial port
The commands are in Ducky Script syntax
The commands are read from the serial port by the arduino and executed as keyboard/mouse actions
Furthermore it’s possible to get the host machine’s power state from the USB state via the serial port

For example, this will open up notepad and type some text in it:

service: mqtt.publish
data:
  topic: esphome/nodemcu/run
  payload: |
    GUI r
    DELAY 200
    STRING notepad.exe
    ENTER
    DELAY 200
    STRING Hello World

You can do basically everything that a keyboard and mouse can do. Media keys and power buttons are also supported. See github project for more information.
More examples:

nickrout · January 9, 2022, 7:26pm

I have just tested a rpi pico, which work out of the box as a HID device. My use case is slightly different, connecting physical buttons and a rotary encoder to a media player. You could join the pico to an esp to make a wifi device.

jwm · January 11, 2022, 4:13pm

I guess my arduino sketch wouldn’t work on the pico? But yeah, the concept should be applicable. Has the pico any major advantages?

nickrout · January 15, 2022, 3:29am

The pico had the advantage of being

small
available within a few days in a country that goes to sleep during summer.
it runs micropython and circuitpython which meant I could shamelessly use a howto I found on the net.

gekberlin · August 2, 2022, 9:40am

Is there any chance that, if you use an esp32 that you’ll receive an command from esphome and then send a keystroke with ble to a ble connected host?

jwm · August 2, 2022, 12:39pm

That seems like a solution for you: ESPHome BLE Keyboard

You can use the receive-command-part from my project and the send-part from that project.

rubin110 · August 14, 2022, 3:26am

Thank you! This is exactly what I need to wake up a PC from sleep which doesn’t have wake on LAN via the BIOS.

Can I just buy one of the Ducky built hardware dongles and flash over your project?

jwm · August 15, 2022, 8:52am

Hi, the device you’ve linked uses a different communication protocol (I²C instead of serial), so you would have to adjust that in both firmwares. I haven’t tried it myself though.

RichardSE · January 5, 2023, 7:59pm

Really cool project, @jwm
What do you think of the idea to let the Bluetooth radio of a ESP32 emulate a Bluetooth keyboard directly to the computer?

I found this, so the emulation part could work, I guess. But don’t know if you can combine the code to use Bluetooth to the computer, and Wi-Fi to the Home Assistant server.

Edit: I now see that you wrote about something similar a few posts from the end.
But feel free to comment anyway.

gist.github.com

https://gist.github.com/manuelbl/66f059effc8a7be148adb1f104666467

README.md

# ESP32 as Bluetooth Keyboard

With its built-in Bluetooth capabilities, the ESP32 can act as a Bluetooth keyboard. The below code is a minimal example of how to achieve it. It will generate the key strokes for a message whenever a button attached to the ESP32 is pressed.

For the example setup, a momentary button should be connected to pin 2 and to ground. Pin 2 will be configured as an input with pull-up.

In order to receive the message, add the ESP32 as a Bluetooth keyboard of your computer or mobile phone:

1. Go to your computers/phones settings
2. Ensure Bluetooth is turned on

This file has been truncated. show original

main.cpp

/* 
 * Sample program for ESP32 acting as a Bluetooth keyboard
 * 
 * Copyright (c) 2019 Manuel Bl
 * 
 * Licensed under MIT License
 * https://opensource.org/licenses/MIT
 */

//

This file has been truncated. show original

jwm · January 5, 2023, 8:29pm

I tried that ble project and had some issues with android controls (power button IIRC) and some reconnect problems after sleep of the android-box. So I’m still using the arduino over usb.

In general I find NicoHood’s usb library (which my project uses) much more powerfull especially with special keys (even gamepad is possible) or different language layouts.

RichardSE · January 5, 2023, 8:36pm

Thanks for the reply. I’ll definitely build your project if the Bluetooth one isn’t stable enough.

gorgor_gonzales · April 20, 2023, 9:29pm

hey, I know this is a late follow up, but did you have any success with the Wifi Duck?
I’m also looking to power on my laptop and wol does not work through my thunderbolt dock, so this looks like an interesting solution…

pandaboy6621 · January 28, 2024, 12:14am

I successfully got this to work when plugged into my desktop. However when I plugged it into my tesmart KVM (HKS0202A2U) It wouldn’t pick up on the HID Keystrokes. Do you have any ideas why this didn’t work as expected?

jwm · January 31, 2024, 10:21pm

Since it has hotkey switching, I guess that it’s not passing through the arduino. Instead it will interpret the keyboard commands itself first and mimic the commands to the pc?

The arduino board firmware does setup 4 hid devices in the code. Maybe the kvm switch cannot handle that. You can try to enable only the keyboard, maybe this will work.

EDIT: Here is a possibly related issue: KVM Switch (Disable CDC Serial for better BIOS/UEFi/Custom compatibility) · Issue #225 · NicoHood/HID · GitHub