Assist Pipeline not working for Atom Echo due to random UDP listening port of Home Assistant Core

ESPHome
1

Hi,
I was following this guide $13 voice assistant for Home Assistant - Home Assistant but am struggling to use the Atom Echo as an assist device.

Having completed the setup as per the instructions, the Atom Echo is running standalone connected via Wifi and I am seeing it as an ESPHome device in Homeassistant with 8 entities. The ESPHome dashboard is available and I can configure assist pipeline etc. and am seeing the button switching status when pressed.

image
image2028×1430 265 KB

However, when trying to send voice commands, nothing is happening. The assist pipeline is waiting for the recording to finish and I’m getting the following error in whisper:

ERROR:asyncio:Task exception was never retrieved
future: <Task finished name=‘Task-14’ coro=<AsyncEventHandler.run() done, defined at /usr/local/lib/python3.9/dist-packages/wyoming/server.py:28> exception=ValueError(“can’t extend empty axis 0 using modes other than ‘constant’ or ‘empty’”)>
Traceback (most recent call last):
File “/usr/local/lib/python3.9/dist-packages/wyoming/server.py”, line 35, in run
if not (await self.handle_event(event)):
File “/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/handler.py”, line 69, in handle_event
segments, _info = self.model.transcribe(
File “/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/faster_whisper/transcribe.py”, line 124, in transcribe
features = self.feature_extractor(audio)
File “/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/faster_whisper/feature_extractor.py”, line 152, in call
frames = self.fram_wave(waveform)
File “/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/faster_whisper/feature_extractor.py”, line 98, in fram_wave
frame = np.pad(frame, pad_width=padd_width, mode=“reflect”)
File “/usr/local/lib/python3.9/dist-packages/numpy/lib/arraypad.py”, line 819, in pad
raise ValueError(
ValueError: can’t extend empty axis 0 using modes other than ‘constant’ or ‘empty’

I’m running home assistant core 2023.11.3 as docker compose image and have added and connected piper, whisper and openwakeword as separate images (latest)via whyoming protocol. The assist pipeline is working fine when triggered from my notebook via voice commands. Presumably whisper is not getting any audio.

The 8 entities associated with the Atom Echo are the following:

  • binary_sensor.m5stack_atom_echo_0f9900_assist_in_verwendung
  • binary_sensor.m5stack_atom_echo_0f9900_button
  • button.m5stack_atom_echo_0f9900_factory_reset
  • light.m5stack_atom_echo_0f9900_m5stack_atom_echo_0f9900
  • select.m5stack_atom_echo_0f9900_assist_pipeline
  • select.m5stack_atom_echo_0f9900_zu_ende_gesprochen_erkennung
  • switch.m5stack_atom_echo_0f9900_use_listen_light
  • switch.m5stack_atom_echo_0f9900_use_wake_word

I have seen in screenshots of other users that they have an audio volume slider for the speaker in their default ESPHome dashboard that I’m missing, presumable for some reason the microphone and speaker entities of the Atom Echo have not been added to Home Assistant / ESPHome.

Any ideas how to add mic and speaker here? I have tried the same process referenced above three times, each time with the same result, and am running out of ideas.

Appreciate any support in getting this fixed.
Many thanks & kind regards
Wolfgang

2

p.s.: the Atom Echo is pulsing blue when the button is pressed and “assist in use” is enabled, so it should be listening.

3

some additional info: as per this post, presumably this is a firewall issue. My setup is to run everything in dedicated docker compose containers in a docker local network and make individual ports available via nginx reverse proxy to other devices in my wlan. The referenced post suggests that HA is opening a random port that the ESPhome device should connect to afterwards. Is there any way to predefine the port or port range for such connection?

1 Like
4

found similar posts Wake word & Satellite audio stream using seemingly random UDP high (> 1024) port · Issue #105106 · home-assistant/core · GitHub, ESPHome voice_assistant does not work with docker bridge networking · Issue #95654 · home-assistant/core · GitHub and Reddit - Dive into anything, apparently it is not intended for home assistant to get a configurable port range for inbound UDP, as the supported network mode is host network. But even then there might be issues with firewalls between HA and voice devices, so I would highly appreciate a configurable option for UDP ports opened by HA within a predefined range only.

5

I have essentially an identical setup to yours (all docker containers on a bridge network with an nginx reverse proxy for the specific ports) and have exactly the same problem with the logs implying no data is sent to whisper. Having inspected a packet capture, I can confirm that Home Assistant opens a random port and is trying to receive data from the M5 sending on port 6055.

A temporary fix that I’ve found to work was to change my Home Assistant network from being on my docker bridge network to host type, i.e. using network_mode: "host" in my docker compose file for the Home Assistant container, instead of using the docker bridge network and forwarding an external port to container port 8124. I’d prefer the container network to be isolated from the host’s network interface but my LAN is only accessible externally via a VPN so I’m not overly concerned from a security standpoint.

That doesn’t quite solve (y)our problem but it at least I was able to confirm my M5 works!

1 Like
6

Thank you, that helped me on my firewall setup

1 Like
7

I found this really frustrating, at first I thought it might have been some kind of mDNS issue, but in the end it was the random UDP port being blocked that caused the issue. I opened all UDP ports to the device on the firewall setup I have, which I don’t find to be a good solution. It might be possible to narrow it down by digging into the code.

The command would be something similar to this, in iptables, where the IP address is the esphome source device:

iptables -I INPUT 1 -s 192.168.0.1 -p udp -m udp -j ACCEPT

Also, for reference, esphome uses the following mdns services, which you may need to allow on/across the network for ota updates and other services:

_esphomelib._tcp.local
_http._tcp.local
8

I created a pull request to address this using a fixed port.