Text Assistant not recognizing any phrases in German

gschmidl · May 15, 2023, 1:53pm

I did a clean install of HA on a Yellow yesterday, only copying over data from my previous installation for some Add-Ons (e.g. importing the zigbee2mqtt topography into ZHA, Grocy database, that kind of thing) so I’m reasonably sure I didn’t screw up any base functionality. I then updated to the latest HA Core released yesterday.

And now Assistant, in text mode, will not understand anything I type in German. The language of HA and everything in the pipeline is set to German, and it prompts and responds in German, but nothing is understood.

This includes sentences I know used to work on the old install. Debug just shows it matches no intent. I’ve looked at the example sentences in German and tried a number of phrasings. Nothing works, I always get “sorry, I didn’t understand”.

“schalte licht büro an”
“schalte licht im büro ein”
“schalte büro licht ein”
etc.

When I switch the Assistant to English, it works immediately. “turn on Büro light” does just that. When I switch back, it doesn’t. I’m starting to wonder if the intent files are just missing, but I don’t know where they’d be stored.

Any ideas?

gschmidl · May 15, 2023, 9:34pm

Ok, I have added the following to configuration.yaml and reloaded YAML.

conversation:
  intents:
    YearOfVoice:
      - "wie geht das jahr der stimme"

intent_script:
  YearOfVoice:
    speech:
      text: "Great! We're at over 40 languages and counting."

It still doesn’t work.

mfreudenberg · August 25, 2023, 6:30am

Hi @gschmidl,

did you get it working? I have recently setup Assist on my HA instance with the openwhisper and piper add-ons. Unfortunately, i am facing similar issues like you. In some cases i do get a response from the assistant (for my AVM DECT smart plug).

But when i try to get the temperature of my heating in my bathroom (for example) i can’t make up a sentence, that returns a response. I tried to use this as a template for my sentences.

My first questions would be:

How do you name your entities?
Do you use any aliases in you assistant?

gschmidl · September 29, 2023, 7:28pm

I never got it working and the bug was closed without anyone doing anything.

mfreudenberg · October 1, 2023, 12:50pm

so recently, i was able to switch on my lamp using text. But still voice is not working. I’ve already checked several devices (Android, iOS, Laptop). But none of them seem to be able to hear my voice. So it seems, that something is wrong with either transmitting the voice, or “listening” on my HA side.
I’ve checked my config, the whisper-instance is running. I don’t know where i could be searching for an error message.

paddy0174 · October 1, 2023, 1:16pm

Has either of you tried the Assist feature in dev tools? You can test your sentences there, but I’m starting to get the feeling, that’s not it.

For me, it looks like you’re not getting the correct intent.

Let’s see and start with the basics:

all integrations necessary are there? Please post your configuration.yaml to check.
the pipelines are setup correctly? Please check and show.
are these custom sentences? If so, how are they “installed”? Please post the file.

With these we should have a starting point to investigate further.

mfreudenberg · October 1, 2023, 7:33pm

Has either of you tried the Assist feature in dev tools? You can test your sentences there, but I’m starting to get the feeling, that’s not it.
Yes, some sentences did worked. Here is an example:

image428×512 27.7 KB

It works with switches, but with others it’s not working.

all integrations necessary are there? Please post your configuration.yaml to check.
Um, yes i think so. Here is my configuration.yml

# Loads default set of integrations. Do not remove.
default_config:

http:
  use_x_forwarded_for: true
  trusted_proxies:
    - 192.168.0.20

homematic:
  interfaces:
    rf:
      host: 192.168.0.10
      resolvenames: xml
      port: 2001
      username: "secret"
      password: "secret"
    ip:
      host: 192.168.0.10
      port: 2010
    groups:
      host: 192.168.0.10
      port: 9292
      resolvenames: "xml"
      username: "secret"
      password: "secret"
      path: /groups

  hosts:
    ccu2:
      host: 192.168.0.10
      port: 2001
      username: "secret"
      password: "secret"

# Load frontend themes from the themes folder
frontend:
  themes: !include_dir_merge_named themes

# Text to speech
tts:
  - platform: google_translate

automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

wake_on_lan:

# i added the lines below from this forum thread.
conversation:
  intents:
    YearOfVoice:
      - "wie geht das jahr der stimme"

intent_script:
  YearOfVoice:
    speech:
      text: "Great! We're at over 40 languages and counting."

My wyoming config:

My piper config:

The Piper logs:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service piper: starting
s6-rc: info: service piper successfully started
s6-rc: info: service discovery: starting
DEBUG:wyoming_piper.download:Downloading https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/voices.json to /data/voices.json
DEBUG:wyoming_piper.download:Loading /data/voices.json
DEBUG:wyoming_piper.process:Starting process for: de_DE-thorsten-high (1/1)
DEBUG:wyoming_piper.download:Checking /data/de_DE-thorsten-high.onnx
DEBUG:wyoming_piper.download:Checking /data/de_DE-thorsten-high.onnx.json
DEBUG:wyoming_piper.download:Checking /share/piper/de_DE-thorsten-high.onnx
DEBUG:wyoming_piper.download:Missing /share/piper/de_DE-thorsten-high.onnx
DEBUG:wyoming_piper.download:Checking /share/piper/de_DE-thorsten-high.onnx.json
DEBUG:wyoming_piper.download:Missing /share/piper/de_DE-thorsten-high.onnx.json
DEBUG:wyoming_piper.download:Downloading https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/de/de_DE/thorsten/high/de_DE-thorsten-high.onnx.json to /data/de_DE-thorsten-high.onnx.json
INFO:wyoming_piper.download:Downloaded /data/de_DE-thorsten-high.onnx.json (https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/de/de_DE/thorsten/high/de_DE-thorsten-high.onnx.json)
DEBUG:wyoming_piper.download:Downloading https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/de/de_DE/thorsten/high/de_DE-thorsten-high.onnx to /data/de_DE-thorsten-high.onnx
INFO:wyoming_piper.download:Downloaded /data/de_DE-thorsten-high.onnx (https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/de/de_DE/thorsten/high/de_DE-thorsten-high.onnx)
DEBUG:wyoming_piper.process:Starting piper process: /usr/share/piper/piper args=['--model', '/data/de_DE-thorsten-high.onnx', '--config', '/data/de_DE-thorsten-high.onnx.json', '--output_dir', '/tmp/tmp0vv8n5am', '--json-input', '--noise-scale', '0.667', '--length-scale', '1.0', '--noise-w', '0.333']
INFO:__main__:Ready
[21:20:37] INFO: Successfully send discovery information to Home Assistant.
s6-rc: info: service discovery successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
DEBUG:wyoming_piper.handler:Sent info
DEBUG:wyoming_piper.handler:Sent info
DEBUG:wyoming_piper.handler:Sent info

My Whisper config:

The Whisper logs:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service whisper: starting
s6-rc: info: service whisper successfully started
s6-rc: info: service discovery: starting
INFO:__main__:Ready
[21:13:31] INFO: Successfully send discovery information to Home Assistant.
s6-rc: info: service discovery successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started

My Voice Assistant configuration (1/2):

My Voice Assistant configuration (2/2):

are these custom sentences? If so, how are they “installed”? Please post the file.

I don’t think so. At least some sentences do match (see above).

I think the issue is, that somehow the voice assistant does not “hear” anything. I have tried multiple devices as written in the post above. None of them, seem to take any audio via mic. I can see the “billowing” microphone icon on each devices, but my voice is not recognised by the assistant. I can type the sentences nontheless, which are recognized.

Any idea, what could be wrong?

I have enabled debugging log on both whisper and piper.

gschmidl · October 2, 2023, 12:06pm

It didn’t even work with Home Assistant Cloud. I used the debugger, but all it told me was that nothing was recognized.

As the cloud test subscription expired and my current CM4 is too low-RAM to run the pipeline locally, I can’t do anything with it at the moment - but simply switching piper and whisper to English made everything work, so IMO it can’t have been a configuration error.

mfreudenberg · October 3, 2023, 7:04pm

Hi @gschmidl,

does your voice command work? I think that all of my stuff seems to be configured correct, but only the voice reception does not work.

gschmidl · October 9, 2023, 10:59am

Sorry for the late reply - I don’t have a working whisper/piper right now so I can’t check, unfortunately. Last time I tried with Home Assistant Cloud it did not recognize anything unless I switched to English.

mfreudenberg · October 17, 2023, 10:07am

Today, i tried it with the newest app on android and it worked! Still i had to wait a few seconds, because i was on mobile and connected via VPN. I think i also changed the language model to tiny, which might be the main reason it worked.

gschmidl · October 27, 2023, 1:14pm

It now works for me with the new CM4 with more RAM as well. Hooray!

josefhajda · November 25, 2023, 6:10pm

Hi there. I am quite new to this topic but I have similar problems with setting up Assist…
I created the pipeline and I can’t get anything out of it. I created automations with sentences and The work fine but none of the standard sentences which should work, works.

I checked file manager and I can’t find a folder “config” with a subfolder for “de”.
Is it possible that my installation for whatever reason does not have the German sentences installed?

Any ideas? As hardware I use the yellow with the biggest CM4 I could find…

thanks!

josefhajda · November 25, 2023, 6:37pm

I think I can correct myself… was able to get some replies. Like activating the air condition in the living room or asking for the temperature of the Tado Thermostat.

is it possible that when you are used to Alexa, the Homeassistant Assist is just by far not as good? So perhaps are my expectations to high?

jhopmann2000 · November 26, 2023, 4:35am

Well, I never had Alexa, hence I cannot compare.

But I had everything local (No cloud) running on a PI4 with an Atom M5, the Assist function in my browser and the App on Android. I made the same experience.

For the Atom M5 I had to talk close (<20 cm) to achieve anything.
(My wife looked strange when I started shouting at this box)

Then I switched to Home Assistant C L O U D and that changed everything.
95% sentences recognized correct.
Result within less a second (compared to >5 sec before)

Additionally setting some alias helped a lot. E.g. some areas have the name “eatingarea” and this is not “eating area”.

Maybe my experience gives you some idea of the current situation.

josefhajda · November 26, 2023, 2:24pm

Hey @jhopmann2000
Thank you very much. this really helps. I already realized the alias thing and that I probably need to create automations for the things I want to do often. For me it mostly area few specific lamps and the TV Setup. I can live with creating the aliases and Routines. An Example for me is “Shield an” which was recognized as “Schild an”… I can handle that.

At the moment I am in the beginning and did not even move all things over from Alexa but I will do step by step and also implement some Atom M5 at critical spots (bed, desk etc.) and as soon as all of this is done perhaps I try the cloud. Not a fan of abo models like that but here it is for a good cause I think…

mfreudenberg · December 11, 2023, 12:15pm

Reporting back after some time.

Meanwhile i was able to get my assistant working. The problem as it seems was, that my instance was missing the custom_sentences folder with the corresponding languages (en, de …). I just cloned the intents repo on my instance and copied all the content of the sentences folder into the folder /homeassistant (which is linked to /config in my filesystem).
After a reboot of my HA it was working perfectly!

In addition i started the home assistant cloud trial and can say, that the recognition works very well. I’d say that 99% of my commands are recognized. I have to note, that i tried that at home, where there is usually quiet.

I was wondering if i can run the fast-whisperer instanze on another “beefier” machine (maybe even with a dedicated GPU), but that’s probably a question for another topic.