Simplifying Whisper's task

I want to use speech to text in HA for basic tasks. Year of the voice is all round solution for handling everything related to voice.

Tryied various models and often the words aren’t correctly recognized, getting aliases of them.
I was thinking if there’s a way to reduce the task complexity by using only few word.
The verbs are a few (maybe they could grow but not too much)

  • open
  • close
  • turn on
  • turn off

Other words are the alias I inserted for every entity i need, to list a few:

  • light
  • blind
  • kitchen
  • room

So the vocabulary it’s really short.
I saw that every whisper model has it’s vocubaly, but it seems that if I modify it the model is downloaded again overwriting my custom vocabulary.txt file.

What’s the best way to handle my basic scenario?

Thanks in advance for any help
Carlo

p.s.
to modify the vocabulary on my ha supervisor addon container I use
docker exec -it addon_core_whisper bash
then inside I do
apt-get update
apt-get install nano
cd /data
ls
cd medium-int8
nano vocabulary.txt