Thanks for the suggestions!
I got wyoming-faster-whisper to work on the Apple M4 as an Wyoming endpoint. turbo and medium.en now both runs at about 5-8s. I assume this is because faster-whisper is not using MPS or MLX.
Will try to tinker with whisper.cpp and VOSK.