After many months of experimenting, I finally reached a point where I’m genuinely happy with the results.
At first I thought I would need a much larger model (and much more expensive hardware) to get natural Czech dialogue. Instead, I spent my time building a high-quality dataset and training a LoRA.
The LoRA focuses on improving:
- natural Czech word order
- conversational flow
- emotional consistency
- personality consistency
- long conversations
It does not try to increase factual knowledge. The goal was simply to make the model sound like a native Czech speaker.
Training was done with Unsloth on Qwen3.5-9B .
Main settings:
- Rank: 64
- Alpha: 128
- Dropout: 0.05
- Epochs: 20
- Learning rate: 8e-5
One thing that surprised me was inference.
At first I thought the LoRA was overtrained because the model became overly philosophical and sometimes repetitive.
It turned out the training was fine.
Reducing the LoRA multiplier to 0.6 completely changed the behavior and produced much more natural conversations.
That single setting made a much bigger difference than I expected.
I’m still improving it, but this is the first time I’ve felt that the model actually sounds like the Czech assistant I wanted to build.
Thanks to everyone working on Qwen and Unsloth. These tools made this possible.https://huggingface.co/ondrazarici/lora.cz.gguf