I know this is super late but if anyone else runs across this, here is a basic answer that should help point you in the right direction:
In short, you are probably not applying your card mod efficiently. Look here for more info on how to optimize card mod loading times.
If you really want to dive into details, this is the best source for everything card mod (also has info on some performance aspects).
Nevertheless, be aware that card mod is basically a method to change the styling of a card after it is generated, not a card itself. As such, you will always have some “flashes” but with smart usage these can be so fast they are hardly noticeable. Of course the processing time will always be longer/slower on older/less-powerfull devices. Remember that the more card mod you apply, the longer it will take to process it, this is unavoidable.
If you really want the card to load correctly right from the start, you’ll probably have more luck with something like the custom button card.