After reading this post, I set refresh_value: true and delay: 3 in my configuration.yaml file.
If I turn the light on and off at the switch, the card and the light are 180 degrees out of phase.
If the light and switch are on and I turn the switch off, the card drops one interval.
When I turn the light back on at the switch, the card indicates off.
If I turn the light off at the switch, the card indicates on.
Running Raspberry Pi 3, hass.io 64 bit. z-wave controller is Aeotec Z-Stick Gen5
What does it show on the States page? Does the state & brightness reported there agree with what you see in Lovelace?
Are the config parameters that control transition time the default values that result in a transition time of 2.97 sec? Or does the dimmer take longer to turn on and off? Have you tried setting the delay to a larger value, especially if the dimmer takes longer than 3 sec to turn on and off?