Rotary Encoder from discarded wireless mouse - strange oddity

Hello,

I have salvaged a rotary encoder (scrollwheel) from a defective wireless mouse and connected it to an ESP32 with ESPHome. The encoder works fine, but every ~5 steps when rotated quickly, the encoder generates a pulse in the opposite direction (e.g when I quickly scroll down, it suddenly registers a scrolling step upwards). The mouse already had that exact same issue when it was in normal operation, which is why it was discarded in the first place.
Can I fix this in the ESPHome code?

I currently have this YAML for the encoder:

sensor:
  - platform: rotary_encoder
    id: r_enc
    name: "Rotary Encoder"
    resolution: 2
    pin_b:
      number: 18
      inverted: true
      mode:
        input: true
        pullup: true
    pin_a: 
      number: 19
      inverted: true
      mode:
        input: true
        pullup: true
    filters:
      debounce: 0.25s

Best regards
Aaron