Smoothly dim a mix of Zigbee/Hue and WiFi/Shelly lights from a Hue Dimmer Switch or similar

Hi all - long-time lurker with a problem I can’t seem to crack on my own.

Goal:

Smoothy dim a combination of Hue and “dumb” LEDs in the same room from both a physical switch (e.g. Hue 4-button Dimmer Switch) and Home Assistant.

Background:

I like a lot of light, so my room currently has a Shelly Dimmer 2 controlling my single overhead light (high output LEDs), along with several lamps (Hue bulbs) to fill in the gaps (currently running through the Hue bridge v2).

What I have tried so far:

I installed a Shelly Dimmer 2 for my overhead fixture. I then created a HA light group with both the Shelly and the Hue native light group. This works fine for on/off and dimming to a specific value (e.g. go directly from 100% to 37%).

What does not work is smooth dimming when holding down a physical button on the Hue Switch. The lights instead jump between brightness levels.

I see two possible causes for this:

  1. The switch is sending a bunch of sequential button presses to HA, which in turn is sending sequential “brightness down X%” commands.
    1. I assume this could be resolved by HA sending a single “continuously reduce the brightness until I stop transmitting” command, if that even exists…
    2. This supposedly works in DeConz, although I assume only for Zigbee devices/not for a Shelly?
  2. This could be a side-effect of me using a non-recommended WiFi-based bridge (Sonoff ZBBridge running Tasmota). Maybe a large number of my button presses are just being lost?.

Some possible solutions I have considered:

  1. Purchase a USB ZigBee stick and pray it works better (although my gut says it won’t matter).
  2. Replace my Shelly with a zigbee-based dimmer such as the Enbrighten 43080. Add the dimmer to a ZigBee light group alongside the Hue bulbs. Pray this allows for proper dimming of the whole group, not just the bulbs attached to the dimmer.

Any advice would be appreciated, either on how to debug or possible solution.

Thank you in advance!

Edit: Fixed formatting

I have this working with Conbee II + Deconz + NodeRed + Hue lights. The setup is fairly specific but it works really wel.