Bridged philips hue lights to homekit do not keep color inside apple Home app

I have a strange issue where I am able to change the color of my philips hue lights with the color wheel in the Home app on my iPhone, but the color picker jumps back to the original color when I let go. The bulb itself does change color and stays the way it is.

This results in my scenes not working properly.

I’m using Homekit and Homekit controller integrations to bridge my philips hue lamps/bridge.

For clarification, this does not happen when I add the hue lamps directly into Homekit without home assistant.

Some system details

System Health

version: core-2021.12.5
installation_type: Unknown
dev: false
hassio: false
docker: true
user: abc
virtualenv: false
python_version: 3.9.5
os_name: Linux
os_version: 4.4.59+
arch: x86_64
timezone: UTC

logged_in: false
can_reach_cert_server: ok
can_reach_cloud_auth: ok
can_reach_cloud: failed to load: timeout

dashboards: 1
resources: 0
mode: auto-gen