LED BLE connects on ESPHOME proxy but can not be added to Home Assistant

I am trying to add an existing LED Device I have. For this, I installed the bluetooth_proxy on an ESP32 device I have running ESPHome.

The Light is found on Home Assistant and showed to request integration. When I tried to add it, I see that the ESP32 tries to connect to it, and connects:

[15:25:02][I][esp32_ble_client:064]: [0] [78:9C:E7:0D:33:4D] 0x00 Attempting BLE connection
[15:25:02][I][esp32_ble_client:144]: [0] [78:9C:E7:0D:33:4D] Connected

But on Home Assistant the action fails with two error logs

Logger: homeassistant.components.bluetooth_tracker.device_tracker
Source: components/bluetooth_tracker/device_tracker.py:125
Integration: bluetooth_tracker (documentation, issues)
First occurred: 15:27:29 (9 occurrences)
Last logged: 15:28:09

Error looking up Bluetooth device
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bluetooth/bluez.py", line 289, in _gethcisock
    sock = _bt.hci_open_dev (device_id)
_bluetooth.error: no available bluetoot devices

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/bluetooth_tracker/device_tracker.py", line 165, in perform_bluetooth_update
    friendly_name = await hass.async_add_executor_job(lookup_name, mac)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/bluetooth_tracker/device_tracker.py", line 125, in lookup_name
    return bluetooth.lookup_name(mac, timeout=5)  # type: ignore[no-any-return]
  File "/usr/local/lib/python3.10/site-packages/bluetooth/bluez.py", line 61, in lookup_name
    sock = _gethcisock ()
  File "/usr/local/lib/python3.10/site-packages/bluetooth/bluez.py", line 291, in _gethcisock
    raise BluetoothError ("error accessing bluetooth device")
bluetooth.btcommon.BluetoothError: error accessing bluetooth device
Logger: homeassistant.components.bluetooth_tracker.device_tracker
Source: components/bluetooth_tracker/device_tracker.py:72
Integration: bluetooth_tracker (documentation, issues)
First occurred: 15:27:29 (9 occurrences)
Last logged: 15:28:09

Couldn't discover Bluetooth devices: [Errno 19] No such device

If I try to trigger the action again ESPHome logs say Connection already established and Home Assistant triggers the same two errors as before.

Eventually, the device disappears from Home Assistant.

Can anyone point me to what I am doing wrong?

Thank you

Edit: I also see this log

Logger: homeassistant
Source: components/esphome/bluetooth/client.py:519
First occurred: 21:32:50 (1 occurrences)
Last logged: 21:32:50

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/led_ble/led_ble.py", line 489, in _execute_timed_disconnect
    await self._execute_disconnect()
  File "/usr/local/lib/python3.10/site-packages/led_ble/led_ble.py", line 501, in _execute_disconnect
    await client.stop_notify(read_char)
  File "/usr/local/lib/python3.10/site-packages/bleak/__init__.py", line 735, in stop_notify
    await self._backend.stop_notify(char_specifier)
  File "/usr/src/homeassistant/homeassistant/components/esphome/bluetooth/client.py", line 102, in _async_wrap_bluetooth_operation
    return await func(self, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/esphome/bluetooth/client.py", line 723, in stop_notify
    characteristic = self._resolve_characteristic(char_specifier)
  File "/usr/src/homeassistant/homeassistant/components/esphome/bluetooth/client.py", line 519, in _resolve_characteristic
    raise BleakError(f"Characteristic {char_specifier} was not found!")
bleak.exc.BleakError: Characteristic None was not found!