Local homebridge integration

I have a supervised homeassistant installation running on my RPi 3, the same server also runs a homebridge. Homeassistant is able to detect homebridge (as BaseStation xxxxx via Homekit Accessory) but when I tried to enter the pairing code I kept getting the following error:

Incorrect HomeKit code. Please check it and try again.

What I know so far:

  1. There is no other Apple Homekit devices at home.
  2. I am able to connect to this HomeBridge via my IPhone, so the homebridge is set up properly.

Wondering if there is any other places I should be checking? Thanks in advance!