Use Remote of Android TV Chromecast as Home Assistant remote?

Just wondering, would you be able to use the remote that comes with the Chromecast Android TV dongle for Home Assistant instead of the Android TV dongle? I already use my Harmony to operate the Android TV Chromecast dongle. I.e. pair it with Home Assistant instead of the Android TV instance and then assign actions to the buttons.