How to prevent switch toggle when there is a delay on the binary sensor?

Hello,
I did setup a binary sensor connected to the USB output of the TV to monitor on/off state of my TV. When TV is on there is 5V on the USB.
There is also a switch that uses an IR transmitter to toggle the state.
When I switch on the TV from the HA, there is no problem. The esp32 picks up the “On” state immediately. However, when I switch off, the power to USB is still on for about 2 to 3 seconds. This makes the switch toggle between the states on the frontend. I tried adding a delay on the turn off command, I tried publishing the state manually for the turn off command but neither of them worked.
Is there any other smart way to do this?

Here is the code:

# Monitor TV State
binary_sensor:
  - platform: gpio
    name: "TV Status"
    id: tv_power_status
    pin:
      number: 34
      mode: INPUT_PULLDOWN

 # IR Switch
  - platform: template
    name: "TV Power"
    lambda: |-
      if (id(tv_power_status).state) {
        return true;
      } else {
        return false;
      }
    turn_on_action:
      - remote_transmitter.transmit_lg:
          data: 0x20DF10EF
          nbits: 32
    turn_off_action:
      - remote_transmitter.transmit_lg:
          data: 0x20DF10EF
          nbits: 32
      - lambda: |-
          id(tv_power_status).publish_state(false);

The sensor is already working. The problem is when switch toggles off it toggles back on because the binary sensor is still on.