Turning the relay on when slider is moved

I am trying to control X9C with a slider (number component). I also have a relay connected to GPIO of ESP32.

What I am trying to achieve is:

(1) Relay should be OFF when ESP is powered on or rebooted.
(2) Control the resistance of X9C with number component.
(3) When I move the slider with the relay turned off, the relay is automatically turned on.

The relay turns on and off the device I want to control. Therefore, the relay must be ON for the slider to work. → the reason I need (3) above

When I unplug the device and plug it back on, I want to keep the device turned off (the device is in OFF state as long as the relay is OFF).

Since ESP is getting power as long as the device plugged in (independent from whether the device is on or off), I do not want to ESP to turn on the relay at power-on or reboot.

While i have no problem achieving (2) and (3) above, I am struggling with achieving (1).
With the code below, relay is always turned on when ESP is powered on or rebooted.

What am I missing here?

output:
  - platform: x9c
    id: x9c_potentiometer
    cs_pin: GPIO0
    inc_pin: GPIO1
    ud_pin: GPIO3
    initial_value: 0.01
    step_delay: 2us


switch:
  - platform: gpio
    id: relay1
    name: relay1
    pin: GPIO10
    restore_mode: RESTORE_DEFAULT_OFF

sensor:
  - platform: rotary_encoder
    id: r_encoder
    name: "Rotary encoder"
    pin_a: GPIO5
    pin_b: GPIO6
    min_value: 0
    max_value: 100
    restore_mode: RESTORE_DEFAULT_ZERO
    on_value:
      then:
        - output.set_level:
            id: x9c_potentiometer
            level: !lambda |-
              return x / 100.0;
    
number:
  - platform: template
    name: "Resistance"
    id: x9c_position_number
    min_value: 0
    max_value: 100
    step: 1
    optimistic: true
    set_action:
      then:
        - if:
            condition:
              - switch.is_off: relay1
            then:
              - switch.turn_on: relay1
              - delay: 2s
        - output.set_level:
            id: x9c_potentiometer
            level: !lambda |-
                return x / 100.0;

Then you should use restore_mode: ALWAYS_OFF

If that doesn’t fix it, capture esphome logs when you boot your esp.