[Errno 111] Connection refused - altough ESP32 Cam is connected to Wifi

ESPHome
1

I am using an esp32cam (connected via USB Adapter ESP32-CAM-MB) and have successfully run the wizard and my ESP is connected to my Wi-Fi network at 192.168.1.10 (However, the name that I see in my router configuration is different).

When trying to upload the modified configuration, it always fails with the error ERROR Connecting to 192.168.1.10:3232 failed: [Errno 111] Connection refused
I have tried to set a static ip, set use_address (to the ip as well as using <esp-hostname>.localname), but none of it worked. This is my configuration

esphome:
  name: my-esp32cam

esp32:
  board: esp32cam
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  password: ""

ota:
  password: ""

wifi:
  ssid: "my-ssid"
  password: "ssid-pw"
  use_address: 192.168.1.10
  manual_ip:
    static_ip: 192.168.1.10
    gateway: 192.168.1.1
    subnet: 255.255.255.0

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "my-esp32cam Fallback Hotspot"
    password: "F6mmLx0rZX60"

captive_portal:

switch:
  - platform: gpio
    name: "Living Room Dehumidifier"
    pin: 5

What could be the issue for that?