Connection Issue With ESP32 Olimex Board

After updating to 1.20.0 the sensors on my Olimex board became unavailable. This has happened in the past after a restart while using the api (was using MQTT instead because of this) so I went to remove the integration and re-add it but it was no longer there. So I tried to add it and it said something about missing the api so I added it back in and removed mqtt from the config but it still wouldn’t add it. I tried to reinstall via ESPHome OTA but it wouldn’t connect even though my router showed it on the network. So I tried to reflash the board via USB and was able to get it connected to my WIFI and added the integration. When I tried to reconnect via POE the log says Errno 111. So it seems to work fine on WIFI but not via POE. Here is my config:

esphome:
  name: master_bedroom_closet_multi_sensor
  platform: ESP32
  board: esp32-poe
  
ethernet:
  type: LAN8720
  mdc_pin: GPIO23
  mdio_pin: GPIO18
  clk_mode: GPIO17_OUT
  phy_addr: 0
  power_pin: GPIO12  

#wifi:
#  ssid: "Wxxxxxt IOT"
#  password: "xxxxxxxxx"

  # Enable fallback hotspot (captive portal) in case wifi connection fails
#  ap:
#    ssid: "MB Closet Multi Sensor"
#    password: "SxxxxxV"

#captive_portal:

# Enable logging
logger:

# Enable Home Assistant API
api:

#mqtt:
#  broker: 192.168.1.120
#  username: mqtt_broker
#  password: Ba9xxxxfrdt

ota:

binary_sensor:
  - platform: gpio
    pin: GPIO14
    name: "Master Bedroom Closet Motion Sensor"
    device_class: motion 
    filters:
      - delayed_off: 15000ms
      - delayed_on: 0ms
      
sensor:
  - platform: dht
    pin: GPIO15
    model: dht22
    temperature:
      name: "Master Bedroom Closet Temperature"
      filters:
        - lambda: return x * (9.0/5.0) + 32.0;
      unit_of_measurement: "°F"
    humidity:
      name: "Master Bedroom Closet Humidity"
    update_interval: 60s

Any ideas what might be going on?