M5stack Paper Battery Life

I am trying to use an M5stack Paper as a fridge display for Temperature and Calendar events. I am using hass-lovelace-kindle-screensaver to capture a dashboard image and then the online image component to grab the image and present it on the screen. I am using the code below but I am only getting about 2 days of battery life. I feel like I should be able to squeeze out at least a week on charge but I must be missing something. Does anyone have any insight on what may be draining the battery down so quickly or what I can change to get more battery life?

substitutions:
  device_name: m5stack-paper1

esphome:
  name: ${device_name}
  name_add_mac_suffix: true
  on_boot:
    - priority: 220.0
      then:
        - it8951e.clear
        - delay: 100ms
        - it8951e.updateslow
    - priority: -100.0
      then:
        - delay: 10s
        - it8951e.updateslow
api:
  encryption:
    key: ""

ota:
  - platform: esphome
    password: ""

esp32:
  board: m5stack-grey
  framework:
    type: arduino

external_components:
  - source:
      type: local
      path: /config/components

# Enable logging
logger:
  level: INFO

# Enable psram
psram:

spi:
  clk_pin: GPIO14
  mosi_pin: GPIO12
  miso_pin: GPIO13

i2c:
  sda: GPIO21
  scl: GPIO22

display:
  - platform: it8951e
    id: m5paper_display
    cs_pin: GPIO15
    reset_pin: GPIO23
    reset_duration: 100ms
    busy_pin: GPIO27
    rotation: 270
    # reversed: True
    reversed: False
    update_interval: never
    lambda: |-
      // Draw the image my_image at position [x=0,y=0]
      // with front color "OFF" and back color "ON"
      it.image(0, 0, id(calender), COLOR_OFF, COLOR_ON);

#interval:
  #- interval: 60min
    #then:
      #- component.update: calender
touchscreen:
  - platform: gt911
    display: m5paper_display
    id: gt911_touchscreen
    interrupt_pin: GPIO36
    transform:
      mirror_x: true
      mirror_y: false
      swap_xy: true
    calibration:
      x_min: 0
      x_max: 540
      y_min: 0
      y_max: 960
      
wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  power_save_mode: "HIGH"
  use_address: 
  #on_connect:
    #- component.update: calender


switch:
  - platform: restart
    id: restart_switch
    name: ${device_name} Restart

time:
  - platform: homeassistant
    id: ha_time
    on_time_sync: 
      - bm8563.write_time
  - platform: bm8563
    id: rtc_time
    sleep_duration: 3600000ms
    on_time:
      - seconds: 0
        hours: /1
        then:
          - component.update: m5paper_display
          - component.update: calender
          - component.update: m5paper_battery_level
      - seconds: 0
        minutes: 0
        then:
          - bm8563.apply_sleep_duration
          - delay: 1s
          - m5paper.shutdown_main_power

  - platform: homeassistant
    id: homeassistant_time
    timezone: NA/REGINA
    on_time_sync:
      - bm8563.write_time

m5paper:
  battery_power_pin: GPIO5
  main_power_pin: GPIO2


http_request:
  verify_ssl: false

online_image:
  - url: "http://192.168.1.50:5001"
    format: png
    id: calender
    type: BINARY
    

sensor:
  - platform: adc
    disabled_by_default: true
    pin: GPIO35
    name: ${device_name} battery voltage
    id: m5paper_battery_voltage
    update_interval: 60min
    attenuation: 11db
    filters:
      - multiply: 2 #1,27272727
  - platform: template
    name: ${device_name} battery level
    id: m5paper_battery_level
    unit_of_measurement: '%'
    device_class: "battery"
    state_class: "measurement"
    icon: mdi:battery-high
    update_interval: 60min
    lambda: |-
      constexpr float min_level = 3.52;
      constexpr float max_level = 4.15;
      return ((id(m5paper_battery_voltage).state - min_level) / (max_level - min_level)) * 100.00;
    filters:
      - clamp:
          min_value: 0
          max_value: 100

binary_sensor:
  - platform: gpio
    name: ${device_name} right button
    id: right_button
    icon: mdi:gesture-tap-button
    pin:
      number: GPIO37
      inverted: true
  - platform: gpio
    name: ${device_name} BTN/PWR button
    icon: mdi:gesture-tap-button
    pin:
      number: GPIO38
      inverted: true
  - platform: gpio
    name: ${device_name} left button
    icon: mdi:gesture-tap-button
    pin:
      number: GPIO39
      inverted: true

You should be using deep sleep, and wake only every x minutes (half an hour?) to update your display.