Median average helper spikes when device becomes unavailable

I have a median average helper to get the average of a few humidity sensors. One of my sensors randomly becomes unavailable sometimes. It is very low compared to my order sensors. When it becomes unavailable the humidity jumps up a lot. Here is an example:
image
Is there a way that I can avoid seeing these spikes?

I’ve wrestled with this problem before. Assuming you are using the Statistics Integration, I was able to circumvent this problem (1) by using a sensor value and not an attribute as the source such that the historical data would be recorded in the HA logger and (2) by blocking automations from triggering around HA reboot with a start up sensor. You may choose to filter automations sufficiently distant from unavailable time-periods.

template:
  - trigger:
      - platform: homeassistant
        event: start
    binary_sensor:
      - name: "Starting Up"
        state: "true"
        auto_off:
          seconds: 250