After tinkering with automations and even ESPHome, I decided to give creating my own component a try. As I am not familiar with HA internals so much, I used an LLM for scaffolding and bug hunting but I do speak python "a little" and generally know what I'm doing coding-wise.
So, this climate component can switch a heater and/or a cooler depending on outside temperature. You set your target temperatures and it basically works like any climate component, except that in cooling mode, it only switches the cooling effector on when the outside temperature is lower than the inside temperature and in heating mode, only switches the heating effector on when the outside temperature is higher than the inside temperature. It is possible to configure a hysteresis or "deadband" to prevent a "clicking war" of the relays when the temperature is close to either setpoint.
My use case is my bedroom. I CNC'd an acrylic plate with a bunch of 120mm Noctua fans which covers my typical Finnish small bedroom window in Summer. When it gets cooler outside in the evening, the fans start pumping cooler air inside. That fan contraption takes significantly less power than a full featured aircon would and it's perfectly enough for my purposes.
Link: GitHub - sgofferj/python-ha-component-simple-climate ยท GitHub