There is also another path that so many people seem to ignore. Regular old home assistant in Docker…
Or hassio in docker which you could run on ubuntu quite easily. Personally I just moved to hassos (which is basically a minimal package to run docker in with hassio) if you use esxi you can simply download the vdmk from the HA website. I do not need the command line that much so can’t tell you much about that. Other stuff like plex etc run on another vm so I am not in need of mounting drives either, which to my knowledge is not possible in HassOS (Hass.io)
The only things that will not work well in hass.io are things that have a hard time communicating outside of their docker container or those that have to access something on the system level that the docker container can’t touch… These cases are pretty slim these days. The one that kept me on hassbian for some time was Tensorflow as the official component requires the libraries to be installed locally in the same place as your hass installation, and had no way to communicate across the boundary of a container. Now there are at least a couple options of dockerized tensorflow implementations that work well with hass.io (though at least in the case of the one I use, it’s not yet available as an add-on), they do things like communicate with MQTT.
Command line still works fine for most things unless you need a lot of custom commands that don’t come with most distros. Even if you do, you can make it work with some customization.
So, I don’t know everything you do, but I think if you were doing something that required local python libraries, you’d probably know it. These cases get more and more rare and specific by the day, and there are no longer any I know of that can’t be done in hass.io.
Hass.io is much easier to maintain than hassbian, and I’d recommend it for the vast majority of people. The people who need to run in VENV on bare metal probably already know that. I don’t think you will lose and “freedom and customizability” unless you are in that very narrow selection of people that need that access.
Maybe if you described some of the specific “highly custom” things you want to do, people can tell you if it’s possible in hass.io or not. More than likely it is.