If you do choose to compile it yourself, minor updates that don’t change the executable name (e.g. 3.5.1 to 3.5.4) don’t require you to do anything to the venv.
Just do a make install and check python3 --version is the new version and restart HA and you should be good to go.
I updated to 3.6.9 the other week and even then recreating the venv is not that much effort, you just need to reinstall your list of packages after recreating it. I just followed this