First lets tackle the performance. Since you are fiddling with proxmox and the rest, I infer that you are (more than) familiar with technology to understand that performance is mainly related to the complexity of the HA, integrations, etc, as well as the overall user experience that you want to achieve. 200 ms difference between different environment may or may not be significant for the end user.
Now for the more difficult question, I found this video on youtube that raises an interesting perspective: You are going to spend time with home automation. Depending on what is your passion and appetite for technology drives your choices. Personally i’m new on HA and i have spent the last two weeks trying different installations and methods on different devices (rasp PI, new windogs laptop with Virtualbox, old Centos server with KVM, etc) to settle finally with an old Dell laptop with Ubuntu and hassos running on kvm. It seems to give me adequate performance (i3 with 4 GB RAM) , the benefits of a VM environment, enough flexibility to build on it in the future and moderate consumption on electricity. I believe that any installation on small devices (rasp pi, etc) is short lived if you are going to be serious with home automation, but at the same time a good cheap starting point for experimentation.
So take into consideration the end-goal. Are you going to integrate a couple of devices and finish with smart home, or is it going to be a long term project on which you will keep adding stuff?
Please also note that the above are only my 2 cents and nothing more.