High memory usage after backups

I recently moved from a raspberry pi4b to a proxmox vm, and since the move the memory usage has spiked during backups and remains high till the vm is restarted.
I did use a backup from the raspberry pi instance to restore onto the VM, and I do still have the raspberry pi instance untouched should I need to check something on there.

I fully understand memory usage can spike during the process, but before the move to proxmox it would drop down again after the backup has completed.

Just a note, I have googled this (I think extensively), and its not a case of proxmox reporting different values compared to home assistant.

Home assistant itself shows high (cache) ram usage - screenshot from system monitor;

Which corresponds with what htop is saying, as well as to what proxmox is saying.
Unfortunately new users can only upload one photo, so I cannot post more than that.

The ram usage stays like this until the server is restarted, and consecutive backups does cause the VM to run out of memory, which is currently every 3days (40% jump on each backup)

I have installed glances, and it shows the memory is used by hassio_supervisor (1.06gb at the time of this post), and homeassistant (802m as the time of this post)

Has anyone seen something like this before and perhaps know how to fix it ?

After the backup this morning it jumped again,

Glances showed hassio_supervisor was using 1.5gb at the time I looked.
So its definitely keeping backups in cache and not releasing it.