Code Server / VS Code has only increases in memory usage as the program grows and features are added by the code owners of the add-on.
Personally I have 16gb and don’t care how much it uses, but if you are not that lucky, shut it down when not in use.