Yes the 4 is a good upgrade and is definitely faster. I think you can boot from USB now which will make it a pretty satisfactory solution.
As an aside, you can also use a USB hard drive on Rpi 3. Either by booting off the SD Card then using the USB drive or booting directly off the drive - see this: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md I changed the bit to boot directly and it worked but was unreliable for my drive (a 2.5 inch 250 gb spinning rust) so switched to the “hybrid” mode by keeping in the sd card and booting off that. In any case, definitely an improvement for home assistant. Has eliminated the hesitancy I was experiencing. If you do this make sure you make a good backup of your SD card.