If it is working faster on USB2, it sounds like it has badly implemented USB-Attached-SCSI protocol… Look up UAS Quirks and give the workaround you find a try, and see if it performs better. From memory you get a lot of messages containing the string “uas” in dmesg when it’s occurring, but it’s been a loooooong time since I had a device with that sort of issue.
If that is the issue, once you apply the quirks workaround you get much better performance. Not as good as if UAS was implemented properly by the manufacturer but much much better than USB2 or a SD card