Found the solution, incase anyone comes across this.
I was trying USB 3.0 ports on my PC. I was using a USB 2.0 hub plugged into USB3.0 but apparantly it didn't like that either. I have it direct plugged into a USB 2.0 now and was able to dedicate it to my guest VM.