Issues with HSMs in general

Purpose and threat model are not always suitable
Crypto acceleration or/and security
At least symmetric crypto is often not faster than optimized code on CPU anyway
Attacks from compromised host or/and physical
Potential vulnerabilities
Firmware bugs, design errors, side-channels
Attack surface (too many features, each being a risk - can disable or not?)
No known whitebox audits, source code not available for review
Interfaces (physical, driver, API) and their reliability
Cost is often significant
Especially given that multiple HSMs need to be installed