From a market structure perspective, the concern is that there is no practical way to opt out of interacting with these superfluous market makers, and because of the take fees charged by exchanges, directional traders are effectively forced to subsidize HFTs even though there are other directional traders they could interact with directly.
FORBES: High Frequency Traders And Current Market Structures Penalize Investors