At least five Democratic incumbents also lost -- Rep. Ben Chandler of Kentucky, Rep. Larry Kissell of North Carolina, Rep. Leonard Boswell of Iowa, Rep. Kathy Hochul of New York and Rep. Mark Critz of Pennsylvania.

CNN: GOP retains grip on House

Altmire, who lost a primary battle to colleague Rep. Mark Critz after redistricting landed the pair in the same district, told O'Brien that he voted in favor of contempt because it was consistent with his vote to hold Bush administration White House Counsel Harriet Miers in contempt in 2008.

CNN: NRA sends Democrats a message over Holder contempt vote