The cabinet minister's remarks come after an outcry among some Conservatives that the party is failing to connect with traditional blue-collar members - following its defeat in the Eastleigh by-election on Thursday.
Tory Party vice chairman Michael Fabricant said the party had struggled to present a distinctive message while UKIP had managed to connect with traditional blue-collar Tory voters on issues such as immigration and crime in a way that the Conservatives had not.