Across the moor from the the ancient family seat, the 14th-century Cawdor Castle, is Carnoch, the shepherd's summer dwelling that dates back to the 1890s.
Wales Office Minister Stephen Crabb said the closure of the Cawdor Barracks - which Mr Hammond said was no longer "fit for purpose" - was "disappointing".
Mr Hammond said the Ministry of Defence (MoD) was working with the Welsh government to maximise the use of St Athan, while the MoD will dispose of the Cawdor site.