Mr Cameron's visit came as First Minister Jack McConnell travelled to London to endorse a partnership agreement with Tony Blair on policy co-operation.
Mr Cameron's position was met with incredulity from opposition backbenchers, with former Home Secretary Jack Straw arguing that Mr Hunt's behaviour demonstrated a "palpable dereliction of duty".