Cartagena, Colombia (CNN) -- A group of Secret Service agents and officers sent to Colombia ahead of President Barack Obama were relieved of duty and returned home amid allegations of misconduct that involved prostitution, according to two U.S. government sources familiar with the investigation.