The 71-year-old Madoff pleaded guilty to the charges in 2009 and is now incarcerated at the Butner Federal Correction Complex, a medium-security prison in eastern North Carolina.
Department of Correction spokeswoman Kelly Nantel said Geoghan had been held in protective custody to shield him from the general prison population, but he still had some contact with other inmates.
This saga started on Tuesday when Robbins was taken from an Indianapolis prison to Illinois for a court hearing on unrelated weapons and drug charges, according to the Indiana Department of Correction.