The expression probably comes from the physical act of turning your back toward someone, instead of speaking to him face-to-face.

VOA: special.2009.12.13