The expression probably comes from the physical act of turning your back toward someone, instead of speaking to him face-to-face.

VOA: special.2009.12.13

Suppose standing on the bridge next to the fat man, I didn't have to push him, suppose he was standing over a trap door that I could open by turning a steering wheel like that.

假设我站在桥上，胖子就在我旁边，我不用去推他，假设他踩在一扇活板门上方,而活板门可以通过转动方向盘来开启。

耶鲁公开课 - 公正课程节选