The expression probably comes from the physical act of turning your back toward someone, instead of speaking to him face-to-face.

VOA: special.2009.12.13

And the head you are seeing at the back there he was trying to touch is his wife's face.

他用手于去触摸的那个人,是他的妻子。

斯坦福公开课 - 7个颠覆你思想的演讲课程节选

the face being drawn back in a universal expression that every human everywhere would be able to recognize.

用一种普遍的面部表情，脸被后拉,任何地点，任何人都能辨认。

耶鲁公开课 - 心理学导论课程节选