The expression probably comes from the physical act of turning your back toward someone, instead of speaking to him face-to-face.

VOA: special.2009.12.13

Huge teeth stuck out of his face, and his expression showed much hate and violence.

VOA: special.2010.07.17

He had a big, fat, red face and a sleepy expression in his eyes.

VOA: special.2009.01.17

He had a big,fat,red face and a sleepy expression in his eyes.

VOA: special.2010.06.12

His expression would make her face go pale.

VOA: special.2009.09.26