The expression probably comes from the physical act of turning your back toward someone, instead of speaking to him face-to-face.

VOA: special.2009.12.13

The expression meant that the person who cannot hold a candle to you is not fit even to be your servant.

VOA: special.2011.05.29

To keep your head above water is a colorful expression that means staying out of debt.

VOA: special.2009.05.10

One of these is the expression, carry a chip on your shoulder.

VOA: special.2010.01.17

One such expression is to lose your heart to someone.

VOA: special.2010.07.18