Of course, it's plausible that the culture came first drinking milk gave some advantage to all, because milk has nutrition other than lactose in it, but it especially benefited adults with mutations that allowed lactose digestion.
WSJ: Matt Ridley on Human Culture's Effect on Genes | Mind & Matter