The favour was repaid by United in the 21st minute when Loovens rose free to glance Shunsuke Nakamura's front-post corner past Lukasz Zaluska and put Celtic 1-0 ahead.
United continued to press, but Celtic killed off their challenge when Majstorovic dived low to fire into the near corner of the net from Mulgrew's free-kick to the frontpost.
Poor defence allowed Allen Chilten to touch down under the post, then on the half-hour Caerphilly were in front after Roddy Boobyer crossed in the corner.