He only saw his father once after he was a year old, and then his mother as mentioned sent him back to live with his grandparents when he was in the 5th grade.
The residences that line her street, where fellow artists Gilbert and George also live, were built in 1729 for the Huguenot silk weavers who had settled in London and as such are Grade II listed and protected.