Over a few decades this would amount to a minute's difference, but over several hundred years this would mean the atomic clock time-scale and the time-scale based on the Earth's rotation would be out by an hour.
For one, the light that falls on your eyes is meaningless because it could mean anything, it's what we do with information that makes the difference, in how we find patterns and associate those patterns with a useful meaning.