I sometimes wonder if this simply comes from a lifetime of being late, missing appointments and generally expecting oneself to forget something important.
I also hate being scared to do something because I might forget time and miss the important appointment. Usually it is possible to set a timer. But sometimes it feels dumb to set a timer for 10 minutes but then I hate waiting. Luckily for me it all is still on the scale of annoying rather than hindering my life in any major way. But still a lot of time goes into just being early and not doing shit because other shit is up later.